Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Why the private beta

4got is in a private beta. This page explains why and what that means for you.

The short version

I built this in my free time and I don’t want to expose early testers to security vulnerabilities that haven’t been found yet. A search engine handles sensitive data by nature — your queries reveal what you’re thinking about, what you’re worried about, what you’re researching. I’d rather bring in a small number of people at a time, find the obvious holes, and fix them before opening the doors to everyone.

The longer version

This project started as a personal itch. I wanted a search engine that didn’t suck, didn’t track me, and didn’t require a data center to run. I ralph’d it together in my free time — evenings, weekends, whenever I had the energy. It’s not backed by a company, it’s not funded, and there’s no team. It’s just me and a mass of Claude Code subagents.

That means my bandwidth for handling bug reports, security disclosures, and feature requests is limited. Open-sourcing a security-sensitive application before it’s ready invites problems I can’t respond to fast enough. A private beta lets me control the pace:

  • Bring in a few people. They use it for real. They find bugs I missed.
  • Fix everything they find. No new testers until the current batch is stable.
  • Repeat. Each wave finds fewer issues. The software converges toward something I’m comfortable putting my name on.

This is the same approach Mitchell Hashimoto used for Ghostty, and for similar reasons — not to manufacture hype or exclusivity, but to match the rate of incoming problems to the rate at which one person can fix them.

What this is not

This is not a “sign up for the waitlist and we’ll get back to you” situation. If you know me, ask me. If you don’t know me but you’re genuinely interested, join the community and make yourself known. I’m not trying to be gatekeepy — I just need to be able to sleep at night knowing I’m not serving search results through code that hasn’t been properly reviewed.

When will it be public?

When we’re confident in three things:

  1. Security. The code has been audited enough that I’m not losing sleep over someone’s queries leaking, their session getting hijacked, or the server getting popped through the image proxy or AI research agent.
  2. Direction. The architecture is settled enough that early adopters aren’t going to hit breaking changes every week. The federation protocol, the engine spec, the config format — these need to be stable before other people build on top of them.
  3. Obvious bugs. The kind of stuff that makes you look stupid. Engines that silently return zero results. Oracles that give wrong answers. Cache corruption. The stuff that’s embarrassing in hindsight but invisible when you’re the only person testing.

The code will be released under AGPL-3.0. There’s no open-core play, no commercial license, no bait-and-switch. The private beta is purely a quality gate.

How to get access

Access to the beta is via mTLS client certificates — see Friend Access for the technical details. In practice: I generate a certificate for you, hand it to you (preferably in person), and your browser uses it to authenticate. No accounts, no passwords, no third-party identity provider.

If you want in, reach out. I’m not hard to find.

Quickstart

Get 4got running in 5 minutes.

Build

git clone <repo-url> 4got && cd 4got
cd go-server && CGO_ENABLED=1 go build -o 4got . && cd ..

Configure

cp data/config.default.kdl data/config.kdl

Edit data/config.kdl — at minimum change owner-secret to something unique. The default is intentionally invalid.

For AI research, also copy and configure data/ai-research.default.kdl to data/ai-research.kdl with your OpenRouter/Groq/Ollama API key.

Run

go-server/4got

Starts on port 8888. Visit http://localhost:8888.

First use

Your first search triggers the IQ test (image captcha). Solve it once and you get a 24-hour session cookie. Returning users get a quicker PoW challenge instead.

Key pages

  • Settings — /settings (engine toggles, privacy, theme, banners)
  • Admin login — /admin/login (enter your owner-secret)
  • Trending — /trending (popular searches + live feed)
  • History — /history (enable in settings first)
  • AI Research — search the same query twice (double-Enter) or include “deep research” / “deep dive” in your query (requires deluxe group + AI config)
  • SSHssh localhost -p 2222 "your query" (if ssh-port configured)

Federation

To connect with a friend’s instance, add their URL to data/peers.kdl:

peer "https://friend.example.com" {
    secret "shared_peer_secret"
}

Both instances must use the same peer-secret in their config.

User Guide

4got is a metasearch engine. It queries dozens of upstream engines simultaneously, deduplicates and ranks the results, and serves them to you without tracking. This page covers everything you can do from the search bar.

Type a query and press Enter. Results come back ranked by cross-engine consensus: pages that multiple engines agree on float to the top.

Category tabs across the top switch between result types:

TabWhat you get
WebGeneral web results
ImagesPhotos, illustrations, stock images
VideosYouTube, Vimeo, PeerTube, Dailymotion, etc.
NewsRecent articles from wire services and aggregators
MusicTracks from SoundCloud, Bandcamp, Deezer, etc.
ScienceAcademic papers from arXiv, PubMed, Crossref, etc.
ITCode repos, package registries, docs
FilesTorrents (Pirate Bay, 1337x, Solid Torrents, etc.)
MapsOpenStreetMap and Photon geocoding
SocialReddit, Lemmy, Mastodon, HN, Lobsters
BooksOpen Library, Gutenberg, Goodreads
Q&AStack Exchange network (SO, Super User, Unix SE, etc.)
PackagesLinux distro packages (Arch, NixOS, Alpine, Void, AUR, Flathub)
Weatherwttr.in and Open-Meteo
LyricsGenius

Quoted phrases

Wrap terms in double quotes to require an exact phrase match:

"rust async trait" stabilization

Engines that support phrase matching will enforce it; the rest filter client-side.

Time filters

Add a time constraint from the settings or URL parameter time=:

ValueWindow
day or dPast 24 hours
week or wPast 7 days
month or mPast 30 days
year or yPast 365 days

Example: /search?q=linux+kernel&time=week

Bang commands

Start your query with ! followed by a shortcode. Bangs come in three flavors.

Redirect bangs

These take you directly to the target site. No 4got results page.

BangTarget
!gGoogle
!d or !ddgDuckDuckGo
!bBrave Search
!wWikipedia
!ytYouTube
!ghGitHub
!rReddit
!soStack Overflow
!aAmazon
!imdbIMDb
!mapsOpenStreetMap
!nixNixOS Packages
!archArch Linux Packages
!aurAUR
!cratescrates.io
!npmnpm
!pypiPyPI
!mdnMDN Web Docs

Example: !gh rust async runtime opens GitHub search for “rust async runtime”.

Engine bangs

These run the search through 4got but force results from a single engine.

BangEngine
!braveBrave
!startpageStartpage
!mojeekMojeek
!marginaliaMarginalia
!wibyWiby
!hackernews or !hnHacker News
!youtubeYouTube
!soundcloud or !scSoundCloud
!flickrFlickr
!pexelsPexels
!yandexYandex
!qwantQwant
!bingBing
!baiduBaidu
!yahooYahoo
!wikipedia or !wikiWikipedia
!googleGoogle
!duckduckgoDuckDuckGo

Oracle bangs

These redirect to an external instant-answer service.

BangService
!wolfram or !waWolfram Alpha Short Answers
!ddg-iaDuckDuckGo Instant Answers API

Oracles (instant answers)

Oracles are local computations or lightweight API calls that produce an answer box above the search results. No upstream search engine is involved. They trigger automatically when your query matches a pattern.

Calculator

Evaluates arithmetic expressions inline.

2 + 2                  → 4
15% of 200             → 30
sqrt(144)              → 12
2^10                   → 1024
sin(1.5)               → 0.9974949866
log10(1000)            → 3
7 mod 3                → 1

Supported functions: sqrt, cbrt, sin, cos, tan, asin, acos, atan, log (base 10), log2, log10, ln, abs, ceil, floor, round, exp.

Supported operators: +, -, *, /, %/mod, ^/**.

Unit converter

Converts between units of length, weight, volume, speed, and temperature.

10 km to miles         → 6.2137119224 miles
72 fahrenheit to celsius → 22.2222222222 celsius
5 gallons to liters    → 18.92705 liters
100 kph to mph         → 62.1371192237 mph

Supported units:

  • Length: km, m, cm, mm, mi, yd, ft, in
  • Weight: kg, g, lb, oz, st
  • Volume: l, ml, gal, qt, pt, cup, floz
  • Speed: mph, kph, mps, knot
  • Temperature: celsius/c, fahrenheit/f, kelvin/k

Timezone

Shows current time in a city or IANA timezone.

time in tokyo          → Monday, May 12, 2026 2:30 AM JST
time in UTC            → Sunday, May 11, 2026 5:30 PM UTC
time in America/Chicago → Sunday, May 11, 2026 12:30 PM CDT

Knows 70+ cities including Tokyo, New York, London, Paris, Berlin, Sydney, Mumbai, Shanghai, Seoul, Dubai, and more.

Base converter

Converts between decimal, hex, binary, and octal.

0xff in decimal        → 255
255 in hex             → 0xFF
255 in binary          → 0b11111111
0b1010 in decimal      → 10
0o77 in hex            → 0x3F

Color converter

Converts between hex, RGB, and HSL. Shows a color swatch.

#FF5500                → rgb(255, 85, 0) · hsl(20, 100%, 50%)
rgb(0, 128, 255)       → #0080FF · hsl(210, 100%, 50%)
hsl(120, 100, 50)      → #00FF00 · rgb(0, 255, 0)

Also accepts 3-digit hex (#F50).

Currency converter

Live exchange rates via exchangerate.host.

100 usd to eur         → 92.15 EUR
50 gbp in jpy          → 9,432.50 JPY

Uses standard 3-letter ISO currency codes (USD, EUR, GBP, JPY, BTC, etc.).

IP geolocation

Look up the physical location and ISP of a public IP address.

8.8.8.8                → Mountain View, California, United States · ISP: Google LLC
1.1.1.1                → Los Angeles, California, United States · ISP: Cloudflare

Also responds to where is 8.8.8.8 and geolocate 1.1.1.1.

Private IP helper

Detects private/reserved IPs and tells you what range they belong to.

192.168.1.1            → 192.168.1.1 is a private/reserved address in range 192.168.0.0/16 (Class C private)

Reverse DNS

Looks up PTR records.

rdns 8.8.8.8           → dns.google
reverse dns 1.1.1.1    → one.one.one.one
nslookup 8.8.4.4       → dns.google

DNS lookup

Type a bare domain name (or prefix with dig/dns) to see its A, AAAA, MX, and TXT records.

example.com            → A 93.184.216.34, MX 0 ., TXT "v=spf1 -all"
dig archlinux.org      → A 95.217.163.246, MX 10 mail.archlinux.org, ...

Package versions

Check the latest version of an npm or PyPI package.

latest lodash          → lodash 4.17.21
npm express            → express 4.19.2
pip requests version   → requests 2.31.0 — Python HTTP for Humans
pypi flask             → Flask 3.0.2 — ...

Radio stations

Find internet radio stations by location or genre.

radio tokyo            → [list of stations with inline audio players]
fm berlin              → [list of stations with inline audio players]

Results come from radio-browser.info.

Translation

Translate text explicitly or automatically.

translate hello world to french       → bonjour le monde
translate こんにちは to english        → hello

Auto-translation: if you search in a language that isn’t in your known_languages list, 4got translates the query and shows the translation in an oracle box above the results.

Translation backends rotate by priority: DeepL, Papago (CJK), Lingva, NLLB, Microsoft. Rate limits are per-cookie. See translation.kdl for configuration.

Wolfram Alpha

Queries the Wolfram Alpha Short Answers API. Requires an API key (server-level or per-user via the apikey_wolframalpha cookie).

population of france   → 67.75 million people (2023)
distance earth to mars → 225 million km (average)

Trivial/conversational queries (greetings, etc.) are suppressed. Negative results are cached for 30 days to avoid redundant API calls.

DuckDuckGo Instant Answers

Returns definitions, abstracts, and factual answers from DuckDuckGo’s Instant Answer API.

define serendipity     → [Definition from Wiktionary]
albert einstein        → [Wikipedia abstract]

DDG IA and Wolfram are tried in that order. The first one that returns a non-empty answer wins.

AI research agent

When you search the same query twice (or click the “Research” button), 4got activates the AI research agent. This is a tool-using LLM that:

  1. Reads the top search results
  2. Fetches and analyzes the linked pages
  3. Runs code in a sandboxed Lua environment if needed
  4. Streams findings to you in real time via the oracle box

The agent shows an ETA timer and intermediate results as it works. When done, the final synthesized answer replaces the stream.

The research agent requires server-side configuration (see ai-research.md). Operators choose the LLM provider (Cerebras, OpenRouter, Ollama, or any OpenAI-compatible API) and can restrict access to specific user groups.

Settings

Visit /settings to customize your search experience. All preferences are stored as cookies (nothing server-side unless you enable history).

SettingCookieWhat it does
Safe searchsafesearch1 (on) or 0 (off)
Temperature unittemp_unitC or F
RegionregionISO country code hint sent to engines (e.g. us, de, jp)
Engine badgesbadgesalways (show source badge on every result), multiple (only when >1 engine), never
Fanout hard windowhard_windowMaximum ms to wait for engine responses (default 1000)
Fanout idle timeoutidle_timeoutStop waiting if no new results arrive in this many ms (default 500)
Cache max agecache_max_ageCache TTL in minutes (default 1440)
Priority enginepriority_engineEngine that must respond before results are shown
NSFW bannersnsfw_bannersShow NSFW banners on the landing page
Historyhistory_enabledEnable/disable search history recording
Browser historybrowser_historyStore search history in browser localStorage
Server historyserver_historyStore search history on the server
Share searchesshare_searchesAllow searches to appear in the trending/live feed
Live feedlive_feedShow live search feed on the trending page
AutocompleteautocompleteEnable/disable search suggestions (disable by setting to off)
Show scoresshow_scoresShow ranking scores next to results
Prefer indieprefer_indieBoost results from independent/small web sites
Progressive SSEprogressive_sseStream results as they arrive via server-sent events
Speedtest graphspeedtest_graphShow engine response time graph
Smart tab oraclessmart_tab_oraclesShow oracle answers in non-web category tabs
Show faviconsshow_faviconsDisplay site favicons next to results
Search methodsearch_methodPOST or GET for the search form
Bookmarksbookmark_modeEnable/disable bookmark saving
Show filtersshow_filtersShow time/region filter bar on results page
Disabled enginesdisabled_<category>Per-category comma-separated engine IDs to skip
Engine weightsweight_<engine>Override ranking weight per engine (higher = more influence)
URL rewritesrewrite_<service>Per-service hostname overrides (YouTube, Reddit, Twitter, etc.)
Archive linksarchive_linksShow Wayback Machine archive links on results
Known languagesknown_languagesLanguages you read, comma-separated ISO 639-1 codes (controls auto-translation)
Tracker detectiontracker_detectionHighlight known tracking domains in results
Translation privacytranslation_privateAvoid sending translations through third-party APIs
Hidden tabshidden_tabsComma-separated list of category tabs to hide
AI streamai_streamEnable/disable streaming for AI research results
Research debugresearch_debugShow debug info during AI research
Wolfram API keyapikey_wolframalphaPersonal Wolfram Alpha API key (uses server key if empty)

Settings can also be set via URL parameters on /settings?nsfw=yes&region=de&... to create a shareable preferences link.

URL rewriting

By default, 4got rewrites result URLs to point at privacy-friendly frontend alternatives:

ServiceDefault rewrite
YouTubeyewtu.be (Invidious)
Redditold.reddit.com
Twitter/Xnitter.net
Mediumscribe.rip
Imgurrimgo
TikTokProxiTok
FandomBreezeWiki

Override any rewrite in settings with a custom hostname, or set it to off/none/disabled to keep original URLs.

Privacy features

  • No tracking. No analytics scripts, no user profiles, no fingerprinting.
  • No accounts. Preferences are stored as cookies in your browser, never on the server.
  • Proof-of-work gate. First-time visitors solve a visual captcha (IQ test). Returning visitors solve a client-side PoW challenge. This keeps bots and scraper farms out without requiring JavaScript tracking.
  • URL rewriting. Results from major trackers are rewritten to privacy frontends (see above).
  • Tracking parameter stripping. UTM tags, fbclid, gclid, and other tracking parameters are removed from all result URLs.
  • Image proxy. Images can be proxied through the 4got server to prevent your IP from leaking to image hosts.
  • No server-side history by default. Search logging is off unless you explicitly enable it in settings.
  • OpenSearch support. Add 4got as a browser search engine via /opensearch.
  • RSS feeds. Subscribe to search results via /feed/search?q=... without visiting the site.

API

4got exposes a JSON API at /api/v1/search?q=QUERY&cat=web. Per-category endpoints also exist: /api/v1/web, /api/v1/images, etc. See /api on your instance for full documentation, or /api/openapi.yaml for the OpenAPI 3.0 spec.

Keyboard shortcuts

  • Enter — Search
  • Enter twice (same query) — Trigger AI research agent
  • Tab — Cycle category tabs

Configuration Reference

4got uses KDL (KDL Document Language) for all configuration. All config files live in the data/ directory.

config.kdl

The main configuration file. All fields have sensible defaults.

FieldTypeDefaultDescription
portint8888HTTP listen port
hoststring“0.0.0.0”Listen address
owner-secretstring“CHANGE_ME”Admin password. Also used to derive Ed25519 signing keys for session cookies.
deluxe-secretstring“”Legacy cookie-based deluxe tier secret. Leave empty to disable.
pow-difficultyint22Proof-of-work difficulty (number of leading zero bits in SHA-256 hash)
captcha-bypass-hoursint24How long a captcha solution grants session access
pow-bypass-secondsint300How long a PoW solution grants session access
max-searches-per-dayint200Per-IP daily search limit
fanout-hard-window-msint1000Maximum time to wait for engine responses
fanout-idle-timeout-msint500Stop waiting if no new results arrive within this window
cache-ttl-web-sint1800Cache TTL for web results (seconds)
cache-ttl-images-sint7200Cache TTL for image results
cache-ttl-news-sint300Cache TTL for news results
prefetch-thresholdint2Minimum access count before a cache entry is eligible for background prefetch
prefetch-before-expiry-sint60Prefetch entries within this many seconds of expiry
nsfw-landingboolfalseShow NSFW banners on the landing page
data-dirstring“data”Path to the data directory (config files, DB, banners, captcha assets)
db-pathstring“data/4got.db”Path to the SQLite database
peersstring[][]Legacy peer URLs (prefer peers.kdl instead)
peer-secretstring“”Shared secret for peer authentication
self-heal-enabledboolfalseEnable autonomous self-healing agents for broken engines
quotes-modestring“all”Landing page quote visibility: “all”, “deluxe”, “off”
api-keysblock{}Third-party API keys (wolframalpha, deepl, freesound, spotify-client-id, spotify-client-secret)

Environment variable overrides

Every config.kdl field has a corresponding FGOT_* environment variable:

  • FGOT_PORT, FGOT_HOST, FGOT_OWNER_SECRET, FGOT_DELUXE_SECRET
  • FGOT_POW_DIFFICULTY, FGOT_CAPTCHA_HOURS, FGOT_POW_SECONDS
  • FGOT_MAX_SEARCHES, FGOT_FANOUT_HARD_WINDOW_MS, FGOT_FANOUT_IDLE_TIMEOUT_MS
  • FGOT_CACHE_WEB_S, FGOT_CACHE_IMAGES_S, FGOT_CACHE_NEWS_S
  • FGOT_DATA_DIR, FGOT_DB_PATH
  • FGOT_CONFIG — override the config file path entirely

KDL config files

4got uses KDL (KDL Document Language) for structured subsystem configs. Each subsystem looks for its config in two places:

  1. A dedicated file (e.g. data/peers.kdl)
  2. A named section inside any *.kdl file in data/ (e.g. a peers { ... } block inside data/conf.kdl)

You can use either approach but not both for the same subsystem. This lets you keep everything in a single conf.kdl or split into per-subsystem files.

peers.kdl

Configures the friend/peer network for cache sharing and service proxying.

// How many friends to wait for before proceeding.
// Float 0-1: proportion of 'up' friends (1.0 = all, 0.5 = half)
// Int >= 1: exact count
// Negative: "ok to miss" semantics (-2 = ok missing 2 friends)
wait-for 1.0

// Store results received from friends in local cache
store-friend-cache true

// Mark a friend as 'down' if they fail more than this proportion
// of requests over this time window.
failure-threshold 0.01 1d

// Services this instance allows friends to proxy through us
allow-proxy "translation" "wolfram" "autocomplete"

// Always wait for a specific friend (by name)
// always-wait-for "alice"

// Prefer a friend for specific services
// prefer-friend "alice" for="translation,wolfram"

peer "instance-b" {
    url "http://127.0.0.1:8890"
    // Optional mTLS:
    // cert "/path/to/client.crt"
    // key "/path/to/client.key"
}

groups.kdl

Defines user tiers/groups and their permissions.

group "deluxe" {
    allow "sidecar" "server_api_keys" "friend_proxy" "priority_fanout"
    label "deluxe"
}

group "researcher" {
    allow "server_api_keys"
    label "researcher"
}

Known permissions: sidecar, server_api_keys, friend_proxy, priority_fanout. Groups are assigned to users by cookie hash via the admin panel (/admin/users).

captcha.kdl

Configures the pre-rendered captcha (IQ test) pool.

pool {
    distorted-target 500     // target pool size for distorted tiles
    distorted-batch 50       // generate this many per cycle
    prebuilt-target 100      // target pool size for assembled captchas
    prebuilt-batch 20        // assemble this many per cycle
    cycle-count 25           // number of generation cycles at startup
    cycle-interval-min 5     // minutes between background replenishment cycles
}

autocomplete.kdl

Configures autocomplete suggestion backends and their weights.

backend "duckduckgo" weight=1 enabled=true
backend "brave" weight=1 enabled=true
backend "qwant" weight=1 enabled=true
backend "yahoo" weight=1 enabled=true
backend "wikipedia" weight=1 enabled=true
backend "google" weight=1 enabled=false

Higher weight = more influence when merging suggestions from multiple backends.

translation.kdl

Configures translation backends with priority routing and per-user rate limits.

// Language sets (non-overlapping)
set "cjk" "ja" "zh" "ko"
set "eu" "de" "fr" "nl" "pl" "es" "pt" "it" "ro" "sv" "da" "no" "fi" "cs" "hu" "el"

// Backends: priority (higher=tried first), languages ("all" or set name),
// limit-per-cookie, reset hours
backend "deepl" priority=3 languages="eu" limit=500 reset-hours=24
backend "papago" priority=3 languages="cjk" limit=50 reset-hours=24
backend "lingva" priority=2 languages="all" limit=100 reset-hours=24
backend "nllb" priority=1 languages="all" limit=0
backend "microsoft" priority=2 languages="all" limit=30 reset-hours=24

instances.kdl

Configures the instance browser (shows other 4got instances).

fetch-interval-min 60

friends {
    // instance "https://search.friend.example.com"
}

public {
    // instance "https://public.4got.example.com"
}

ai-research.kdl

Configures the AI research agent that synthesizes search results.

enabled true
mode "lua"                    // "lua" (sandboxed) or "bash" (dangerous)
provider "cerebras"
model "llama3.1-8b"
api-key "csk-..."
base-url "https://api.cerebras.ai/v1"
max-runtime-seconds 60
require "deluxe"              // group permission required

See ai-research.md for full details.

image-proxy.kdl (optional)

Configures the SQLite-backed image proxy cache.

enabled true
hold-time-days 30             // evict images unserved for this many days
jxl-reencode true             // reencode to JXL (requires cjxl binary)
redownload-threshold 999      // re-fetch original after N serves
require "deluxe"              // group permission required ("" for everyone)

The unified conf.kdl option

Instead of separate files, you can put everything in a single data/conf.kdl:

peers {
    wait-for 1.0
    peer "instance-b" {
        url "http://127.0.0.1:8890"
    }
}

groups {
    group "deluxe" {
        allow "sidecar" "server_api_keys"
        label "deluxe"
    }
}

translation {
    backend "lingva" priority=2 languages="all" limit=100 reset-hours=24
}

The resolver scans all *.kdl files for matching section blocks. It is an error to have both peers.kdl and a peers { ... } section in another file.

Users can customize their experience via cookies set on the /settings page:

CookieValuesDescription
safesearch1, 0Safe search filter (1 = on, 0 = off)
temp_unitC, FTemperature unit for weather oracle
regionISO country codeRegion hint for engines
badgesalways, multiple, neverEngine badge display mode
hard_windowinteger msMaximum time to wait for engine responses (default 1000)
idle_timeoutinteger msStop waiting if no new results in this window (default 500)
cache_max_ageinteger minutesCache TTL (default 1440)
priority_engineengine ID or “none”Engine that must respond before results are shown
disabled_web, disabled_images, etc.comma-separated engine IDsDisable specific engines per category
weight_<engine>integer >= 1Override engine weight in result ranking
rewrite_<service>hostname, “off”, “none”, “disabled”Override URL rewriting per service
known_languagescomma-separated ISO 639-1 codesLanguages the user knows (for auto-translation)
apikey_wolframalphaAPI key stringPersonal Wolfram Alpha API key
history_enabled“1”Enable search history recording
autocomplete“off” to disableEnable autocomplete suggestions
bookmark_modeon, offEnable bookmark saving
archive_linkson, offShow archive links on results
show_scores“1”Show ranking scores next to results
prefer_indie“1”Boost results from independent sites
search_methodPOST, GETHTTP method for search form
hidden_tabscomma-separated tab namesCategory tabs to hide
tracker_detection“1”Highlight known tracking domains

Engine KDL Specification

This document specifies the KDL format for defining generic engines in data/engines.kdl. Generic engines are parsed by go-server/generic_engine.go and registered alongside hand-written engines (Brave, Google, etc.) at startup. A single engines.kdl file defines all ~274 generic engines.

Engine block structure

Each engine is a top-level engine node with a quoted string ID and a child block of properties:

engine "my_engine" {
    name "My Engine Display Name"
    categories "web" "news"
    search-url "https://example.com/search?q={query}&page={page}"
    method "GET"
    result-type "json"
    results-path "data.items"
    title-path "title"
    url-path "link"
    content-path "description"
    page-size 20
    paging true
    first-page 1
}

The engine ID (the string after engine) is the unique identifier used in logs, cache keys, weight cookies, and the admin UI. It must be a short slug (lowercase, underscores allowed).

Attributes reference

Identity

AttributeTypeRequiredDefaultDescription
namestringNoSame as IDHuman-readable display name shown in engine badges and the settings page.
categoriesstring listYes(none – engine is skipped if empty)Space-separated quoted category slugs. Valid values: web, images, videos, news, music, science, it, files, maps, social, books, qa / q&a, packages, weather, lyrics. An engine with no parseable categories is silently skipped during registration.

Request

AttributeTypeRequiredDefaultDescription
search-urlstringYesURL template. Placeholders: {query} (URL-encoded query), {page} / {pageno} (page number = (page-1) + first-page), {offset} (offset = (page-1) * page-size + first-page), {lang} (hardcoded "en" currently).
methodstringNo"GET"HTTP method. "GET" or "POST".
body-templatestringNo""POST body template (same placeholders as search-url). Only used when method is "POST".
headerssub-blockNo(none)Custom HTTP headers as key-value pairs. Default headers (User-Agent, Accept, Accept-Language, DNT) are always sent; entries here override or add to them.

Response parsing – HTML mode

Used when result-type is "html" (or "HTML" or "xpath").

AttributeTypeRequiredDefaultDescription
result-typestringYesSet to "html" for CSS-selector-based parsing.
results-selectorstringYes (for HTML)""CSS selector for result container elements. Each match becomes one result.
title-selectorstringNo""CSS selector (relative to result container) for the title text.
url-selectorstringNo""CSS selector for the URL. Extracts href attribute first; falls back to text content.
content-selectorstringNo""CSS selector for the snippet/description text.
thumbnail-selectorstringNo""CSS selector for a thumbnail image URL (used for image/video results).

Response parsing – JSON mode

Used when result-type is "json".

AttributeTypeRequiredDefaultDescription
result-typestringYesSet to "json" for dot-path-based JSON parsing.
results-pathstringNo"" (top-level array)Dot-separated path to the results array in the JSON response. Example: "message.items" navigates {"message":{"items":[...]}}.
title-pathstringNo""Dot-path to the title field within each result object.
url-pathstringNo""Dot-path to the URL field.
content-pathstringNo""Dot-path to the snippet/description field.
thumbnail-pathstringNo""Dot-path to a thumbnail URL field.

URL construction

AttributeTypeRequiredDefaultDescription
url-prefixstringNo""String prepended to every extracted URL. Use for APIs that return relative paths (e.g. "https://developer.mozilla.org" + /en-US/docs/...).
url-templatestringNo""Template for building URLs from result fields. Placeholders: {title} (spaces replaced with _), {id}, {url}. Takes precedence over url-prefix when set. Example: "https://www.artic.edu/artworks/{id}".

Pagination

AttributeTypeRequiredDefaultDescription
page-sizeintNo10Results per page. Used to compute {offset}.
pagingboolNofalseWhether this engine supports pagination.
first-pageintNo0The page/offset number for the first page. Most APIs use 0 or 1.

Content processing

AttributeTypeRequiredDefaultDescription
content-html-to-textboolNofalseStrip HTML tags from the content/snippet field before display. Useful for APIs like StackExchange that return HTML in JSON.

Lifecycle

AttributeTypeRequiredDefaultDescription
disabledboolNofalseIf true, the engine is parsed but not registered. Use to temporarily disable an engine without removing its config.

Federation

AttributeTypeRequiredDefaultDescription
sharestringNoauto-detectedControls whether results from this engine are shared with peer instances. Values: "none" (never share – used for engines with API keys to avoid leaking quota), "cache" (share cached results but don’t proxy live queries – the default), "proxy" (full proxy: friends can trigger live queries through us). When omitted, the engine factory auto-detects: if the URL or headers contain an API key pattern (api_key=, Authorization, etc.), defaults to "none"; otherwise defaults to "cache".
tor-outboundboolNofalseRoute this engine’s HTTP requests through the local Tor SOCKS5 proxy (127.0.0.1:9050). Use for engines that aggressively IP-ban search aggregators. Requires a running Tor daemon.

Examples

JSON API engine (MDN)

engine "mdn" {
    name "MDN Web Docs"
    categories "web"
    search-url "https://developer.mozilla.org/api/v1/search?q={query}&page={page}"
    method "GET"
    result-type "json"
    results-path "documents"
    title-path "title"
    url-path "mdn_url"
    url-prefix "https://developer.mozilla.org"
    content-path "summary"
    page-size 10
    paging true
    first-page 1
}

HTML scraping engine (Anaconda)

engine "anaconda" {
    name "Anaconda"
    categories "web"
    search-url "https://anaconda.org/search?q={query}&page={page}"
    method "GET"
    result-type "html"
    url-prefix "https://anaconda.org"
    results-selector "tbody tr"
    title-selector "td h5"
    url-selector "td h5 a:last-of-type"
    content-selector "td p"
    page-size 10
    paging true
    first-page 0
}

Engine with custom headers (Crossref)

engine "crossref" {
    name "Crossref"
    categories "web"
    search-url "https://api.crossref.org/works?query={query}&rows=20&offset={offset}"
    method "GET"
    result-type "json"
    results-path "message.items"
    title-path "title"
    url-path "URL"
    content-path "abstract"
    content-html-to-text true
    headers {
        "User-Agent" "4got/1.0 (https://4got.org; mailto:admin@4got.org)"
    }
    page-size 20
    paging true
    first-page 0
}

POST engine with body template (ArtStation)

engine "artstation" {
    name "www.artstation.com"
    categories "images"
    search-url "https://www.artstation.com/api/v2/search/projects.json"
    method "POST"
    result-type "json"
    results-path "data"
    title-path "title"
    url-path "permalink"
    content-path "description"
    thumbnail-path "smaller_square_cover_url"
    body-template "{\"query\":\"{query}\",\"page\":{page},\"per_page\":20,\"sorting\":\"relevance\",\"pro_first\":\"1\",\"filters\":[],\"additional_fields\":[\"description\"]}"
    page-size 20
    paging true
    first-page 1
}

URL template engine (Art Institute of Chicago)

engine "artic" {
    name "www.artic.edu"
    categories "images"
    search-url "https://api.artic.edu/api/v1/artworks/search?q={query}&page={page}&limit=20"
    method "GET"
    result-type "json"
    results-path "data"
    title-path "title"
    url-path "id"
    url-template "https://www.artic.edu/artworks/{id}"
    content-path "thumbnail.alt_text"
    page-size 20
    paging true
    first-page 1
}

How the generic engine factory works

At startup, LoadAndRegisterGenericEngines() in generic_engine.go:

  1. Reads data/engines.kdl via resolveKDLSection() (which supports both standalone files and engines { ... } sections inside a unified KDL file).
  2. Parses the raw bytes into []GenericEngineConfig via parseEngineKDL(), a line-by-line KDL parser that handles engine blocks, headers sub-blocks, and all property types.
  3. Skips entries with empty ID, no categories, or disabled true.
  4. Sets defaults: page-size defaults to 10, name defaults to the ID.
  5. Creates a GenericEngine struct (which implements the Engine interface) for each config and registers it in the global engine registry.
  6. Determines the share policy: explicit share attribute wins; otherwise auto-detects based on whether the URL or headers contain API key patterns.

When Search() is called on a GenericEngine:

  1. Substitutes {query}, {page}, {offset}, {lang} in the URL template (and body template if POST).
  2. Makes the HTTP request, routing through Tor if tor-outbound is set.
  3. Records the raw HTML for drift detection (drift.go).
  4. Dispatches to parseHTML() (goquery CSS selectors) or parseJSON() (dot-path navigation) based on result-type.
  5. Returns []WebResult with each result’s Engine field set to the engine ID.

Testing a new engine via the admin playground

The admin playground at /admin/engines/new (owner-only) provides a web form for configuring and testing generic engines without editing KDL files directly.

  1. Log in as owner at /admin/login.
  2. Navigate to /admin/engines/new.
  3. Fill in the engine identity (ID, display name, categories).
  4. Set the search URL template with {query} and {page} placeholders.
  5. Choose HTML or JSON result type and fill in the corresponding selectors/paths.
  6. Enter a test query and click “Test Engine” – this POSTs to /admin/engines/test, which creates a temporary GenericEngine in memory, runs a search, and returns results as JSON displayed in the form.
  7. Iterate on selectors until results look correct.
  8. Click “Save Engine” to append the generated KDL block to data/engines.kdl and register it in memory immediately (no restart needed).

The test endpoint does not persist anything – only “Save Engine” writes to disk.

Engine Catalog

4got ships with 274 engine configurations in data/engines.kdl plus 18 handwritten engines in Go. Engines are grouped by category. Most require no API key; exceptions are noted.

Disabled engines are included but inactive by default. Enable them by setting disabled false in engines.kdl.

Handwritten engines (Go)

These are implemented directly in Go for speed, reliability, or because they need custom scraping logic.

EngineIDCategoriesNotes
Bravebraveweb, images, newsHTML scraping
DuckDuckGoduckduckgowebHTML scraping
GooglegooglewebHTML scraping
StartpagestartpagewebHTML scraping
WikipediawikipediawebMediaWiki API
MojeekmojeekwebHTML scraping
MarginaliamarginaliawebJSON API
WibywibywebJSON API
Hacker NewshackernewsnewsAlgolia API
YouTubeyoutubevideosHTML scraping
SoundCloudsoundcloudmusicJSON API
FlickrflickrimagesJSON API
PexelspexelsimagesJSON API
Yandexyandexweb, imagesHTML scraping
Qwantqwantweb, images, newsJSON API
Bingbingweb, images, news, videosHTML scraping
BaidubaiduwebHTML scraping
Yahooyahooweb, newsHTML scraping

Sidecar engines (Python + Playwright)

These require the Python sidecar for JavaScript rendering:

EngineWhat it handles
Google (sidecar)JS-heavy Google SERPs
Qwant (sidecar)Qwant’s SPA rendering
Yandex (sidecar)Yandex captcha bypass

KDL-configured engines by category

Web (159 engines)

General web search. The largest category.

EngineIDNotes
MDN Web DocsmdnJSON API
StackExchangestackexchangeAPI, Q&A results
Open LibraryopenlibBook search
npmnpmPackage registry
AnacondaanacondaHTML scraping
Arch WikiarchwikiREST API
WikidatawikidataJSON API
CrossrefcrossrefAcademic DOIs
Apple App Storeapple_app_storeiTunes API
BoardreaderboardreaderForum search
bpb.debpbGerman civic education
CachyOScachy_os
crates.iocratesRust packages
Docker Hubdocker_hub
GitHubgithubRepo search
GitHub Codegithub_codeCode search
GitHub Issuesgithub_issues
GitHub Topicsgithub_topics
GitLabgitlab
GitLab Snippetsgitlab_snippets
Grokipediagrokipedia
Hacker NewshackernewsAlgolia API
hex.pmhexElixir packages
Hugging FacehuggingfaceAI models
LemmylemmyFediverse forums
MastodonmastodonFediverse posts
Microsoft Learnmicrosoft_learn
mwmblmwmblOpen-source index
NVDnvdCVE/vulnerability DB
OpenAlexopenalexAcademic
OpenLibraryopenlibrary
Pirate BaypiratebayTorrents
Redditreddit
tagesschau.detagesschauGerman news (web+news)
TootfindertootfinderMastodon search
Void Linuxvoidlinux
Wikipediawikipedia
Simple Wikipediasimple_wikipedia
Wikibookswikibooks
Wikinewswikinews
Wikiquotewikiquote
Wikisourcewikisource
Wikivoyagewikivoyage
Wiktionarywiktionary
Wikimedia Commonswikicommons
Internet Archiveinternet_archive
LibreTranslatelibre_translate
PubMedpubmed
RubyGemsrubygems
CTANctanLaTeX packages
cdnjscdnjsJS CDN
pub.devpub_devDart packages
CocoaPodscocoapodsiOS packages
OpenBSD Portsopenbsd_ports
TVmazetvmazeTV shows
TMDBtmdbMovies
OpenLibrary Booksopenlibrary_search
LibriVoxlibrivoxAudiobooks
Project GutenberggutenbergFree ebooks
GiphygiphyGIFs; uses public API key
OpenStreetMapopenstreetmapNominatim
PhotonphotonGeocoder
Open-Meteoopen_meteoWeather data
Alpine Packagesalpine_packages
HomebrewhomebrewDisabled
Flathubflathub
Snapcraftsnapcraft
YTSytsMovie torrents
Discogsdiscogs
Last.fmlastfm
Urban Dictionaryurban_dictionary
DDG Definitionsduckduckgo_definitions
SVGReposvgrepo
SteamsteamGames
PyPIpypiPython packages
1337x1337xTorrents
APKMirrorapkmirrorAndroid APKs
eBayebay
F-DroidfdroidFOSS Android apps
Solid Torrentssolidtorrents
WordnikwordnikDictionary
fyydfyydPodcasts; disabled
Podcast Indexpodcast_index
Semantic Scholarsemantic_scholarAcademic
SourceHutsourcehut
Giteagitea
lib.rslibrs / lib_rsRust crate search
Lobsterslobste_rs
PackagistpackagistPHP packages
Radio Browserradio_browser
Dailymotiondailymotion
Bandcampbandcamp
Stack Overflow Tagsstackoverflow_main
Anaconda Cloudanaconda_cloud
arXivarxivAcademic preprints
DBLPdblpCS bibliography
IA Scholarinternet_archive_scholar
NyaanyaaAnime torrents
Library Genesislibgen
Codebergcodeberg
pkg.go.devpkg_go_devGo packages
HackagehackageHaskell packages
NuGetnuget.NET packages
MavenmavenJava packages
pkgs.orgpkgs_orgCross-distro package search
Libraries.iolibraries_ioMulti-registry
Anna’s Archiveannas_archive
Z-Libraryz_library
Metacriticmetacritic
Rotten Tomatoesrotten_tomatoes
Letterboxdletterboxd
AniDBanidb
VNDBvndbVisual novels
PCGamingWikipcgamingwiki
IsThereAnyDealisthereanydealGame prices
BoardGameGeekboardgamegeek
WordReferencewordreference
Lingueelinguee
DeepLdeeplDisabled
Brave Gogglesbrave_goggles
SearXsearxMeta-meta-search
Whooglewhoogle
Stractstract
Yepyep
Rightdaoright_dao
Presearchpresearch
Alexandriaalexandria
Petal Searchpetalsearch
Swisscowsswisscows
MetaGermetager
eToolsetools
Dogpiledogpile
Gibirugibiru
searx.spacesearx_spaceInstance list
Monoclesmonocles
Mullvad Letamullvad_leta
4get.ca4get_caSibling metasearch
Emojipediaemojipedia
Color Hexcolorhex
AlternativeToalternativeto
Goodreadsgoodreads
IMDBimdb_search
itch.ioitch_ioIndie games
Are.naare_na
Kickstarterkickstarter
Product Huntproduct_hunt
IEEE Xploreieee
Springerspringer_open
COREcoreAcademic
DOAJdoajOpen access journals
ZenodozenodoResearch data
ORCIDorcid
UnpaywallunpaywallOpen access finder
OSFosfPreprints
ANN AnimeannDisabled
KitsukitsuAnime/manga
MyAnimeListjikanVia Jikan API
RAWGrawgGames
MangaDexmangadex
Open Food Factsopenfoodfacts
MusicBrainzmusicbrainz
OpenAIREopenairedatasets
Exchange Rateexchangerate
EcosiaecosiaDisabled
MojeekmojeekDisabled
MarginaliamarginaliaDisabled
Etsyetsy
CRANcranR packages; disabled
CPANcpanPerl packages
Conda-Forgeconda_forgeDisabled
RepologyrepologyDisabled
HowLongToBeathowlongtobeatDisabled
IGDBigdbDisabled
BASE Academicbase_search
Fatcat Scholarfatcat
Google Scholargoogle_scholarDisabled

Images (15 engines)

EngineIDNotes
Art Institute of Chicagoartic
ArtStationartstationPOST API
findthatmemefindthatmemeMeme search
OpenverseopenverseCC-licensed
Pixabaypixabay
Adobe Stockadobe_stock
Unsplashunsplash
Wallhavenwallhaven
Pexelspexels
Flickrflickr
Wikimedia Commonswikicommons_images
Giphygiphy
SVGReposvgrepoVector graphics
DeviantArtdeviantartDisabled
PixivpixivDisabled

Videos (13 engines)

EngineIDNotes
BilibilibilibiliChinese video
BitChutebitchute
CCC Mediaccc_mediaChaos Computer Club
Dailymotiondailymotion
IMDbimdb
Internet Archive Videoarchive_video
InvidiousinvidiousYouTube frontend
OdyseeodyseeLBRY
PeerTubepeertubeFediverse video
Vimeovimeo
YouTube (no API)youtube_noapiDisabled
RumblerumbleDisabled
360kan360search_videosDisabled

News (10 engines)

EngineIDNotes
Yahoo Newsyahoo_news
Hacker Newshackernews_news
Reutersreuters
tagesschau.detagesschauGerman; also in web
Wikinewswikinews
The Guardianthe_guardianUses test API key
ANSA NewsansaItalian wire service
AP Newsap_newsDisabled
Bing Newsbing_newsDisabled
Google Newsgoogle_newsDisabled

Music (13 engines)

EngineIDNotes
Bandcampbandcamp
Deezerdeezer
Discogsdiscogs
Geniusgenius
JamendojamendoCC music
Last.fmlastfm
LibriVoxlibrivoxAudiobooks
Mixcloudmixcloud
MusicBrainzmusicbrainz
Radio Browserradio_browserInternet radio
SoundCloudsoundcloud
YouTube (no API)youtube_noapiAlso videos; disabled
FreesoundfreesoundDisabled; needs API key

Science (12 engines)

EngineIDNotes
arXivarxiv
BASEbase_search_sciBielefeld Academic
COREcore_sci
Crossrefcrossref_sci
dblpdblpCS papers
DOAJdoaj_sciOpen access
IA Scholarinternet_archive_scholar
OpenAlexopenalex_sci
PubMedpubmed_sci
Semantic Scholarsemantic_scholar_sci
Zenodozenodo_sci
Google Scholargoogle_scholarDisabled

IT (15 engines)

EngineIDNotes
Codebergcodeberg
crates.iocrates_ioRust
Docker Hubdocker_hub_it
GitHub Reposgithub_repos_it
GitLabgitlab_it
hex.pmhex_itElixir
MDNmdn_it
npmnpm_search
NuGetnuget_it.NET
Packagistpackagist_itPHP
pkg.go.devpkg_go_devGo
pub.devpub_dev_itDart
PyPIpypi_packagesPython
RubyGemsrubygems_it
Stack Overflowstackoverflow

Files (9 engines)

EngineIDNotes
1337x1337x_files
Anna’s Archiveannas_archive_files
Library Genesislibgen
NyaanyaaAnime torrents
Pirate Baypiratebay_files
Solid Torrentssolidtorrents_files
YTSyts_filesMovie torrents
BT4Gbt4gDisabled
TorrentGalaxytorrentgalaxyDisabled

Social (6 engines)

EngineIDNotes
Hacker Newshackernews_social
Lemmylemmy_social
Lobsterslobsters_social
Mastodonmastodon_social
Redditreddit_social
Tootfindertootfinder_social

Books (4 engines)

EngineIDNotes
Goodreadsgoodreads_books
Internet Archiveinternet_archive_books
Open Libraryopenlibrary_books
Project Gutenberggutenberg_books

Q&A (6 engines)

EngineIDNotes
Ask Ubuntuaskubuntu_qa
Math SEmath_se_qa
Server Faultserverfault_qa
StackExchangestackexchange_qa
Super Usersuperuser_qa
Unix & Linux SEunix_se_qa

Packages (9 engines)

EngineIDNotes
Alpine Packagesalpine_packages_pkg
Arch Wikiarchwiki_packages
AURaur
Flathubflathub_pkg
Gentoo Wikigentoo_wiki
NixOS Packagesnixos_packages
Void Linuxvoid_packages_pkg
Homebrewhomebrew_pkgDisabled
RepologyrepologyDisabled

Maps (2 engines)

EngineIDNotes
OpenStreetMap (Nominatim)nominatim_maps
Photon Geocoderphoton_maps

Weather (2 engines)

EngineIDNotes
wttr.inwttr_in
Open-Meteoopen_meteo_weather

Lyrics (1 engine)

EngineIDNotes
Genius Lyricsgenius_lyrics

Oracles

Oracles are instant-answer widgets shown above search results. They trigger on specific query patterns and provide rich, interactive answers without requiring a click-through. Oracles are checked in CheckOracles() in oracles.go. Access can be restricted per-group via oracle-permissions.kdl.

Local oracles (no external calls)

These run entirely on the server with no network requests. They are mutually exclusive — the first match wins and skips all remaining oracles.

OraclePermission keyTrigger patternsWhat it shows
Calculatorcalculator2+2, sqrt(16), 15% of 200, sin(1.5), 2^8Arithmetic, trig, logarithms, percentage calculations
Unit Converterunit_converter10 km to miles, 72 fahrenheit to celsius, 5 gallons to liters, 100 fl oz to mlLength, weight, volume, speed, data storage, cooking units, temperature, force/mass
Base Converterbase_converter0xff in decimal, 255 in hex, 255 in binary, 0b1010 in decimal, 0o77 in hexDecimal, hex, binary, octal conversions
Colorcolor#FF5500, #f50, rgb(255,85,0), hsl(20,100%,50%)Color swatch with hex/RGB/HSL conversions
Private IPprivate_ipBare IPv4/IPv6 addresses in private ranges (192.168.1.1, 10.0.0.1)Identifies the private range and offers HTTP/HTTPS links
Timezonetimezonetime in tokyo, time in new york, time in UTC+5Current time in 70+ cities and IANA timezones
UUIDuuidAny UUID string (e.g. 550e8400-e29b-41d4-a716-446655440000)Version, variant, and embedded timestamp (v1/v7)
Timestamptimestamp10-digit or 13-digit numbers, now, current timestamp, unix timestampConverts Unix timestamps to human dates, or shows current timestamp
JWTjwtAny header.payload.signature JWT tokenDecodes header and payload, shows algorithm and expiry
CIDRcidr192.168.1.0/24, 10.0.0.0/8Network, broadcast, host range, usable hosts, netmask
Hash/Base64hashMD5 (32 hex), SHA-1 (40 hex), SHA-256 (64 hex), SHA-512 (128 hex), or base64 strings (40+ chars)Identifies hash type, or decodes base64 to text
Croncron5-field cron expressions like 0 */2 * * 1-5Human-readable schedule description with per-field explanation
DOIdoi10.1000/xyz123 or doi.org/10.1000/xyz123Links to publisher (doi.org) and Sci-Hub
Timertimertimer 5 minutes, timer 30s, countdown 10m, stopwatchInteractive countdown timer or stopwatch widget
Randomrandomflip a coin, roll a die, roll 3d20, random 1 to 100Coin flip, dice rolls (NdX), random number in range (crypto/rand)
Passwordpasswordpassword, generate password, password 32 charsCryptographically random password with copy button (4-256 chars)
QR Codeqrqr https://example.com, qr code hello worldInline SVG QR code (up to ~271 bytes, versions 1-10)
Chmodchmodchmod 755, chmod rwxr-xr-xNumeric-to-symbolic and symbolic-to-numeric conversion with explanation
Roman NumeralromanXIV in decimal, 42 in romanRoman numeral to/from decimal conversion (1-3999)
Date Arithmeticdate_arithmeticdays between Jan 1 and Dec 31, 30 days from now, days until Christmas, 90 days agoDay counts, future/past dates, with IFC calendar dates

Network oracles (external API calls, no key required)

These make external network requests but do not require any API key.

OraclePermission keyTrigger patternsWhat it showsAPI used
Reverse DNSreverse_dnsrdns 8.8.8.8, reverse dns 1.1.1.1, nslookup 8.8.4.4, ptr <IP>PTR records for an IP addressSystem DNS resolver
DNS Lookupdns_digexample.com, dig example.com, dns example.comA, AAAA, MX, TXT recordsSystem DNS resolver
IP Geolocationip_geoBare public IPv4 like 8.8.8.8, where is 1.1.1.1, geolocate <IP>City, region, country, ISP, ASN, PTRipapi.co
My IPmy_ipmy ip, whats my ip, ip, show my ipUser’s public IP with geolocation and reverse DNSipapi.co
Currencycurrency100 usd to eur, 50 gbp in jpy, usd to eurExchange rate with forward and inverse ratesfawazahmed0/exchange-api (jsDelivr CDN)
WHOISwhoiswhois example.comDomain registrar, creation/expiry dates, nameserversRDAP (rdap.org)
Speedtestspeedtestspeedtest, speed test, internet speed, bandwidth testInteractive download/upload speed test with latency and jitterServer’s own /api/v1/speedtest/* endpoints
Benchmarkbenchmarkbenchmark, browser benchmark, system benchmarkJS/DOM/Crypto/Canvas/Memory benchmark with scores; includes speedtest if permittedClient-side JS + server speedtest
Translationtranslationtranslate hello to french, translate X to YText translation with language detectionConfigured translation backends
Auto-translatetranslation(Automatic on non-native-language queries)Detects foreign language queries and translates to user’s known languageConfigured translation backends
Mapmapmap of paris, map tokyo, directions to berlinOpenStreetMap tile with links to OSM, Apple, Google, HERE, Bing, OsmAnd, Waze, Yandex MapsNominatim geocoding + OSM tiles
WikipediawikipediaEntity-like queries (1-3 capitalized words), who is X, what is XInfobox card with image, summary, and Wikipedia linkWikipedia REST API
DDG Instant Answersddg_ia(Any query — lowest priority among knowledge oracles)Abstract, definition, or answer card from DuckDuckGoDuckDuckGo Instant Answer API
Package Versionpackageslatest lodash, npm lodash, pypi requests, pip flask versionLatest version with link to registrynpm Registry / PyPI
Radioradioradio new york, fm tokyo, radio stations in berlinUp to 5 radio stations with inline audio playersradio-browser.info
Sports Scoressportsnfl scores, nba scores, lakers scores, premier league scoresLive/recent scores for NFL, NBA, MLB, NHL, MLS, EPL, La Liga, Bundesliga, Serie A, Ligue 1, Champions League, WNBA, NCAAESPN API
SSL Certificatessl_certssl example.com, cert example.com, certificate example.comCertificate issuer, validity dates, SANs, protocol version, cipher suiteDirect TLS connection
Earthquakeearthquakeearthquakes, earthquakes today, recent earthquakes, earthquakes in californiaRecent significant earthquakes with magnitude, location, depth, and timeUSGS Earthquake API
Flightsflightsflight UA123, flight status AA456, flights from JFK to LAXLive flight tracking by callsign; route planning with bearing and distanceOpenSky Network API
Recipereciperecipe for pasta, how to make brownies, how to cook risottoStep-by-step recipe with ingredients, thumbnail, and source linkTheMealDB

API-keyed oracles (require configuration)

These oracles require API keys configured in config.kdl under api-keys { ... }. Without the key, the oracle silently returns nothing.

OraclePermission keyTrigger patternsWhat it showsAPI key configAPI used
Wolfram Alphawolfram(Any query — lowest priority knowledge oracle)Computational answers, facts, mathwolframalpha "..."Wolfram Alpha Short Answers API
Weatherweatherweather, weather in paris, forecast tokyo, temperature in londonCurrent temperature, humidity, wind, conditions with iconOptional: openweathermap "..."Open-Meteo (free, always); OpenWeatherMap (if key set, tabbed multi-source)
Dictionarydictionarydefine serendipity, meaning of ubiquitous, what does ephemeral mean, urban slangMulti-source tabbed definitionsOptional: mw_collegiate, mw_thesaurus, mw_learners, mw_medical, mw_spanish, mw_school, mw_elementary, mw_intermediate, mw_intermediate_thesaurus, wordnikFree Dictionary API + Wiktionary + Urban Dictionary (always); up to 9 Merriam-Webster products + Wordnik (with keys)
Stockstock$AAPL, TSLA stock, stock price MSFTPrice, change, open/high/low, previous closeOptional: alpha_vantage "...", twelvedata "...", finnhub "..."Alpha Vantage, Twelve Data, Finnhub (multi-source tabbed if multiple keys)
Cryptocrypto$BTC, bitcoin price, price of ethereum, crypto solanaUSD/EUR/BTC price, 24h change, market capNone (free API)CoinGecko
Inflation / CPIinflationinflation, cpi, consumer price index, dollar inflationAnnual inflation rate with 12-month sparkline chartNone (free API)Bureau of Labor Statistics
Purchasing Powerinflationpurchasing power of dollar, dollar worth in 1970Historical purchasing power table or year-specific comparisonNone (free API)Bureau of Labor Statistics CPI-U data
Air Qualityair_qualityair quality, aqi, pollution, air quality in beijingAQI value, category, dominant pollutant, color-coded displayOptional: aqicn "..."Open-Meteo Air Quality (free, always); AQICN (with key, tabbed multi-source)
Newsnewsnews about AI, latest news on climate, headlines technologyMulti-source news headlines with thumbnailsOptional: currentsapi "...", newsdata "...", gnews "..."Google News RSS (free, always); Currents API, Newsdata.io, GNews (with keys, tabbed)
Music / Spotifymusicplay bohemian rhapsody, listen to radiohead, song despacito, Artist — TrackSpotify embed player with album info and alternative service linksOptional: spotify-client-id "..." + spotify-client-secret "..."Spotify Web API (with keys); links-only fallback without keys

Knowledge oracles (fire concurrently)

Three knowledge oracles fire simultaneously on every query and the best result wins. Priority order: Wikipedia > DDG Instant Answers > Wolfram Alpha.

PriorityOraclePermission keyAPI
0 (highest)Wikipedia InfoboxwikipediaWikipedia REST API (no key)
1DDG Instant Answersddg_iaDuckDuckGo IA API (no key)
2 (lowest)Wolfram AlphawolframWolfram Alpha Short Answers API (requires key)

Smart tab oracles

When smart_tab_oracles is enabled (default: on), certain oracles auto-fire on category-specific tabs even without explicit trigger words:

TabAuto-fired oracles
MapsMap + Weather (for the displayed location), IP Geolocation
MusicMusic (with forced “play” prefix), Radio
ScienceCalculator, Unit Converter
ITMy IP, DNS Lookup, CIDR Calculator, Base Converter
WeatherWeather (with “weather” prefix)

AI Research tools

When AI mode is enabled (lite or full), the AI research agent has access to these tools. These are not query-triggered oracles but tools the AI can invoke during research:

ToolDescription
fetchFetches and extracts text from URLs; supports depth=2 for same-domain crawling
psychoscrapeCitation-aware web scraper that extracts structured claims with source attribution
youtube_transcriptExtracts timestamped transcripts/captions from YouTube videos
luaSandboxed Lua execution environment with HTTP, JSON, math, and string libraries
sendShows intermediate findings to the user during research
etaUpdates the user-facing ETA countdown
frictionLogs improvement suggestions for the server developer
pinPins a result to the top of the search results
forgetRemoves a result from the search results
friend_searchSearches federated peer instances for additional results

API keys

Most engines scrape public pages and need no key. Engines that benefit from or require API keys:

ServiceConfig keyRequired?
Wolfram Alphaapi-keys { wolframalpha "..." }Optional; enables Wolfram oracle
DeepLapi-keys { deepl "..." }Optional; translation backend
Freesoundapi-keys { freesound "..." }Required for freesound engine
Spotifyapi-keys { spotify-client-id / spotify-client-secret }Optional; enables Spotify embed in music oracle
OpenWeatherMapapi-keys { openweathermap "..." }Optional; adds second weather source
Alpha Vantageapi-keys { alpha_vantage "..." }Optional; stock quotes
Twelve Dataapi-keys { twelvedata "..." }Optional; stock quotes
Finnhubapi-keys { finnhub "..." }Optional; stock quotes
AQICNapi-keys { aqicn "..." }Optional; adds second air quality source
Currents APIapi-keys { currentsapi "..." }Optional; news oracle source
Newsdata.ioapi-keys { newsdata "..." }Optional; news oracle source
GNewsapi-keys { gnews "..." }Optional; news oracle source
MW Collegiateapi-keys { mw_collegiate "..." }Optional; dictionary source
MW Thesaurusapi-keys { mw_thesaurus "..." }Optional; dictionary source (synonyms/antonyms)
MW Learner’sapi-keys { mw_learners "..." }Optional; dictionary source
MW Medicalapi-keys { mw_medical "..." }Optional; dictionary source
MW Spanishapi-keys { mw_spanish "..." }Optional; bilingual dictionary source
MW Schoolapi-keys { mw_school "..." }Optional; dictionary source
MW Elementaryapi-keys { mw_elementary "..." }Optional; dictionary source
MW Intermediateapi-keys { mw_intermediate "..." }Optional; dictionary source
MW Int. Thesaurusapi-keys { mw_intermediate_thesaurus "..." }Optional; dictionary source
Wordnikapi-keys { wordnik "..." }Optional; dictionary source
GiphyHardcoded public keyNone needed
The GuardianHardcoded test keyNone needed
PexelsPer-engine header in engines.kdlOptional

Adding a new engine

Add an engine block to data/engines.kdl:

engine "my_engine" {
    name "My Engine"
    categories "web"
    search-url "https://api.example.com/search?q={query}&page={page}"
    method "GET"
    result-type "json"
    results-path "results"
    title-path "title"
    url-path "url"
    content-path "snippet"
    page-size 20
    paging true
    first-page 1
}

Restart 4got. The engine registers automatically.

Template variables: {query}, {page}/{pageno}, {offset}, {lang}.

For HTML scraping, set result-type "html" and use CSS selectors (results-selector, title-selector, url-selector, content-selector).

Architecture

Request flow

User query
    │
    ▼
┌─────────────────────────────────────────────────────────┐
│ handleSearch()                                          │
│                                                         │
│  1. Parse bangs ──── !g, !w, etc. ──→ 302 redirect     │
│     │ (no match)                                        │
│     ▼                                                   │
│  2. Auth check                                          │
│     │                                                   │
│     ├── No session, first visit ──→ /captcha (IQ test)  │
│     ├── No session, returning ───→ PoW challenge page   │
│     │                              (client-side mining)  │
│     ▼                                                   │
│  3. Parse search params (query, category, cookies)      │
│     │                                                   │
│     ▼                                                   │
│  4. Local cache lookup ── hit ──→ skip to step 8        │
│     │ (miss)                                            │
│     ▼                                                   │
│  5. Friend cache lookup ── hit ──→ store locally,       │
│     │ (miss)                        skip to step 8      │
│     ▼                                                   │
│  6. Fanout (singleflight-coalesced)                     │
│     ├── Engine 1 ──→ goroutine ──→ HTTP scrape/API      │
│     ├── Engine 2 ──→ goroutine ──→ HTTP scrape/API      │
│     ├── Engine N ──→ goroutine ──→ HTTP scrape/API      │
│     └── Friends  ──→ goroutine ──→ peer cache query     │
│     │                                                   │
│     │  hard window (1000ms default) or                  │
│     │  idle timeout (500ms) with must-include gate       │
│     ▼                                                   │
│  7. Merge + deduplicate + rank                          │
│     ├── URL normalization + dedup                       │
│     ├── Cross-engine consensus scoring                  │
│     ├── Per-engine weight × position decay              │
│     ├── Intent-based boost (code, academic, images...)  │
│     ├── Tracking param stripping                        │
│     └── Store merged results in cache                   │
│     │                                                   │
│     ▼                                                   │
│  8. Render                                              │
│     ├── Check oracles (calculator, unit, timezone, etc.)│
│     ├── URL rewriting (YouTube→Invidious, etc.)         │
│     ├── Auto-translate if foreign query detected        │
│     ├── Log search (if history enabled)                 │
│     └── Render HTML template with results + timeline    │
│                                                         │
│  9. Push to peer caches (background goroutine)          │
└─────────────────────────────────────────────────────────┘

Auth flow

New visitor                 Returning visitor
    │                            │
    ▼                            ▼
┌────────────┐           ┌─────────────┐
│ IQ Test    │           │ PoW WASM    │
│ (captcha)  │           │ (SHA-256    │
│ Image grid │           │  mining)    │
└────┬───────┘           └──────┬──────┘
     │ correct                  │ valid nonce
     ▼                          ▼
┌──────────────────────────────────────┐
│ Ed25519-signed session cookie        │
│ (AuthCaptcha or AuthPow)             │
│ TTL: captcha=24h, pow=5min           │
└──────────────────────────────────────┘

Owner: POST /admin/login with owner_secret
       → AuthOwner session cookie

Groups: cookie hash → group lookup (groups.kdl)
        → permissions: sidecar, server_api_keys,
          friend_proxy, priority_fanout

Federation protocol

Instance A                         Instance B
    │                                   │
    │  GET /api/v1/cache               │
    │  ?q=QUERY&cat=web                │
    │  Authorization: Bearer PEER_SEC  │
    │ ─────────────────────────────→   │
    │                                   │
    │  200 OK (cached results)          │
    │  or 204 No Content               │
    │ ←─────────────────────────────   │
    │                                   │
    │  POST /api/v1/cache              │
    │  (push fresh results to friend)  │
    │ ─────────────────────────────→   │
    │                                   │
    │  POST /api/v1/proxy              │
    │  {"service":"translation",       │
    │   "params":{...}}                │
    │ ─────────────────────────────→   │
    │                                   │
    │  200 OK (translation result)     │
    │ ←─────────────────────────────   │
    │                                   │
    │  GET /ami4got                    │
    │ ─────────────────────────────→   │
    │  {"version":"2.0",               │
    │   "engine_count":15, ...}        │
    │ ←─────────────────────────────   │

Proxyable services: translation, wolfram, autocomplete
Auth: shared peer_secret as Bearer token
Wait policy: configurable (all friends, N friends, miss-ok)
Failure tracking: auto-mark down at threshold (e.g. 1% over 1d)

AI research agent loop

User searches same query twice
         │
         ▼
GET /api/v1/research?q=QUERY  (SSE stream)
         │
         ▼
┌─────────────────────────────────────────┐
│ RunAIResearch()                         │
│                                         │
│  System prompt (data/metaprompt.md)     │
│  + top search result URLs               │
│         │                               │
│         ▼                               │
│  LLM generates tool calls:             │
│  ┌──────────────────────────────────┐   │
│  │ fetch(url)  → HTTP GET + strip   │   │
│  │ lua(code)   → sandboxed Lua VM   │   │
│  │ send(text)  → SSE "message"      │   │
│  │ eta(s,job)  → SSE "eta"          │   │
│  │ pin(text)   → permanent context  │   │
│  │ forget(id)  → evict old result   │   │
│  │ friction(s) → operator feedback  │   │
│  └──────────────────────────────────┘   │
│         │                               │
│         │ (loop until done or timeout)  │
│         ▼                               │
│  SSE "done" event with final answer     │
│                                         │
│  Cache result for replay               │
└─────────────────────────────────────────┘

Provider: any OpenAI-compatible API
          (Cerebras, OpenRouter, Ollama, custom)
Sandbox:  Lua (recommended) or bash (containers only)
Timeout:  max-runtime-seconds (default 60)
MCP:      optional external tool servers via stdio/HTTP

File organization

Go server (go-server/)

FileLinesPurpose
main.go503Entry point, config loading, startup sequence
routes.go364HTTP route registration table
handlers_search.go455Search and landing page handlers
handlers_settings.go281Settings page GET/POST
handlers_auth.go280Captcha and PoW handlers
handlers_admin.go~200Admin log and login
handlers_api.go~150JSON API endpoint
handlers_misc.go~100About, health, robots
admin_users.go210User group management
engine.go483Engine interface, types, registry, HTTP client
generic_engine.go453KDL-driven engine factory
engine_brave.go377Brave scraper
engine_google.go301Google scraper
engine_bing.go405Bing scraper
engine_yahoo.go351Yahoo scraper
engine_baidu.go307Baidu scraper
engine_duckduckgo.go230DDG scraper
engine_startpage.go~170Startpage scraper
engine_yandex.go~170Yandex scraper
engine_qwant.go~150Qwant API client
engine_wikipedia.go~120MediaWiki API
engine_youtube.go~130YouTube scraper
engine_hackernews.go~100HN Algolia API
engine_marginalia.go~100Marginalia API
engine_mojeek.go~100Mojeek scraper
engine_wiby.go~80Wiby API
engine_flickr.go~80Flickr API
engine_pexels.go~80Pexels API
engine_soundcloud.go~100SoundCloud API
fanout.go988Concurrent engine dispatch, scoring, dedup, merge
cache.go346In-memory LRU + TTL cache
peercache.go249Friend cache query/push
peers.go268Peer config loading, failure tracking
friendproxy.go327Service proxy between friends
auth.go316Ed25519 sessions, PoW verification
captcha_prerender.go688Pre-rendered captcha pool (IQ test grid)
captcha_image.go307Captcha image compositing
oracles.go893Calculator, unit, timezone, base, translate, DDG IA, Wolfram
oracles_extra.go852Currency, IP geo, DNS, color, packages, radio
airesearch.go1917AI agent: LLM loop, tool dispatch, Lua sandbox
airesearch_sse.go187SSE event streaming
airesearch_cache.go~100Research result caching
mcp_client.go195MCP tool server integration
translation.go614Multi-backend translation with routing
langdetect.go~100Language detection for auto-translate
autocomplete.go302Multi-backend autocomplete merger
classifier.go~100Query intent classification
imageproxy.go433Image proxy with SSRF protection
templates.go975HTML template rendering
db.go304SQLite schema, logging, queries
groups.go263User tier/group loading from KDL
tiers.go222Permission checking
rate_limit.go202Per-IP rate limiting
rewrite.go~100URL hostname rewriting
i18n.go~100Internationalization
instances.go288Instance browser + ami4got
plugins.go214Lua plugin system
selfheal.go~100Autonomous engine repair agents
sidecar.go~100Python sidecar integration
rss.go~100Atom feed generation
openapi.go530OpenAPI 3.0 spec generation
api_compat.go~1004get-compatible API
api_doc.go~100Plain-text API docs
metrics.go~100Prometheus counters
counters.go~50Atomic hit/miss counters
config_validate.go186Config validation at startup
kdl_utils.go~80KDL parsing helpers
kdlresolver.go~80KDL section resolution
helpers.go~80Cookie reading, IP extraction
stubs.go199Compile-time stubs for optional features
types.go~50Shared type definitions

Python sidecar (python-sidecar/)

FileLinesPurpose
server.py96Flask HTTP server
google.py245Google Playwright scraper
yandex.py219Yandex Playwright scraper
qwant.py259Qwant Playwright scraper

KDL configs (data/)

FileControls
config.kdlMain server configuration (port, auth, cache, API keys)
config.default.kdlDefault config template
engines.kdlAll 274 generic engine definitions
peers.kdl / peers.default.kdlFriend network topology and wait policy
groups.kdlUser tiers and permissions
captcha.kdlCaptcha pool sizes and generation cadence
autocomplete.kdlAutocomplete backend weights
translation.kdlTranslation backend priority and rate limits
instances.kdlInstance browser entries
ai-research.kdl / ai-research.default.kdlAI agent provider, model, access control
metaprompt.mdAI agent system prompt

Static assets (static/)

CSS, JavaScript (PoW WASM worker, autocomplete, SSE client), and fonts.

Templates

HTML templates rendered server-side by templates.go. No client-side framework.

Key design decisions

Single binary. The Go server compiles to one static binary (plus CGO for SQLite). No runtime dependencies except the data/ and static/ directories.

KDL over JSON for subsystems. Human-readable, comment-friendly config. The resolver supports both per-file (peers.kdl) and unified (conf.kdl with sections) layouts.

Singleflight coalescing. Concurrent identical queries share one fanout, preventing thundering herd on popular searches.

Ed25519 session cookies. Session tokens are signed, not stored server-side. No session table, no Redis.

Dual-layer anti-abuse. First-time visitors solve a visual captcha (pre-rendered, no JS required for the test itself). Returning visitors solve a WASM proof-of-work (SHA-256 mining). Both issue signed session cookies.

Oracle-first. Queries that match oracle patterns (math, units, DNS, etc.) return instant answers without waiting for engine fanout.

Friend-first cache. Before fanning out to upstream engines, 4got checks local cache, then friend caches. This minimizes upstream load across the friend network.

Design Decisions

Why federation over Tor outbound for IP-ban avoidance. Tor exit nodes are heavily flagged by upstream engines, leading to high captcha rates and slow responses. Federation distributes queries across multiple real IPs (each friend instance has its own), so no single IP gets rate-limited. Tor outbound is still available per-engine (tor-outbound true) for engines that block datacenter IPs but not Tor, but the primary strategy is the friend network.

Why low comment density is intentional. The codebase prefers well-named functions, types, and variables over inline commentary. Comments are reserved for non-obvious architectural decisions, protocol boundaries, and “why” explanations – not restating what the code already says. This keeps the signal-to-noise ratio high during code review and reduces stale-comment drift.

Why KDL over YAML/TOML/JSON for config. KDL supports inline comments, nested blocks without deep indentation, and space-separated multi-value attributes (ideal for categories "web" "news"). YAML is whitespace-sensitive and error-prone; TOML doesn’t handle nested structures cleanly; JSON forbids comments. KDL’s document-oriented model maps naturally to engine definitions and peer configs.

Why SQLite over Redis/Postgres. SQLite is embedded (no external daemon), survives restarts without a separate persistence layer, and handles 4got’s write volume (search logs, cache, stats) comfortably on a single core. The entire database is one file, trivially backed up. Redis would add an operational dependency for the cache layer alone; Postgres is overkill for a single-instance search engine.

Why single-binary Go over microservices. A metasearch engine’s subsystems (fanout, caching, auth, federation) share in-process state (the engine registry, in-memory cache, rate limiter). Splitting them into services would add network hops on the hot path and complicate deployment for self-hosters. The Python sidecar exists only for Playwright-dependent engines that need a real browser, and communicates over localhost HTTP.

Why no BM25. BM25 was implemented and tested but removed. The marginal ranking improvement over the existing position-weighted cross-engine consensus scoring was negligible, and BM25 is vulnerable to keyword stuffing in snippets – an attacker can inflate term frequency in a single engine’s results to dominate rankings. The current scoring (inverse position * engine weight * engine count * authority domain multiplier) is simpler and harder to game.

Why no click tracking. 4got is privacy-first: no result click is recorded, no outbound redirect URL wraps results, and no client-side beacon fires on click. This means ranking cannot incorporate click-through signals, which is an accepted trade-off. The honeypot link (/static/assets/beacon.js) exists solely to detect bots, not to track users.

Why engine goroutines are not cancelled after the user gets their response. When the fanout time budget expires and results are served to the user, the remaining engine goroutines continue running in the background. This is intentional: late-arriving results are merged into the cache, so the next user who searches the same query gets richer results. Cancelling goroutines would save a few connections but would mean cached results are permanently limited to whatever came back within the first time window. The cache is the product; the user’s wait time is the budget.

Why engine health monitoring is notification-based, not autonomous. Engine health snapshots (engine_snapshots.go) store the last working and last error HTML for each engine, and the notification system (notify.go) dispatches structured events to ntfy push notifications, webhooks, and a JSONL event log. Actual remediation happens outside the program — the operator hooks up whatever they want (a Claude agent, a cron job, a human reading notifications). This keeps the search engine’s scope clean: it detects problems and tells you, it doesn’t try to fix them itself. The earlier “self-healing” framing was retired in favor of this more honest and composable design.

Why adaptive PoW difficulty, not static. AdaptivePowDifficulty in auth.go scales the PoW difficulty based on current server load: +1 bit above 100 RPM, +2 above 500, +3 above 1000. This means legitimate users during low-traffic periods solve a ~1 second puzzle, while during attacks the difficulty automatically ramps up. Combined with the image captcha (first visit) and Tor-aware exemptions, this creates a graduated defense that doesn’t punish normal users for attacker behavior.

Deploying 4got

Quick Start

git clone https://github.com/your-user/4got.git && cd 4got
cp data/config.default.kdl data/config.kdl   # edit owner-secret!
docker compose up -d

Open http://localhost:8888 — you have a working search engine. See First-run configuration to set your admin secret and API keys.

Docker Compose

The recommended way to run 4got. The docker-compose.yml builds both the Go server and the optional Python sidecar.

# Start the server only (no browser-backed engines)
docker compose up -d

# Start with the Python sidecar for Google/Qwant/Yandex JS scraping
docker compose --profile sidecar up -d

# View logs
docker compose logs -f server

# Stop everything
docker compose down

Configuration lives in data/config.kdl (mounted as a volume). The SQLite database (data/4got.db) is created automatically on first run.

To customise ports, API keys, or peer settings, copy the override example and edit it:

cp docker-compose.override.yml.example docker-compose.override.yml
# edit docker-compose.override.yml, then:
docker compose up -d

First-run configuration

  1. Copy data/config.default.kdl to data/config.kdl if you haven’t already
  2. Set owner-secret to a strong random string — this is your admin password:
    python3 -c "import secrets; print(secrets.token_urlsafe(32))"
    
  3. Add API keys in the api-keys block (all optional):
    • wolframalpha — instant answers
    • deepl — translation widget
    • spotify-client-id / spotify-client-secret — music results
  4. Configure engine groups in data/groups.kdl (the defaults are sensible)
  5. Start the server and visit /admin/login to authenticate as owner
  6. Optionally set peer-secret and configure data/peers.kdl for federated search across multiple instances

Prerequisites

  • Go 1.23+ (the go.mod targets 1.25 but 1.23+ should work)
  • SQLite3 development headers (e.g. libsqlite3-dev on Debian, sqlite-dev on Alpine)
  • GCC or a C compiler (required by go-sqlite3’s CGO binding)
  • cjxl (optional) for JPEG XL captcha and image proxy reencoding — part of libjxl
  • Playwright (optional) for the Python sidecar that handles JS-rendered engine scraping

Building from source

cd go-server
go build -o 4got .

CGO must be enabled (it is by default). If cross-compiling, set CGO_ENABLED=1 explicitly:

CGO_ENABLED=1 go build -o 4got .

The binary is fully self-contained. Copy it alongside the data/, static/, and templates/ directories.

Directory layout

4got/
  go-server/
    4got          # the binary
  data/
    config.kdl    # main configuration
    *.kdl         # subsystem configs (peers, groups, captcha, etc.)
    4got.db       # SQLite database (created automatically)
  static/         # CSS, JS, fonts
  templates/      # HTML templates

The binary looks for data/config.kdl (relative to CWD) or ../data/config.kdl (when running from go-server/). Override with FGOT_CONFIG=/path/to/config.kdl.

Running directly

cd /path/to/4got
./go-server/4got

By default it listens on 0.0.0.0:8888. Override via config.kdl or environment variables:

FGOT_PORT=9999 FGOT_HOST=127.0.0.1 ./go-server/4got

Running with systemd

A production-ready unit file ships in the repo at 4got.service. Install it:

sudo cp 4got.service /etc/systemd/system/4got.service
sudo systemctl daemon-reload
sudo systemctl enable --now 4got
sudo journalctl -u 4got -f

The unit file targets network-online.target, runs as user/group 4got, applies security hardening (ProtectSystem=strict, ProtectHome=yes, NoNewPrivileges, PrivateTmp), and sets LimitNOFILE=65536 for high-concurrency deployments. Edit ReadWritePaths if your data directory is not /opt/4got/data.

Reverse proxy

nginx

server {
    listen 443 ssl http2;
    server_name search.example.com;

    ssl_certificate     /etc/letsencrypt/live/search.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/search.example.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8888;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # SSE support (AI research agent)
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 120s;
    }

    # Optional: serve static files directly
    location /static/ {
        alias /opt/4got/static/;
        expires 7d;
    }
}

Caddy

search.example.com {
    reverse_proxy 127.0.0.1:8888
}

Caddy handles TLS automatically via Let’s Encrypt.

Docker / Podman

A multi-stage Dockerfile is provided in the project root, based on Debian bookworm-slim (glibc required for CGO/SQLite):

docker build -t 4got .
docker run -d -p 8888:8888 -v 4got-data:/data 4got

Podman is a drop-in replacement – the same commands work with podman instead of docker.

Mount a volume at /data to persist the SQLite database, configuration, and cache across restarts. The docker-compose.yml handles this automatically:

docker compose up -d

Codebase size comparison

Lines of code as of May 2026. Measured with wc -l against source files, excluding vendored dependencies.

ComponentLines
4got Go source (minus tests)20,866
4got test code5,324
4got Go total26,190
4got KDL configs (engine defs, etc.)10,682
4got Python sidecar819
4got templates (HTML)3,326
4got static (JS + CSS)3,299
4get PHP (reference implementation)35,480
SearXNG Python (minus tests)47,425
SearXNG test code3,764
SearXNG Jinja templates2,263

4got’s Go core is ~40% smaller than 4get’s PHP while covering more categories (15 vs 4get’s handful) and adding federation, AI research, oracles, and multi-backend translation. The engine count difference is largely engines.kdl (274 engines declared in config, not code).

PaaS providers

SQLite needs persistent storage on every PaaS. Ephemeral filesystems lose your cache, stats, and search history on every redeploy. Always mount a volume at /data.

Fly.io

A fly.toml is included in the repo root. Create a volume before deploying:

fly launch          # creates the app, detects the Dockerfile
fly volumes create 4got_data --region iad --size 1
fly deploy

The fly.toml mounts the volume at /data and exposes port 8888 internally. Fly handles TLS and routing automatically.

Railway

Railway auto-detects the Dockerfile. Set the PORT environment variable to 8888 in your service settings. Attach a persistent volume mounted at /data to keep SQLite state across deploys.

Render

Create a new Web Service with Docker deployment. Point it at the repo, then add a persistent disk mounted at /data (minimum 1 GB). Set the HTTP port to 8888 in the service configuration.

Coolify

Self-hosted PaaS. Point Coolify at the git repo, it will build from the Dockerfile. Add a persistent volume mapped to /data. No special configuration needed beyond that.

LXC / bare metal

Install Go 1.25+ and a C compiler (GCC), then build from source:

cd go-server
CGO_ENABLED=1 go build -o 4got .

Copy the binary alongside the required directories to the target machine:

rsync -a go-server/4got go-server/templates/ static/ data/ docs/ target:/opt/4got/

The binary runs from the project root and expects data/, static/, and go-server/templates/ as siblings (it searches multiple template paths automatically). See Running with systemd above for a production service file.

Install tesseract-ocr on the host if you want file analysis OCR support.

DNS privacy

To prevent search query leakage to your ISP via DNS resolution, run dnscrypt-proxy in front of your system resolver (Technitium, Unbound, or systemd-resolved). This encrypts outgoing DNS queries to upstream resolvers.

Recommended setup:

  1. Install dnscrypt-proxy and point it at trusted DoH/DoT servers (e.g. Quad9 9.9.9.9, Cloudflare 1.1.1.1).
  2. Configure your local resolver (Technitium/Unbound) to forward to 127.0.0.1:5353 (dnscrypt-proxy’s default listen port).
  3. Ensure 4got’s host uses this resolver chain so engine queries don’t leak plaintext DNS to your ISP.

Or just point Claude Code at this repo and ask it to help you deploy

claude "Deploy 4got to my server at /opt/4got. Set up systemd, nginx reverse proxy on search.example.com, and configure a Wolfram Alpha API key."

Runtime dependencies for banner uploads

Banner transcoding shells out to cjxl (stills → progressive JXL, metadata stripped) and ffmpeg with libaom (animated GIF → animated AVIF). On Debian: apt install libjxl-tools ffmpeg. Without them, banner uploads fail with “Transcoding failed” — searches are unaffected.

Scaling plan

Every server owner needs one. This is the capacity model for a 4got instance, worked for the reference deployment (1 vCPU, 1 GB RAM, 300 mbps, 1 TB/month), and the API-key policy that survives growth.

The surprise: bandwidth is the binding constraint, not compute

Go handles thousands of concurrent connections on one core; a cached search serve is sub-millisecond work. The ceilings, in the order you will actually hit them:

  1. Monthly traffic cap. 1 TB/month ≈ 33 GB/day ≈ 385 KB/s sustained. A results page is ~100 KB raw, ~30 KB with compression, before proxied images. That prices the box at roughly ~1M page views/month as an absolute ceiling — and proxied images eat it far faster, which is why the traffic guardrails throttle heavy transfers first while search stays fast. Watch the Traffic line in settings; the knobs are traffic-* in config.kdl.
  2. Upstream engine tolerance. Every UNCACHED search fans out to dozens–hundreds of engines from one IP. Public engines throttle single IPs long before your hardware blinks. The realistic uncached-fanout budget is hundreds to low thousands per day per engine — the communal cache’s hit rate is therefore the real multiplier of the whole system. Everything that raises hit rate (shared anonymous cache, prefetch, peers) is scaling work.
  3. Compute/RAM. Last by a wide margin: ~50–100 simultaneous uncached fanouts before TLS handshake latency degrades on one core; cached serves in the hundreds per second; SQLite’s single writer is nowhere near stressed at these rates.

Net: thousands of daily active users are realistic on the reference box if the cache hit rate is healthy; millions of monthly uniques require more egress — a bigger pipe, and more usefully, peers: federated instances sharing caches are the horizontal-scaling story, and each peer brings its own upstream IP budget.

API keys: the tiered policy

Server-covered keys do not scale — a 25-req/day API handed to the public is exhausted by breakfast and everyone’s annoyed. The policy that scales:

  • Trusted groups (deluxe: friends, family, beta testers) — keys covered by the server, limits shown in settings.
  • Everyone else — bring your own key. BYO keys can opt into the pool: draws are randomly scrambled across contributed keys, which (a) gives contributors anonymity-by-randomness against the API provider — the provider sees the server’s IP either way, but key-scrambling breaks the which-human-asked-what linkage that identity-tied keys otherwise leak — and (b) absorbs surges, since most people use a fraction of their quota on most days, so a member can burst far beyond their own key some days (their agents too) against the pool’s slack.
  • Grace budget (Monero era): pool members may run negative within a configured grace; the deeper negative, the harder the scheduler works to lend their key out. Balances settle when the ledger lands.
  • ToS note for owners: some API providers prohibit key pooling. Enabling the pool is the owner’s call per API; the per-key limit fields and group gating already let you keep sensitive keys unpooled.

Changing server defaults without fighting users’ cookies

Revoking a covered key, changing a default, renaming a setting — all must be supported behavior, not a migration crisis. Mechanism (planned): a settings epoch — the server stamps an epoch into served settings; cookies carry the epoch they were written under; on mismatch the server migrates known-renamed keys, drops known-dead ones, and (where a change is user-visible) says so once. Until the epoch machinery lands, the rule of thumb: additive changes are free; renames keep a legacy-read fallback (as ai_mode → ai_mode_single/double does today); revocations of covered keys degrade to “bring your own” messaging rather than errors.

Friend Access with mTLS

4got uses mutual TLS (mTLS) client certificates to control who can access your instance. This is a deliberate choice over OAuth, SSO, or any third-party identity provider — your friends prove their identity with a cryptographic certificate that you issued, and no external service is involved in the handshake.

Why mTLS instead of OAuth?

OAuth solves “let strangers log in with their existing accounts.” You’re solving “let my friends in and everyone else out.” Different problem, different tool.

With mTLS:

  • No third party knows who uses your search engine. There’s no Google/GitHub/Discord OAuth callback leaking the fact that your friend searched for something. The authentication is a TLS handshake between the client and your server — no one else is in the loop.
  • No tokens to steal. A stolen OAuth token gives an attacker your session. A stolen client cert without the private key is useless. The private key never leaves your friend’s device.
  • No session hijacking. The TLS handshake happens before HTTP. There are no cookies, headers, or query parameters that a man-in-the-middle or XSS attack could extract.
  • No infrastructure dependency. Your auth doesn’t break when GitHub is down, when Google changes their OAuth scopes, or when Discord decides to deprecate an API version.
  • Revocation is simple. Remove a friend’s certificate serial number from the trust list. Done. No “invalidate all sessions” dance.

The tradeoff is that cert distribution requires a one-time manual step per friend. For a friends-and-family instance, this is a feature, not a bug — it means access can’t be self-service, which is exactly what you want.

How it works

Friend's browser                         Your server
      |                                       |
      |------- TLS ClientHello -------------->|
      |<------ TLS ServerHello + ServerCert --|
      |<------ CertificateRequest ------------|
      |------- ClientCert (signed by your CA) |
      |------- CertificateVerify ------------>|
      |                                       |
      |  (server checks: is this cert signed  |
      |   by my CA? is it not revoked?)       |
      |                                       |
      |<------ TLS Finished ------------------|
      |------- HTTP request ----------------->|
      |<------ Search results ----------------|

If the client doesn’t present a valid certificate, the TLS handshake fails. The connection is refused before any HTTP traffic is exchanged. Unauthorized users never reach 4got — they get a TLS error, not a login page.

Setup

1. Create your Certificate Authority

You are the CA. This is a one-time operation.

mkdir -p /etc/4got/certs && cd /etc/4got/certs

# Generate CA private key
openssl genrsa -out ca.key 4096

# Generate CA certificate (valid 10 years)
openssl req -new -x509 -key ca.key -out ca.crt -days 3650 \
  -subj "/CN=4got Friend CA/O=4got"

Guard ca.key with your life. Anyone with this file can issue client certs that your server will trust. Store it offline when not issuing new certs.

2. Issue a client certificate for a friend

Run this for each friend. Replace alice with their name.

# Generate friend's private key
openssl genrsa -out alice.key 2048

# Generate a certificate signing request
openssl req -new -key alice.key -out alice.csr \
  -subj "/CN=alice/O=4got Friends"

# Sign with your CA (valid 2 years)
openssl x509 -req -in alice.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out alice.crt -days 730

# Package into a .p12 file (this is what your friend installs)
openssl pkcs12 -export -out alice.p12 \
  -inkey alice.key -in alice.crt -certfile ca.crt \
  -name "4got (alice)"

You’ll be prompted for an export password. Pick something simple — your friend needs to type it once during import. You can use an empty password if you trust the delivery channel.

Clean up the intermediate files:

rm alice.key alice.csr alice.crt
# Keep alice.p12 — this is what you give to your friend

3. Give the .p12 to your friend

Preferably via sneakernet — hand them a USB drive, AirDrop it across the table, or NFC beam it from your phone. Physical proximity is the strongest delivery channel because there’s no server, no log, no metadata trail, and no possibility of interception.

If physical handoff isn’t possible, send the .p12 over an end-to-end encrypted channel (Signal, Matrix with verification, encrypted email). Communicate the export password through a different channel than the file itself (e.g., file over Signal, password spoken over a phone call).

Never send client certificates over unencrypted email, Discord DMs, Slack, or any platform that stores messages server-side. The .p12 contains your friend’s private key — if it leaks, you need to revoke and reissue.

4. Configure Caddy

Caddy handles the public-facing TLS (automatic Let’s Encrypt) and verifies client certificates against your CA.

search.yourdomain.com {
    tls {
        client_auth {
            mode require_and_verify
            trusted_ca_cert_file /etc/4got/certs/ca.crt
        }
    }
    reverse_proxy localhost:8888
}

That’s the entire Caddyfile. Caddy:

  • Obtains and renews a Let’s Encrypt certificate for search.yourdomain.com automatically
  • Requires every connecting client to present a certificate signed by your CA
  • Rejects unauthorized connections at the TLS layer (before HTTP)
  • Forwards authenticated requests to 4got on localhost

5. Your friend installs the certificate

Android

Tap the .p12 file (from Files, a download, or a file manager). Android prompts to install it as a user certificate. Enter the export password. Chrome and other browsers will present it automatically when connecting to your instance.

iOS / iPadOS

Tap the .p12 file to open it. Go to Settings → General → VPN & Device Management and install the profile. Enter the export password. Safari and all apps using the system keychain will present it automatically.

macOS

Double-click the .p12 file. Keychain Access opens and imports it. Enter the export password. Safari and Chrome use it automatically. Firefox users: Preferences → Privacy & Security → Certificates → View Certificates → Import.

Windows

Double-click the .p12 file. The Certificate Import Wizard opens. Follow the prompts, enter the export password, and import into the “Personal” store. Edge and Chrome use it automatically. Firefox users: Settings → Privacy & Security → Certificates → View Certificates → Import.

Linux

Firefox: Preferences → Privacy & Security → Certificates → View Certificates → Your Certificates → Import. Select the .p12 file and enter the export password. Chromium-based browsers: use the system NSS certificate store or import via chrome://settings/certificates.

None of this touches VPN settings, proxy configuration, or network setup. The certificate is a TLS identity credential — it lives in the browser/OS certificate store alongside everything else and is presented during the TLS handshake when the server requests it.

Revoking a friend’s access

Quick method: reissue the CA

If you only have a handful of friends and need to revoke one, the simplest approach is to regenerate the CA and reissue certificates to everyone except the revoked friend. This is practical when you have fewer than ~20 friends.

Proper method: Certificate Revocation List (CRL)

For larger deployments or when you don’t want to bother everyone:

# Create a CRL (first time)
openssl ca -gencrl -out /etc/4got/certs/crl.pem \
  -keyfile /etc/4got/certs/ca.key \
  -cert /etc/4got/certs/ca.crt

# Revoke a specific certificate
openssl ca -revoke alice.crt \
  -keyfile /etc/4got/certs/ca.key \
  -cert /etc/4got/certs/ca.crt

# Regenerate the CRL
openssl ca -gencrl -out /etc/4got/certs/crl.pem \
  -keyfile /etc/4got/certs/ca.key \
  -cert /etc/4got/certs/ca.crt

Note: CRL-based revocation with Caddy requires checking the tls directive documentation for your Caddy version, as CRL support has varied across releases. An alternative is to maintain a list of allowed certificate fingerprints and check them in a Caddy handler.

FAQ

Why not Tailscale / WireGuard? Those are great for private access but they require your friends to install a VPN client and join your network. mTLS works in any browser on any device with zero additional software. It’s also invisible to the user after the one-time cert install — they just open the URL and search.

Why not Cloudflare Access? Cloudflare terminates TLS at their edge and inspects all HTTP traffic in plaintext. There is no Cloudflare configuration where they cannot read your search queries. If you’re building a privacy-first search engine, routing all traffic through a surveillance company defeats the purpose.

Why not HTTP basic auth? Passwords get shared, written down, reused, and phished. Client certificates are cryptographic keys bound to a device. Basic auth is also transmitted on every request (Base64 encoded, not encrypted beyond TLS), while client cert auth happens once during the TLS handshake.

Why not OAuth2 Proxy / Authelia / Authentik? These are good tools for multi-tenant SaaS applications. For a friends-only search engine, they add a database, a login page, a session store, password reset flows, and a dependency on an external identity provider — all to solve a problem that mTLS solves with zero moving parts.

Can my ISP see that my friends are connecting? Your ISP can see that an IP address connected to your server on port 443. They cannot see the client certificate, the search queries, or any HTTP content — that’s all inside the TLS tunnel. The mTLS handshake itself is encrypted.

What if a friend loses their device? Revoke their certificate and issue a new one. The old cert becomes useless immediately (once the CRL is updated or the CA is reissued).

Federation: The Friend Network

4got instances can form a peer-to-peer friend network for cache sharing, service proxying, and instance discovery. This is how multiple 4got deployments cooperate to improve speed and resilience.

Philosophy

The 4got protocol is a result-sharing protocol for self-contained queries — queries that are small, universal, and should have roughly the same answer regardless of who asks or where they ask from. Search engines are just one type of intelligence backend that can feed results into this protocol. A friend’s instance might answer a query from Google, from a custom scraper, from an AI agent, from a local database, or from another friend’s cache — the protocol doesn’t care. It exists to share answers between friends, not to replace any particular search provider. There is room for the protocol to be used with all kinds of intelligence backends beyond traditional search engines.

Overview

The friend network provides three capabilities:

  1. Peer cache sharing — friends share cached search results, reducing redundant upstream queries
  2. Service proxy — friends proxy expensive services (translation, Wolfram Alpha, autocomplete) for each other
  3. Instance discovery — the /ami4got endpoint and instance browser let users find other instances

Peer cache sharing protocol

When a search is performed, 4got queries all configured friends in parallel via GET /api/v1/cache?q=<query>&cat=<category>. Friends respond with cached results if available.

Request flow

  1. User searches for “quantum computing”
  2. 4got sends the query to all configured friends simultaneously
  3. Friends check their local cache and respond with results (or 204 No Content)
  4. 4got merges friend results with its own engine results using the standard scoring/deduplication algorithm
  5. If store-friend-cache is enabled, results from friends are stored in the local cache

Wait policy

The wait-for setting in peers.kdl controls how many friends must respond before proceeding:

ValueMeaning
1.0Wait for all friends (proportion: 100%)
0.5Wait for half of the online friends
3Wait for exactly 3 friends
-2OK to miss up to 2 friends
-0.1OK to miss up to 10% of friends

The wait policy interacts with the fanout hard window (fanout-hard-window-ms in config.kdl). Friends that don’t respond within the hard window are skipped regardless of the wait policy.

Failure tracking

The failure-threshold setting tracks friend reliability:

failure-threshold 0.01 1d

This means: if a friend fails more than 1% of requests over a 1-day window, consider them “down” and don’t wait for them. They’ll be retried periodically but won’t block search results.

Always-wait and preferred friends

// Always wait for this friend's cache (even if slow)
always-wait-for "alice"

// Prefer this friend for specific services
prefer-friend "alice" for="translation,wolfram"

Service proxy

Friends can proxy expensive API-backed services for each other. This lets a single Wolfram Alpha API key serve multiple instances, or a single translation backend serve the whole friend network.

How it works

  1. Instance A needs to translate text but has no DeepL API key
  2. Instance A sends POST /api/v1/proxy to friend Instance B with {"service": "translation", "params": {"text": "hello", "target": "fr"}}
  3. Instance B checks its allow-proxy list, finds “translation” is allowed, performs the translation, and returns the result
  4. Instance A uses the result as if it performed the translation locally

Supported services

  • translation — text translation (DeepL, Papago, Lingva, NLLB, Microsoft)
  • wolfram — Wolfram Alpha short answers API
  • autocomplete — search suggestion backends

Configuration

On the provider instance (the one that actually has the API keys):

allow-proxy "translation" "wolfram" "autocomplete"

On the consumer instance (the one that wants to use the services):

prefer-friend "provider-name" for="translation,wolfram"

The consumer will try the preferred friend first, then fall back to local backends.

Authentication

Service proxy requests are authenticated using the peer-secret from config.kdl, sent as a Bearer token.

Instance discovery

The ami4got endpoint

Every 4got instance exposes GET /ami4got (inspired by 4get’s /ami4get.php). This returns a JSON object with instance metadata:

{
  "version": "2.0",
  "api_version": "1",
  "engine_count": 15,
  "stats": {
    "hits": 12345,
    "real_searches": 6789
  }
}

Instance browser

The /instances page shows a list of known 4got instances. It fetches info from configured friends and public instances periodically.

Configure in instances.kdl:

fetch-interval-min 60

friends {
    instance "https://search.friend.example.com"
}

public {
    instance "https://public.4got.example.com"
}

Friend-of-friend discovery

When Instance A fetches /ami4got from Instance B, it can discover Instance B’s own friends. This creates an organic network where adding a single friend connection transitively discovers the broader network.

Setting up two instances as friends

Step 1: Choose a shared secret

Both instances need the same peer-secret in their config.kdl:

peer-secret "a_strong_random_string_shared_between_friends"

Step 2: Configure peers on each instance

On Instance A (data/peers.kdl):

wait-for 1.0
store-friend-cache true
allow-proxy "translation" "wolfram"

peer "instance-b" {
    url "https://search-b.example.com"
}

On Instance B (data/peers.kdl):

wait-for 1.0
store-friend-cache true
allow-proxy "translation" "wolfram"

peer "instance-a" {
    url "https://search-a.example.com"
}

Step 3: Add to instance browser (optional)

On both instances, add each other to instances.kdl:

friends {
    instance "https://search-b.example.com"
}

Step 4: Test

  1. Restart both instances
  2. Search on Instance A — check the logs for “cached on 1 friend” in the results
  3. Visit /admin/log to see engine stats (friend results appear as (friend) entries)

Step 5: Optional mTLS

For additional security, configure mutual TLS between friends:

peer "instance-b" {
    url "https://search-b.example.com"
    cert "/path/to/client.crt"
    key "/path/to/client.key"
}

Both instances need to trust each other’s client certificates.

AI Research Agent

The AI research agent is a tool-using LLM that synthesizes search results into a comprehensive answer. It is triggered when a user searches the same query twice (the “double-search” trigger), indicating they want deeper analysis than raw search results can provide.

How it works

  1. User searches “quantum error correction”
  2. 4got returns normal search results
  3. User searches “quantum error correction” again (or clicks the “Research” button)
  4. 4got starts the AI research agent as a Server-Sent Events (SSE) stream
  5. The agent fetches top search result URLs, reads them, and synthesizes an answer
  6. Intermediate findings stream to the user in real time via the oracle box

Configuration

Edit data/ai-research.kdl:

enabled true
mode "lua"
provider "cerebras"
model "llama3.1-8b"
api-key "csk-..."
base-url "https://api.cerebras.ai/v1"
max-runtime-seconds 60
require "deluxe"

Fields

FieldDescription
enabledMaster switch (true/false)
modeExecution mode: “lua” (sandboxed, recommended) or “bash” (dangerous, host access)
providerLLM provider: “cerebras”, “openrouter”, “ollama”, “openai”, or any OpenAI-compatible endpoint
modelModel name (provider-specific)
api-keyAPI key for the provider
base-urlOverride the API endpoint URL. Defaults are inferred from provider name.
max-runtime-secondsKill the agent after this many seconds
requireGroup permission required to use the agent (e.g. “deluxe”). Empty = everyone.

Providers

Cerebras (recommended for free tier):

provider "cerebras"
model "llama3.1-8b"
api-key "csk-..."
base-url "https://api.cerebras.ai/v1"

Cerebras offers free API access for Llama 3.1 8B with extremely fast inference (~1000 tok/s). Good enough for research synthesis.

OpenRouter (access to many models):

provider "openrouter"
model "anthropic/claude-sonnet-4-20250514"
api-key "sk-or-..."

Ollama (local, no API key needed):

provider "ollama"
model "llama3.1:8b"
base-url "http://localhost:11434/v1"

Any OpenAI-compatible API:

provider "custom"
model "your-model"
api-key "your-key"
base-url "https://your-api.example.com/v1"

Metaprompt customization

The agent’s system prompt is loaded from data/metaprompt.md. This file controls the agent’s personality, tool usage patterns, and output format.

Key sections in the default metaprompt:

  • Context: explains to the LLM where it is and what 4got does
  • Tools: describes each tool and when to use it
  • Behavior rules: stream findings via send(), always use eta() before slow ops, fetch 2-3 URLs minimum
  • Context management: how to use pin() and forget() to manage the limited context window

You can customize the metaprompt to change the agent’s behavior. For example, to make it more concise:

Add to metaprompt.md:

## Output length
Keep your final answer under 200 words. Be direct.

Or to make it focus on academic sources:

## Source priority
Prefer academic sources (arxiv.org, scholar.google.com, .edu domains) over news articles.
Always check if there's a relevant arXiv paper.

Tool reference

send(text)

Shows text to the user immediately as a streaming update. The user sees a live feed of the agent’s findings. Use frequently for intermediate results.

send("Found that Rust 1.84 adds async trait support. Checking release notes...")

eta(seconds, job_name)

Updates the ETA countdown timer shown to the user. Call before any slow operation (fetching URLs, running code).

eta(5, "fetching rust-lang.org blog")

fetch(url)

Fetches a URL and returns the page text with HTML tags stripped. Max 100KB response, 10-second timeout. Supports chunked reading for large pages.

fetch("https://blog.rust-lang.org/")

For large pages, request subsequent chunks:

fetch("https://example.com/long-article", chunk=2)

lua(code)

Executes Lua code in a sandboxed environment. Available standard libraries:

  • string: string.find, string.sub, string.gsub, string.format, etc.
  • table: table.insert, table.remove, table.sort, table.concat
  • math: math.floor, math.ceil, math.random, math.pi, etc.
  • json_decode(s): Parse a JSON string into a Lua table
  • json_encode(t): Serialize a Lua table to JSON
  • http_get(url): Fetch a URL and return the body as a string (uses the Go HTTP client with timeouts)

Example:

local data = http_get("https://api.example.com/data.json")
local parsed = json_decode(data)
for _, item in ipairs(parsed.results) do
    print(item.title .. ": " .. item.score)
end

pin(text)

Pins a short note to the agent’s context that will never be evicted during context window management. Use to preserve key findings across a long research session. Keep pins to 1-2 sentences.

pin("Key finding: the paper by Smith et al. (2026) shows 99.9% error correction rate using surface codes.")

forget(tool_call_id)

Removes a previous tool result from the agent’s context to free up space. Use after extracting what you need from a large fetch result.

forget("call_abc123")

friction(description)

Silently logs an improvement suggestion. These are recorded for the server operator to review. The user never sees them.

friction("No academic search engine configured - adding arXiv would improve research quality")

Double-search trigger UX

The research agent activates when a user searches the same query twice. The UI flow:

  1. First search: normal results displayed
  2. Second search (same query): a “Researching…” panel appears above results with a live SSE stream
  3. The agent’s intermediate send() calls appear as streaming text
  4. The eta() timer shows in the corner
  5. When complete, the final answer replaces the streaming panel

The SSE endpoint is GET /api/v1/research?q=<query>. It returns events:

  • event: message — intermediate text from send()
  • event: eta — ETA update from eta()
  • event: done — final answer, stream ends
  • event: error — error message, stream ends

Lua sandbox capabilities

The Lua sandbox (mode "lua") provides a safe execution environment:

Allowed:

  • All standard string/table/math operations
  • json_decode() and json_encode() for data processing
  • http_get() for fetching URLs (through Go’s HTTP client with timeout and SSRF protection)
  • print() for output (captured as the tool result)
  • tostring(), tonumber(), type(), pairs(), ipairs(), unpack()

Blocked:

  • File system access (io, os)
  • require(), dofile(), loadfile()
  • debug library
  • Arbitrary package loading
  • Long-running loops (killed by max-runtime-seconds)

Bash mode warning: Setting mode "bash" gives the agent full shell access on the host. Only use this if 4got is running inside an isolated container (LXC, Docker, VM).

Friction feedback system

The friction() tool lets the agent report issues it encounters during research. These are logged to the database and can be reviewed by the server operator.

Common friction reports:

  • Missing engine types (e.g. “no academic engine for arXiv queries”)
  • Poor result quality for specific query types
  • Timeout issues with upstream services
  • Suggestions for new oracle types

Friction logs are stored in the ai_research table and can be queried via the admin panel.

Plugin System

The plugin system supports Lua plugins and MCP tool integration.

Lua plugins

Lua plugins are the primary extension mechanism. Drop a .lua file in the plugins/ directory (relative to the data directory) and it will be loaded at startup.

Hooks

Plugins can register handlers for these hooks:

HookArgumentsReturnWhen
on_search(query, category, params)query string, category string, search params tablemodified query string (or nil to keep original)Before search is dispatched to engines
on_result(result, engine_id)result table, engine ID stringmodified result table, or nil to filter it outAfter each engine returns results, before merging
on_render(html, query, category)HTML string, query, categorymodified HTML stringAfter the results page HTML is generated, before sending to client

Plugin structure

-- plugins/example.lua
-- Plugin metadata
plugin = {
    name = "Example Plugin",
    version = "1.0",
    author = "your name",
    description = "An example plugin"
}

-- Called before search dispatch
function on_search(query, category, params)
    -- Modify query, return nil to keep original
    return nil
end

-- Called for each result from each engine
function on_result(result, engine_id)
    -- Return the result to keep it, nil to filter it out
    return result
end

-- Called after HTML is generated
function on_render(html, query, category)
    -- Inject custom HTML, modify existing HTML, etc.
    return html
end

Available Lua APIs

Plugins have access to the same sandbox as the AI research agent:

  • string, table, math standard libraries
  • json_decode(s) / json_encode(t) for JSON processing
  • http_get(url) for fetching URLs
  • log(message) for logging to the server log
  • config_get(key) for reading plugin-specific config values

Example: Ad domain blocker

-- plugins/block_ads.lua
plugin = {
    name = "Ad Blocker",
    version = "1.0",
    description = "Filters results from known advertising domains"
}

local ad_domains = {
    ["doubleclick.net"] = true,
    ["googleadservices.com"] = true,
    ["facebook.com/ads"] = true,
    ["amazon.com/gp/slredirect"] = true,
    ["taboola.com"] = true,
    ["outbrain.com"] = true,
    ["criteo.com"] = true,
}

-- Extract domain from URL
local function get_domain(url)
    local domain = url:match("https?://([^/]+)")
    if domain then
        domain = domain:gsub("^www%.", "")
        return domain
    end
    return ""
end

function on_result(result, engine_id)
    local domain = get_domain(result.url or "")
    if ad_domains[domain] then
        log("Blocked ad domain: " .. domain)
        return nil  -- filter out
    end
    return result
end

Example: Result enrichment

-- plugins/enrich_github.lua
plugin = {
    name = "GitHub Enricher",
    version = "1.0",
    description = "Adds star count and language to GitHub repo results"
}

function on_result(result, engine_id)
    local url = result.url or ""
    local owner, repo = url:match("github%.com/([^/]+)/([^/]+)")
    if owner and repo then
        -- Fetch repo info from GitHub API
        local api_url = "https://api.github.com/repos/" .. owner .. "/" .. repo
        local data = http_get(api_url)
        if data then
            local info = json_decode(data)
            if info and info.stargazers_count then
                result.snippet = result.snippet ..
                    " [★" .. tostring(info.stargazers_count) ..
                    " | " .. (info.language or "unknown") .. "]"
            end
        end
    end
    return result
end

MCP for AI tools

The AI research agent can be extended with MCP (Model Context Protocol) servers to give it access to additional tools beyond the built-in set.

How MCP integration works

  1. Configure MCP servers in ai-research.kdl (or the ai-research section of your unified KDL config)
  2. At startup, 4got connects to each MCP server and discovers available tools
  3. These tools are added to the AI research agent’s tool list alongside the built-in tools
  4. When the agent calls an MCP tool, 4got proxies the call to the appropriate MCP server

Configuration

// In ai-research.kdl
mcp-server "local-kb" {
    command "python3" "-m" "knowledge_base_mcp"
    env "KB_PATH=/path/to/knowledge-base"
}

mcp-server "calculator" {
    url "http://localhost:3001/mcp"
}

MCP servers can be launched as subprocesses (command) or connected to over HTTP (url).

Example: Local knowledge base MCP server

Create an MCP server that gives the AI agent access to a local knowledge base:

# knowledge_base_mcp.py
import json
import sys
import os
from pathlib import Path

KB_PATH = Path(os.environ.get("KB_PATH", "./kb"))

def search_kb(query):
    """Search local markdown files for relevant content."""
    results = []
    for md_file in KB_PATH.glob("**/*.md"):
        content = md_file.read_text()
        if query.lower() in content.lower():
            # Extract surrounding context
            idx = content.lower().index(query.lower())
            start = max(0, idx - 200)
            end = min(len(content), idx + 200)
            results.append({
                "file": str(md_file.relative_to(KB_PATH)),
                "excerpt": content[start:end]
            })
    return results[:5]

# MCP protocol handler
def handle_request(request):
    if request["method"] == "tools/list":
        return {
            "tools": [{
                "name": "search_knowledge_base",
                "description": "Search the local knowledge base for information",
                "inputSchema": {
                    "type": "object",
                    "properties": {
                        "query": {"type": "string", "description": "Search query"}
                    },
                    "required": ["query"]
                }
            }]
        }
    elif request["method"] == "tools/call":
        tool_name = request["params"]["name"]
        if tool_name == "search_knowledge_base":
            query = request["params"]["arguments"]["query"]
            results = search_kb(query)
            return {"content": [{"type": "text", "text": json.dumps(results, indent=2)}]}
    return {"error": "unknown method"}

# stdio transport
for line in sys.stdin:
    request = json.loads(line)
    response = handle_request(request)
    response["id"] = request.get("id")
    print(json.dumps(response), flush=True)

Example: Giving the AI agent calculator access

If the built-in calculator oracle isn’t sufficient, you could connect a Wolfram Language MCP server:

mcp-server "wolfram-lang" {
    command "wolframscript" "-code" "MCPServer[]"
}

This would let the agent evaluate arbitrary Wolfram Language expressions during research.

Plugin loading order

  1. All .lua files in plugins/ are loaded alphabetically
  2. MCP servers are connected in the order they appear in config
  3. Plugins are loaded after all engines are registered but before the HTTP server starts
  4. Plugin hooks run in load order; the first on_search return value that is non-nil wins

Plugin safety

  • Lua plugins run in the same sandbox as the AI research agent (no file/OS access)
  • http_get() goes through Go’s HTTP client with SSRF protection
  • Plugins that throw errors are logged and disabled, they don’t crash the server
  • MCP servers run as separate processes with their own permissions
  • Plugin execution has a per-call timeout (configurable, default 5 seconds)

Planned but not yet implemented

  • Plugin hot-reloading (SIGHUP or admin endpoint to reload without restart)
  • Per-plugin configuration in KDL
  • Plugin marketplace/registry
  • TypeScript/WASM plugin support
  • Plugin-specific rate limiting
  • Plugin metrics in the admin dashboard

Adding a Language

4got ships with 62 locale files. To add a new language:

Steps

  1. Copy data/locales/en.kdl to data/locales/{code}.kdl where {code} is the ISO 639-1 language code (e.g. fr, de, zh).

  2. Translate every key. The format is key "translated value" — one per line.

  3. Do NOT include about_* keys in non-English locales. The about page renders in English only.

  4. RTL languages (ar, fa, he, ur, yi, dv) get automatic dir="rtl" on the HTML tag. No extra work needed.

  5. Restart the server or send SIGUSR1 to hot-reload.

File naming

Use standard ISO 639-1 codes. For regional variants, use underscore: pt_BR.kdl, zh_TW.kdl.

Testing

Set the language cookie manually: document.cookie = "language=fr; path=/; max-age=34560000" and refresh. Or use the language setting in /settings if it exists for your locale.

Contributing to 4got

Dev environment setup

  1. Clone the repo and cd into it.
  2. Install Go 1.25+ (the go.mod requires 1.25.0).
  3. Build: cd go-server && CGO_ENABLED=1 go build -o 4got .
  4. Copy data/config.default.kdl to data/config.kdl and set your owner-secret.
  5. Run: cd .. && go-server/4got

CGO is required for the SQLite driver.

Adding a new KDL engine

Generic engines are defined in data/engines.kdl and parsed by go-server/generic_engine.go. See engine-spec.md for the full KDL schema. The fastest workflow is the admin playground at /admin/engines/new (owner-only): fill in the form, test against a live query, then click “Save Engine” to append the KDL block to disk and register it without a restart.

Adding a new oracle

Oracles are instant-answer widgets shown above search results (calculator, unit converter, currency, etc.).

  1. Write your oracle function in go-server/oracles_extra.go. Follow the existing pattern: the function takes a query string (and optionally *http.Request, http.ResponseWriter, Config) and returns *OracleAnswer or nil.
  2. Register it in go-server/oracles.go inside CheckOracles(). Add a call to your function in the appropriate section (local oracles, API oracles, or knowledge oracles). Order matters: the first match in the local oracle section wins.

Running tests

cd go-server && CGO_ENABLED=1 go test ./...

Code style

  • No comments unless the WHY is non-obvious. Do not explain what the code does if it is clear from reading it.
  • AI-generated files get a header: // AI-generated: <model>
  • Keep functions short. If a function needs a comment block explaining it, it should probably be smaller.

PR expectations

  • CGO_ENABLED=1 go build -o /dev/null . must pass.
  • go vet ./... must be clean.
  • Do not introduce new linter warnings.