Architecture

Request flow

User query
    │
    ▼
┌─────────────────────────────────────────────────────────┐
│ handleSearch()                                          │
│                                                         │
│  1. Parse bangs ──── !g, !w, etc. ──→ 302 redirect     │
│     │ (no match)                                        │
│     ▼                                                   │
│  2. Auth check                                          │
│     │                                                   │
│     ├── No session, first visit ──→ /captcha (IQ test)  │
│     ├── No session, returning ───→ PoW challenge page   │
│     │                              (client-side mining)  │
│     ▼                                                   │
│  3. Parse search params (query, category, cookies)      │
│     │                                                   │
│     ▼                                                   │
│  4. Local cache lookup ── hit ──→ skip to step 8        │
│     │ (miss)                                            │
│     ▼                                                   │
│  5. Friend cache lookup ── hit ──→ store locally,       │
│     │ (miss)                        skip to step 8      │
│     ▼                                                   │
│  6. Fanout (singleflight-coalesced)                     │
│     ├── Engine 1 ──→ goroutine ──→ HTTP scrape/API      │
│     ├── Engine 2 ──→ goroutine ──→ HTTP scrape/API      │
│     ├── Engine N ──→ goroutine ──→ HTTP scrape/API      │
│     └── Friends  ──→ goroutine ──→ peer cache query     │
│     │                                                   │
│     │  hard window (1000ms default) or                  │
│     │  idle timeout (500ms) with must-include gate       │
│     ▼                                                   │
│  7. Merge + deduplicate + rank                          │
│     ├── URL normalization + dedup                       │
│     ├── Cross-engine consensus scoring                  │
│     ├── Per-engine weight × position decay              │
│     ├── Intent-based boost (code, academic, images...)  │
│     ├── Tracking param stripping                        │
│     └── Store merged results in cache                   │
│     │                                                   │
│     ▼                                                   │
│  8. Render                                              │
│     ├── Check oracles (calculator, unit, timezone, etc.)│
│     ├── URL rewriting (YouTube→Invidious, etc.)         │
│     ├── Auto-translate if foreign query detected        │
│     ├── Log search (if history enabled)                 │
│     └── Render HTML template with results + timeline    │
│                                                         │
│  9. Push to peer caches (background goroutine)          │
└─────────────────────────────────────────────────────────┘

Auth flow

New visitor                 Returning visitor
    │                            │
    ▼                            ▼
┌────────────┐           ┌─────────────┐
│ IQ Test    │           │ PoW WASM    │
│ (captcha)  │           │ (SHA-256    │
│ Image grid │           │  mining)    │
└────┬───────┘           └──────┬──────┘
     │ correct                  │ valid nonce
     ▼                          ▼
┌──────────────────────────────────────┐
│ Ed25519-signed session cookie        │
│ (AuthCaptcha or AuthPow)             │
│ TTL: captcha=24h, pow=5min           │
└──────────────────────────────────────┘

Owner: POST /admin/login with owner_secret
       → AuthOwner session cookie

Groups: cookie hash → group lookup (groups.kdl)
        → permissions: sidecar, server_api_keys,
          friend_proxy, priority_fanout

Federation protocol

Instance A                         Instance B
    │                                   │
    │  GET /api/v1/cache               │
    │  ?q=QUERY&cat=web                │
    │  Authorization: Bearer PEER_SEC  │
    │ ─────────────────────────────→   │
    │                                   │
    │  200 OK (cached results)          │
    │  or 204 No Content               │
    │ ←─────────────────────────────   │
    │                                   │
    │  POST /api/v1/cache              │
    │  (push fresh results to friend)  │
    │ ─────────────────────────────→   │
    │                                   │
    │  POST /api/v1/proxy              │
    │  {"service":"translation",       │
    │   "params":{...}}                │
    │ ─────────────────────────────→   │
    │                                   │
    │  200 OK (translation result)     │
    │ ←─────────────────────────────   │
    │                                   │
    │  GET /ami4got                    │
    │ ─────────────────────────────→   │
    │  {"version":"2.0",               │
    │   "engine_count":15, ...}        │
    │ ←─────────────────────────────   │

Proxyable services: translation, wolfram, autocomplete
Auth: shared peer_secret as Bearer token
Wait policy: configurable (all friends, N friends, miss-ok)
Failure tracking: auto-mark down at threshold (e.g. 1% over 1d)

AI research agent loop

User searches same query twice
         │
         ▼
GET /api/v1/research?q=QUERY  (SSE stream)
         │
         ▼
┌─────────────────────────────────────────┐
│ RunAIResearch()                         │
│                                         │
│  System prompt (data/metaprompt.md)     │
│  + top search result URLs               │
│         │                               │
│         ▼                               │
│  LLM generates tool calls:             │
│  ┌──────────────────────────────────┐   │
│  │ fetch(url)  → HTTP GET + strip   │   │
│  │ lua(code)   → sandboxed Lua VM   │   │
│  │ send(text)  → SSE "message"      │   │
│  │ eta(s,job)  → SSE "eta"          │   │
│  │ pin(text)   → permanent context  │   │
│  │ forget(id)  → evict old result   │   │
│  │ friction(s) → operator feedback  │   │
│  └──────────────────────────────────┘   │
│         │                               │
│         │ (loop until done or timeout)  │
│         ▼                               │
│  SSE "done" event with final answer     │
│                                         │
│  Cache result for replay               │
└─────────────────────────────────────────┘

Provider: any OpenAI-compatible API
          (Cerebras, OpenRouter, Ollama, custom)
Sandbox:  Lua (recommended) or bash (containers only)
Timeout:  max-runtime-seconds (default 60)
MCP:      optional external tool servers via stdio/HTTP

File organization

Go server (go-server/)

Python sidecar (python-sidecar/)

KDL configs (data/)

Static assets (static/)

CSS, JavaScript (PoW WASM worker, autocomplete, SSE client), and fonts.

Templates

HTML templates rendered server-side by templates.go. No client-side framework.

Key design decisions

Single binary. The Go server compiles to one static binary (plus CGO for SQLite). No runtime dependencies except the data/ and static/ directories.

KDL over JSON for subsystems. Human-readable, comment-friendly config. The resolver supports both per-file (peers.kdl) and unified (conf.kdl with sections) layouts.

Singleflight coalescing. Concurrent identical queries share one fanout, preventing thundering herd on popular searches.

Ed25519 session cookies. Session tokens are signed, not stored server-side. No session table, no Redis.

Dual-layer anti-abuse. First-time visitors solve a visual captcha (pre-rendered, no JS required for the test itself). Returning visitors solve a WASM proof-of-work (SHA-256 mining). Both issue signed session cookies.

Oracle-first. Queries that match oracle patterns (math, units, DNS, etc.) return instant answers without waiting for engine fanout.

Friend-first cache. Before fanning out to upstream engines, 4got checks local cache, then friend caches. This minimizes upstream load across the friend network.

Design Decisions

Why federation over Tor outbound for IP-ban avoidance. Tor exit nodes are heavily flagged by upstream engines, leading to high captcha rates and slow responses. Federation distributes queries across multiple real IPs (each friend instance has its own), so no single IP gets rate-limited. Tor outbound is still available per-engine (tor-outbound true) for engines that block datacenter IPs but not Tor, but the primary strategy is the friend network.

Why low comment density is intentional. The codebase prefers well-named functions, types, and variables over inline commentary. Comments are reserved for non-obvious architectural decisions, protocol boundaries, and “why” explanations – not restating what the code already says. This keeps the signal-to-noise ratio high during code review and reduces stale-comment drift.

Why KDL over YAML/TOML/JSON for config. KDL supports inline comments, nested blocks without deep indentation, and space-separated multi-value attributes (ideal for categories "web" "news"). YAML is whitespace-sensitive and error-prone; TOML doesn’t handle nested structures cleanly; JSON forbids comments. KDL’s document-oriented model maps naturally to engine definitions and peer configs.

Why SQLite over Redis/Postgres. SQLite is embedded (no external daemon), survives restarts without a separate persistence layer, and handles 4got’s write volume (search logs, cache, stats) comfortably on a single core. The entire database is one file, trivially backed up. Redis would add an operational dependency for the cache layer alone; Postgres is overkill for a single-instance search engine.

Why single-binary Go over microservices. A metasearch engine’s subsystems (fanout, caching, auth, federation) share in-process state (the engine registry, in-memory cache, rate limiter). Splitting them into services would add network hops on the hot path and complicate deployment for self-hosters. The Python sidecar exists only for Playwright-dependent engines that need a real browser, and communicates over localhost HTTP.

Why no BM25. BM25 was implemented and tested but removed. The marginal ranking improvement over the existing position-weighted cross-engine consensus scoring was negligible, and BM25 is vulnerable to keyword stuffing in snippets – an attacker can inflate term frequency in a single engine’s results to dominate rankings. The current scoring (inverse position * engine weight * engine count * authority domain multiplier) is simpler and harder to game.

Why no click tracking. 4got is privacy-first: no result click is recorded, no outbound redirect URL wraps results, and no client-side beacon fires on click. This means ranking cannot incorporate click-through signals, which is an accepted trade-off. The honeypot link (/static/assets/beacon.js) exists solely to detect bots, not to track users.

Why engine goroutines are not cancelled after the user gets their response. When the fanout time budget expires and results are served to the user, the remaining engine goroutines continue running in the background. This is intentional: late-arriving results are merged into the cache, so the next user who searches the same query gets richer results. Cancelling goroutines would save a few connections but would mean cached results are permanently limited to whatever came back within the first time window. The cache is the product; the user’s wait time is the budget.

Why engine health monitoring is notification-based, not autonomous. Engine health snapshots (engine_snapshots.go) store the last working and last error HTML for each engine, and the notification system (notify.go) dispatches structured events to ntfy push notifications, webhooks, and a JSONL event log. Actual remediation happens outside the program — the operator hooks up whatever they want (a Claude agent, a cron job, a human reading notifications). This keeps the search engine’s scope clean: it detects problems and tells you, it doesn’t try to fix them itself. The earlier “self-healing” framing was retired in favor of this more honest and composable design.

Why adaptive PoW difficulty, not static. AdaptivePowDifficulty in auth.go scales the PoW difficulty based on current server load: +1 bit above 100 RPM, +2 above 500, +3 above 1000. This means legitimate users during low-traffic periods solve a ~1 second puzzle, while during attacks the difficulty automatically ramps up. Combined with the image captcha (first visit) and Tor-aware exemptions, this creates a graduated defense that doesn’t punish normal users for attacker behavior.