Why the private beta
4got is in a private beta. This page explains why and what that means for you.
The short version
I built this in my free time and I don’t want to expose early testers to security vulnerabilities that haven’t been found yet. A search engine handles sensitive data by nature — your queries reveal what you’re thinking about, what you’re worried about, what you’re researching. I’d rather bring in a small number of people at a time, find the obvious holes, and fix them before opening the doors to everyone.
The longer version
This project started as a personal itch. I wanted a search engine that didn’t suck, didn’t track me, and didn’t require a data center to run. I ralph’d it together in my free time — evenings, weekends, whenever I had the energy. It’s not backed by a company, it’s not funded, and there’s no team. It’s just me and a mass of Claude Code subagents.
That means my bandwidth for handling bug reports, security disclosures, and feature requests is limited. Open-sourcing a security-sensitive application before it’s ready invites problems I can’t respond to fast enough. A private beta lets me control the pace:
- Bring in a few people. They use it for real. They find bugs I missed.
- Fix everything they find. No new testers until the current batch is stable.
- Repeat. Each wave finds fewer issues. The software converges toward something I’m comfortable putting my name on.
This is the same approach Mitchell Hashimoto used for Ghostty, and for similar reasons — not to manufacture hype or exclusivity, but to match the rate of incoming problems to the rate at which one person can fix them.
What this is not
This is not a “sign up for the waitlist and we’ll get back to you” situation. If you know me, ask me. If you don’t know me but you’re genuinely interested, join the community and make yourself known. I’m not trying to be gatekeepy — I just need to be able to sleep at night knowing I’m not serving search results through code that hasn’t been properly reviewed.
When will it be public?
When we’re confident in three things:
- Security. The code has been audited enough that I’m not losing sleep over someone’s queries leaking, their session getting hijacked, or the server getting popped through the image proxy or AI research agent.
- Direction. The architecture is settled enough that early adopters aren’t going to hit breaking changes every week. The federation protocol, the engine spec, the config format — these need to be stable before other people build on top of them.
- Obvious bugs. The kind of stuff that makes you look stupid. Engines that silently return zero results. Oracles that give wrong answers. Cache corruption. The stuff that’s embarrassing in hindsight but invisible when you’re the only person testing.
The code will be released under AGPL-3.0. There’s no open-core play, no commercial license, no bait-and-switch. The private beta is purely a quality gate.
How to get access
Access to the beta is via mTLS client certificates — see Friend Access for the technical details. In practice: I generate a certificate for you, hand it to you (preferably in person), and your browser uses it to authenticate. No accounts, no passwords, no third-party identity provider.
If you want in, reach out. I’m not hard to find.
Quickstart
Get 4got running in 5 minutes.
Build
git clone <repo-url> 4got && cd 4got
cd go-server && CGO_ENABLED=1 go build -o 4got . && cd ..
Configure
cp data/config.default.kdl data/config.kdl
Edit data/config.kdl — at minimum change owner-secret to something unique. The default is intentionally invalid.
For AI research, also copy and configure data/ai-research.default.kdl to data/ai-research.kdl with your OpenRouter/Groq/Ollama API key.
Run
go-server/4got
Starts on port 8888. Visit http://localhost:8888.
First use
Your first search triggers the IQ test (image captcha). Solve it once and you get a 24-hour session cookie. Returning users get a quicker PoW challenge instead.
Key pages
- Settings — /settings (engine toggles, privacy, theme, banners)
- Admin login — /admin/login (enter your owner-secret)
- Trending — /trending (popular searches + live feed)
- History — /history (enable in settings first)
- AI Research — search the same query twice (double-Enter) or include “deep research” / “deep dive” in your query (requires deluxe group + AI config)
- SSH —
ssh localhost -p 2222 "your query"(if ssh-port configured)
Federation
To connect with a friend’s instance, add their URL to data/peers.kdl:
peer "https://friend.example.com" {
secret "shared_peer_secret"
}
Both instances must use the same peer-secret in their config.
User Guide
4got is a metasearch engine. It queries dozens of upstream engines simultaneously, deduplicates and ranks the results, and serves them to you without tracking. This page covers everything you can do from the search bar.
Basic search
Type a query and press Enter. Results come back ranked by cross-engine consensus: pages that multiple engines agree on float to the top.
Category tabs across the top switch between result types:
| Tab | What you get |
|---|---|
| Web | General web results |
| Images | Photos, illustrations, stock images |
| Videos | YouTube, Vimeo, PeerTube, Dailymotion, etc. |
| News | Recent articles from wire services and aggregators |
| Music | Tracks from SoundCloud, Bandcamp, Deezer, etc. |
| Science | Academic papers from arXiv, PubMed, Crossref, etc. |
| IT | Code repos, package registries, docs |
| Files | Torrents (Pirate Bay, 1337x, Solid Torrents, etc.) |
| Maps | OpenStreetMap and Photon geocoding |
| Social | Reddit, Lemmy, Mastodon, HN, Lobsters |
| Books | Open Library, Gutenberg, Goodreads |
| Q&A | Stack Exchange network (SO, Super User, Unix SE, etc.) |
| Packages | Linux distro packages (Arch, NixOS, Alpine, Void, AUR, Flathub) |
| Weather | wttr.in and Open-Meteo |
| Lyrics | Genius |
Quoted phrases
Wrap terms in double quotes to require an exact phrase match:
"rust async trait" stabilization
Engines that support phrase matching will enforce it; the rest filter client-side.
Time filters
Add a time constraint from the settings or URL parameter time=:
| Value | Window |
|---|---|
day or d | Past 24 hours |
week or w | Past 7 days |
month or m | Past 30 days |
year or y | Past 365 days |
Example: /search?q=linux+kernel&time=week
Bang commands
Start your query with ! followed by a shortcode. Bangs come in three flavors.
Redirect bangs
These take you directly to the target site. No 4got results page.
| Bang | Target |
|---|---|
!g | |
!d or !ddg | DuckDuckGo |
!b | Brave Search |
!w | Wikipedia |
!yt | YouTube |
!gh | GitHub |
!r | |
!so | Stack Overflow |
!a | Amazon |
!imdb | IMDb |
!maps | OpenStreetMap |
!nix | NixOS Packages |
!arch | Arch Linux Packages |
!aur | AUR |
!crates | crates.io |
!npm | npm |
!pypi | PyPI |
!mdn | MDN Web Docs |
Example: !gh rust async runtime opens GitHub search for “rust async runtime”.
Engine bangs
These run the search through 4got but force results from a single engine.
| Bang | Engine |
|---|---|
!brave | Brave |
!startpage | Startpage |
!mojeek | Mojeek |
!marginalia | Marginalia |
!wiby | Wiby |
!hackernews or !hn | Hacker News |
!youtube | YouTube |
!soundcloud or !sc | SoundCloud |
!flickr | Flickr |
!pexels | Pexels |
!yandex | Yandex |
!qwant | Qwant |
!bing | Bing |
!baidu | Baidu |
!yahoo | Yahoo |
!wikipedia or !wiki | Wikipedia |
!google | |
!duckduckgo | DuckDuckGo |
Oracle bangs
These redirect to an external instant-answer service.
| Bang | Service |
|---|---|
!wolfram or !wa | Wolfram Alpha Short Answers |
!ddg-ia | DuckDuckGo Instant Answers API |
Oracles (instant answers)
Oracles are local computations or lightweight API calls that produce an answer box above the search results. No upstream search engine is involved. They trigger automatically when your query matches a pattern.
Calculator
Evaluates arithmetic expressions inline.
2 + 2 → 4
15% of 200 → 30
sqrt(144) → 12
2^10 → 1024
sin(1.5) → 0.9974949866
log10(1000) → 3
7 mod 3 → 1
Supported functions: sqrt, cbrt, sin, cos, tan, asin, acos, atan, log (base 10), log2, log10, ln, abs, ceil, floor, round, exp.
Supported operators: +, -, *, /, %/mod, ^/**.
Unit converter
Converts between units of length, weight, volume, speed, and temperature.
10 km to miles → 6.2137119224 miles
72 fahrenheit to celsius → 22.2222222222 celsius
5 gallons to liters → 18.92705 liters
100 kph to mph → 62.1371192237 mph
Supported units:
- Length: km, m, cm, mm, mi, yd, ft, in
- Weight: kg, g, lb, oz, st
- Volume: l, ml, gal, qt, pt, cup, floz
- Speed: mph, kph, mps, knot
- Temperature: celsius/c, fahrenheit/f, kelvin/k
Timezone
Shows current time in a city or IANA timezone.
time in tokyo → Monday, May 12, 2026 2:30 AM JST
time in UTC → Sunday, May 11, 2026 5:30 PM UTC
time in America/Chicago → Sunday, May 11, 2026 12:30 PM CDT
Knows 70+ cities including Tokyo, New York, London, Paris, Berlin, Sydney, Mumbai, Shanghai, Seoul, Dubai, and more.
Base converter
Converts between decimal, hex, binary, and octal.
0xff in decimal → 255
255 in hex → 0xFF
255 in binary → 0b11111111
0b1010 in decimal → 10
0o77 in hex → 0x3F
Color converter
Converts between hex, RGB, and HSL. Shows a color swatch.
#FF5500 → rgb(255, 85, 0) · hsl(20, 100%, 50%)
rgb(0, 128, 255) → #0080FF · hsl(210, 100%, 50%)
hsl(120, 100, 50) → #00FF00 · rgb(0, 255, 0)
Also accepts 3-digit hex (#F50).
Currency converter
Live exchange rates via exchangerate.host.
100 usd to eur → 92.15 EUR
50 gbp in jpy → 9,432.50 JPY
Uses standard 3-letter ISO currency codes (USD, EUR, GBP, JPY, BTC, etc.).
IP geolocation
Look up the physical location and ISP of a public IP address.
8.8.8.8 → Mountain View, California, United States · ISP: Google LLC
1.1.1.1 → Los Angeles, California, United States · ISP: Cloudflare
Also responds to where is 8.8.8.8 and geolocate 1.1.1.1.
Private IP helper
Detects private/reserved IPs and tells you what range they belong to.
192.168.1.1 → 192.168.1.1 is a private/reserved address in range 192.168.0.0/16 (Class C private)
Reverse DNS
Looks up PTR records.
rdns 8.8.8.8 → dns.google
reverse dns 1.1.1.1 → one.one.one.one
nslookup 8.8.4.4 → dns.google
DNS lookup
Type a bare domain name (or prefix with dig/dns) to see its A, AAAA, MX, and TXT records.
example.com → A 93.184.216.34, MX 0 ., TXT "v=spf1 -all"
dig archlinux.org → A 95.217.163.246, MX 10 mail.archlinux.org, ...
Package versions
Check the latest version of an npm or PyPI package.
latest lodash → lodash 4.17.21
npm express → express 4.19.2
pip requests version → requests 2.31.0 — Python HTTP for Humans
pypi flask → Flask 3.0.2 — ...
Radio stations
Find internet radio stations by location or genre.
radio tokyo → [list of stations with inline audio players]
fm berlin → [list of stations with inline audio players]
Results come from radio-browser.info.
Translation
Translate text explicitly or automatically.
translate hello world to french → bonjour le monde
translate こんにちは to english → hello
Auto-translation: if you search in a language that isn’t in your known_languages list, 4got translates the query and shows the translation in an oracle box above the results.
Translation backends rotate by priority: DeepL, Papago (CJK), Lingva, NLLB, Microsoft. Rate limits are per-cookie. See translation.kdl for configuration.
Wolfram Alpha
Queries the Wolfram Alpha Short Answers API. Requires an API key (server-level or per-user via the apikey_wolframalpha cookie).
population of france → 67.75 million people (2023)
distance earth to mars → 225 million km (average)
Trivial/conversational queries (greetings, etc.) are suppressed. Negative results are cached for 30 days to avoid redundant API calls.
DuckDuckGo Instant Answers
Returns definitions, abstracts, and factual answers from DuckDuckGo’s Instant Answer API.
define serendipity → [Definition from Wiktionary]
albert einstein → [Wikipedia abstract]
DDG IA and Wolfram are tried in that order. The first one that returns a non-empty answer wins.
AI research agent
When you search the same query twice (or click the “Research” button), 4got activates the AI research agent. This is a tool-using LLM that:
- Reads the top search results
- Fetches and analyzes the linked pages
- Runs code in a sandboxed Lua environment if needed
- Streams findings to you in real time via the oracle box
The agent shows an ETA timer and intermediate results as it works. When done, the final synthesized answer replaces the stream.
The research agent requires server-side configuration (see ai-research.md). Operators choose the LLM provider (Cerebras, OpenRouter, Ollama, or any OpenAI-compatible API) and can restrict access to specific user groups.
Settings
Visit /settings to customize your search experience. All preferences are stored as cookies (nothing server-side unless you enable history).
| Setting | Cookie | What it does |
|---|---|---|
| Safe search | safesearch | 1 (on) or 0 (off) |
| Temperature unit | temp_unit | C or F |
| Region | region | ISO country code hint sent to engines (e.g. us, de, jp) |
| Engine badges | badges | always (show source badge on every result), multiple (only when >1 engine), never |
| Fanout hard window | hard_window | Maximum ms to wait for engine responses (default 1000) |
| Fanout idle timeout | idle_timeout | Stop waiting if no new results arrive in this many ms (default 500) |
| Cache max age | cache_max_age | Cache TTL in minutes (default 1440) |
| Priority engine | priority_engine | Engine that must respond before results are shown |
| NSFW banners | nsfw_banners | Show NSFW banners on the landing page |
| History | history_enabled | Enable/disable search history recording |
| Browser history | browser_history | Store search history in browser localStorage |
| Server history | server_history | Store search history on the server |
| Share searches | share_searches | Allow searches to appear in the trending/live feed |
| Live feed | live_feed | Show live search feed on the trending page |
| Autocomplete | autocomplete | Enable/disable search suggestions (disable by setting to off) |
| Show scores | show_scores | Show ranking scores next to results |
| Prefer indie | prefer_indie | Boost results from independent/small web sites |
| Progressive SSE | progressive_sse | Stream results as they arrive via server-sent events |
| Speedtest graph | speedtest_graph | Show engine response time graph |
| Smart tab oracles | smart_tab_oracles | Show oracle answers in non-web category tabs |
| Show favicons | show_favicons | Display site favicons next to results |
| Search method | search_method | POST or GET for the search form |
| Bookmarks | bookmark_mode | Enable/disable bookmark saving |
| Show filters | show_filters | Show time/region filter bar on results page |
| Disabled engines | disabled_<category> | Per-category comma-separated engine IDs to skip |
| Engine weights | weight_<engine> | Override ranking weight per engine (higher = more influence) |
| URL rewrites | rewrite_<service> | Per-service hostname overrides (YouTube, Reddit, Twitter, etc.) |
| Archive links | archive_links | Show Wayback Machine archive links on results |
| Known languages | known_languages | Languages you read, comma-separated ISO 639-1 codes (controls auto-translation) |
| Tracker detection | tracker_detection | Highlight known tracking domains in results |
| Translation privacy | translation_private | Avoid sending translations through third-party APIs |
| Hidden tabs | hidden_tabs | Comma-separated list of category tabs to hide |
| AI stream | ai_stream | Enable/disable streaming for AI research results |
| Research debug | research_debug | Show debug info during AI research |
| Wolfram API key | apikey_wolframalpha | Personal Wolfram Alpha API key (uses server key if empty) |
Settings can also be set via URL parameters on /settings?nsfw=yes®ion=de&... to create a shareable preferences link.
URL rewriting
By default, 4got rewrites result URLs to point at privacy-friendly frontend alternatives:
| Service | Default rewrite |
|---|---|
| YouTube | yewtu.be (Invidious) |
| old.reddit.com | |
| Twitter/X | nitter.net |
| Medium | scribe.rip |
| Imgur | rimgo |
| TikTok | ProxiTok |
| Fandom | BreezeWiki |
Override any rewrite in settings with a custom hostname, or set it to off/none/disabled to keep original URLs.
Privacy features
- No tracking. No analytics scripts, no user profiles, no fingerprinting.
- No accounts. Preferences are stored as cookies in your browser, never on the server.
- Proof-of-work gate. First-time visitors solve a visual captcha (IQ test). Returning visitors solve a client-side PoW challenge. This keeps bots and scraper farms out without requiring JavaScript tracking.
- URL rewriting. Results from major trackers are rewritten to privacy frontends (see above).
- Tracking parameter stripping. UTM tags, fbclid, gclid, and other tracking parameters are removed from all result URLs.
- Image proxy. Images can be proxied through the 4got server to prevent your IP from leaking to image hosts.
- No server-side history by default. Search logging is off unless you explicitly enable it in settings.
- OpenSearch support. Add 4got as a browser search engine via
/opensearch. - RSS feeds. Subscribe to search results via
/feed/search?q=...without visiting the site.
API
4got exposes a JSON API at /api/v1/search?q=QUERY&cat=web. Per-category endpoints also exist: /api/v1/web, /api/v1/images, etc. See /api on your instance for full documentation, or /api/openapi.yaml for the OpenAPI 3.0 spec.
Keyboard shortcuts
- Enter — Search
- Enter twice (same query) — Trigger AI research agent
- Tab — Cycle category tabs
Configuration Reference
4got uses KDL (KDL Document Language) for all configuration. All config files live in the data/ directory.
config.kdl
The main configuration file. All fields have sensible defaults.
| Field | Type | Default | Description |
|---|---|---|---|
port | int | 8888 | HTTP listen port |
host | string | “0.0.0.0” | Listen address |
owner-secret | string | “CHANGE_ME” | Admin password. Also used to derive Ed25519 signing keys for session cookies. |
deluxe-secret | string | “” | Legacy cookie-based deluxe tier secret. Leave empty to disable. |
pow-difficulty | int | 22 | Proof-of-work difficulty (number of leading zero bits in SHA-256 hash) |
captcha-bypass-hours | int | 24 | How long a captcha solution grants session access |
pow-bypass-seconds | int | 300 | How long a PoW solution grants session access |
max-searches-per-day | int | 200 | Per-IP daily search limit |
fanout-hard-window-ms | int | 1000 | Maximum time to wait for engine responses |
fanout-idle-timeout-ms | int | 500 | Stop waiting if no new results arrive within this window |
cache-ttl-web-s | int | 1800 | Cache TTL for web results (seconds) |
cache-ttl-images-s | int | 7200 | Cache TTL for image results |
cache-ttl-news-s | int | 300 | Cache TTL for news results |
prefetch-threshold | int | 2 | Minimum access count before a cache entry is eligible for background prefetch |
prefetch-before-expiry-s | int | 60 | Prefetch entries within this many seconds of expiry |
nsfw-landing | bool | false | Show NSFW banners on the landing page |
data-dir | string | “data” | Path to the data directory (config files, DB, banners, captcha assets) |
db-path | string | “data/4got.db” | Path to the SQLite database |
peers | string[] | [] | Legacy peer URLs (prefer peers.kdl instead) |
peer-secret | string | “” | Shared secret for peer authentication |
self-heal-enabled | bool | false | Enable autonomous self-healing agents for broken engines |
quotes-mode | string | “all” | Landing page quote visibility: “all”, “deluxe”, “off” |
api-keys | block | {} | Third-party API keys (wolframalpha, deepl, freesound, spotify-client-id, spotify-client-secret) |
Environment variable overrides
Every config.kdl field has a corresponding FGOT_* environment variable:
FGOT_PORT,FGOT_HOST,FGOT_OWNER_SECRET,FGOT_DELUXE_SECRETFGOT_POW_DIFFICULTY,FGOT_CAPTCHA_HOURS,FGOT_POW_SECONDSFGOT_MAX_SEARCHES,FGOT_FANOUT_HARD_WINDOW_MS,FGOT_FANOUT_IDLE_TIMEOUT_MSFGOT_CACHE_WEB_S,FGOT_CACHE_IMAGES_S,FGOT_CACHE_NEWS_SFGOT_DATA_DIR,FGOT_DB_PATHFGOT_CONFIG— override the config file path entirely
KDL config files
4got uses KDL (KDL Document Language) for structured subsystem configs. Each subsystem looks for its config in two places:
- A dedicated file (e.g.
data/peers.kdl) - A named section inside any
*.kdlfile indata/(e.g. apeers { ... }block insidedata/conf.kdl)
You can use either approach but not both for the same subsystem. This lets you keep everything in a single conf.kdl or split into per-subsystem files.
peers.kdl
Configures the friend/peer network for cache sharing and service proxying.
// How many friends to wait for before proceeding.
// Float 0-1: proportion of 'up' friends (1.0 = all, 0.5 = half)
// Int >= 1: exact count
// Negative: "ok to miss" semantics (-2 = ok missing 2 friends)
wait-for 1.0
// Store results received from friends in local cache
store-friend-cache true
// Mark a friend as 'down' if they fail more than this proportion
// of requests over this time window.
failure-threshold 0.01 1d
// Services this instance allows friends to proxy through us
allow-proxy "translation" "wolfram" "autocomplete"
// Always wait for a specific friend (by name)
// always-wait-for "alice"
// Prefer a friend for specific services
// prefer-friend "alice" for="translation,wolfram"
peer "instance-b" {
url "http://127.0.0.1:8890"
// Optional mTLS:
// cert "/path/to/client.crt"
// key "/path/to/client.key"
}
groups.kdl
Defines user tiers/groups and their permissions.
group "deluxe" {
allow "sidecar" "server_api_keys" "friend_proxy" "priority_fanout"
label "deluxe"
}
group "researcher" {
allow "server_api_keys"
label "researcher"
}
Known permissions: sidecar, server_api_keys, friend_proxy, priority_fanout. Groups are assigned to users by cookie hash via the admin panel (/admin/users).
captcha.kdl
Configures the pre-rendered captcha (IQ test) pool.
pool {
distorted-target 500 // target pool size for distorted tiles
distorted-batch 50 // generate this many per cycle
prebuilt-target 100 // target pool size for assembled captchas
prebuilt-batch 20 // assemble this many per cycle
cycle-count 25 // number of generation cycles at startup
cycle-interval-min 5 // minutes between background replenishment cycles
}
autocomplete.kdl
Configures autocomplete suggestion backends and their weights.
backend "duckduckgo" weight=1 enabled=true
backend "brave" weight=1 enabled=true
backend "qwant" weight=1 enabled=true
backend "yahoo" weight=1 enabled=true
backend "wikipedia" weight=1 enabled=true
backend "google" weight=1 enabled=false
Higher weight = more influence when merging suggestions from multiple backends.
translation.kdl
Configures translation backends with priority routing and per-user rate limits.
// Language sets (non-overlapping)
set "cjk" "ja" "zh" "ko"
set "eu" "de" "fr" "nl" "pl" "es" "pt" "it" "ro" "sv" "da" "no" "fi" "cs" "hu" "el"
// Backends: priority (higher=tried first), languages ("all" or set name),
// limit-per-cookie, reset hours
backend "deepl" priority=3 languages="eu" limit=500 reset-hours=24
backend "papago" priority=3 languages="cjk" limit=50 reset-hours=24
backend "lingva" priority=2 languages="all" limit=100 reset-hours=24
backend "nllb" priority=1 languages="all" limit=0
backend "microsoft" priority=2 languages="all" limit=30 reset-hours=24
instances.kdl
Configures the instance browser (shows other 4got instances).
fetch-interval-min 60
friends {
// instance "https://search.friend.example.com"
}
public {
// instance "https://public.4got.example.com"
}
ai-research.kdl
Configures the AI research agent that synthesizes search results.
enabled true
mode "lua" // "lua" (sandboxed) or "bash" (dangerous)
provider "cerebras"
model "llama3.1-8b"
api-key "csk-..."
base-url "https://api.cerebras.ai/v1"
max-runtime-seconds 60
require "deluxe" // group permission required
See ai-research.md for full details.
image-proxy.kdl (optional)
Configures the SQLite-backed image proxy cache.
enabled true
hold-time-days 30 // evict images unserved for this many days
jxl-reencode true // reencode to JXL (requires cjxl binary)
redownload-threshold 999 // re-fetch original after N serves
require "deluxe" // group permission required ("" for everyone)
The unified conf.kdl option
Instead of separate files, you can put everything in a single data/conf.kdl:
peers {
wait-for 1.0
peer "instance-b" {
url "http://127.0.0.1:8890"
}
}
groups {
group "deluxe" {
allow "sidecar" "server_api_keys"
label "deluxe"
}
}
translation {
backend "lingva" priority=2 languages="all" limit=100 reset-hours=24
}
The resolver scans all *.kdl files for matching section blocks. It is an error to have both peers.kdl and a peers { ... } section in another file.
Per-user cookie settings
Users can customize their experience via cookies set on the /settings page:
| Cookie | Values | Description |
|---|---|---|
safesearch | 1, 0 | Safe search filter (1 = on, 0 = off) |
temp_unit | C, F | Temperature unit for weather oracle |
region | ISO country code | Region hint for engines |
badges | always, multiple, never | Engine badge display mode |
hard_window | integer ms | Maximum time to wait for engine responses (default 1000) |
idle_timeout | integer ms | Stop waiting if no new results in this window (default 500) |
cache_max_age | integer minutes | Cache TTL (default 1440) |
priority_engine | engine ID or “none” | Engine that must respond before results are shown |
disabled_web, disabled_images, etc. | comma-separated engine IDs | Disable specific engines per category |
weight_<engine> | integer >= 1 | Override engine weight in result ranking |
rewrite_<service> | hostname, “off”, “none”, “disabled” | Override URL rewriting per service |
known_languages | comma-separated ISO 639-1 codes | Languages the user knows (for auto-translation) |
apikey_wolframalpha | API key string | Personal Wolfram Alpha API key |
history_enabled | “1” | Enable search history recording |
autocomplete | “off” to disable | Enable autocomplete suggestions |
bookmark_mode | on, off | Enable bookmark saving |
archive_links | on, off | Show archive links on results |
show_scores | “1” | Show ranking scores next to results |
prefer_indie | “1” | Boost results from independent sites |
search_method | POST, GET | HTTP method for search form |
hidden_tabs | comma-separated tab names | Category tabs to hide |
tracker_detection | “1” | Highlight known tracking domains |
Engine KDL Specification
This document specifies the KDL format for defining generic engines in data/engines.kdl. Generic engines are parsed by go-server/generic_engine.go and registered alongside hand-written engines (Brave, Google, etc.) at startup. A single engines.kdl file defines all ~274 generic engines.
Engine block structure
Each engine is a top-level engine node with a quoted string ID and a child block of properties:
engine "my_engine" {
name "My Engine Display Name"
categories "web" "news"
search-url "https://example.com/search?q={query}&page={page}"
method "GET"
result-type "json"
results-path "data.items"
title-path "title"
url-path "link"
content-path "description"
page-size 20
paging true
first-page 1
}
The engine ID (the string after engine) is the unique identifier used in logs, cache keys, weight cookies, and the admin UI. It must be a short slug (lowercase, underscores allowed).
Attributes reference
Identity
| Attribute | Type | Required | Default | Description |
|---|---|---|---|---|
name | string | No | Same as ID | Human-readable display name shown in engine badges and the settings page. |
categories | string list | Yes | (none – engine is skipped if empty) | Space-separated quoted category slugs. Valid values: web, images, videos, news, music, science, it, files, maps, social, books, qa / q&a, packages, weather, lyrics. An engine with no parseable categories is silently skipped during registration. |
Request
| Attribute | Type | Required | Default | Description |
|---|---|---|---|---|
search-url | string | Yes | – | URL template. Placeholders: {query} (URL-encoded query), {page} / {pageno} (page number = (page-1) + first-page), {offset} (offset = (page-1) * page-size + first-page), {lang} (hardcoded "en" currently). |
method | string | No | "GET" | HTTP method. "GET" or "POST". |
body-template | string | No | "" | POST body template (same placeholders as search-url). Only used when method is "POST". |
headers | sub-block | No | (none) | Custom HTTP headers as key-value pairs. Default headers (User-Agent, Accept, Accept-Language, DNT) are always sent; entries here override or add to them. |
Response parsing – HTML mode
Used when result-type is "html" (or "HTML" or "xpath").
| Attribute | Type | Required | Default | Description |
|---|---|---|---|---|
result-type | string | Yes | – | Set to "html" for CSS-selector-based parsing. |
results-selector | string | Yes (for HTML) | "" | CSS selector for result container elements. Each match becomes one result. |
title-selector | string | No | "" | CSS selector (relative to result container) for the title text. |
url-selector | string | No | "" | CSS selector for the URL. Extracts href attribute first; falls back to text content. |
content-selector | string | No | "" | CSS selector for the snippet/description text. |
thumbnail-selector | string | No | "" | CSS selector for a thumbnail image URL (used for image/video results). |
Response parsing – JSON mode
Used when result-type is "json".
| Attribute | Type | Required | Default | Description |
|---|---|---|---|---|
result-type | string | Yes | – | Set to "json" for dot-path-based JSON parsing. |
results-path | string | No | "" (top-level array) | Dot-separated path to the results array in the JSON response. Example: "message.items" navigates {"message":{"items":[...]}}. |
title-path | string | No | "" | Dot-path to the title field within each result object. |
url-path | string | No | "" | Dot-path to the URL field. |
content-path | string | No | "" | Dot-path to the snippet/description field. |
thumbnail-path | string | No | "" | Dot-path to a thumbnail URL field. |
URL construction
| Attribute | Type | Required | Default | Description |
|---|---|---|---|---|
url-prefix | string | No | "" | String prepended to every extracted URL. Use for APIs that return relative paths (e.g. "https://developer.mozilla.org" + /en-US/docs/...). |
url-template | string | No | "" | Template for building URLs from result fields. Placeholders: {title} (spaces replaced with _), {id}, {url}. Takes precedence over url-prefix when set. Example: "https://www.artic.edu/artworks/{id}". |
Pagination
| Attribute | Type | Required | Default | Description |
|---|---|---|---|---|
page-size | int | No | 10 | Results per page. Used to compute {offset}. |
paging | bool | No | false | Whether this engine supports pagination. |
first-page | int | No | 0 | The page/offset number for the first page. Most APIs use 0 or 1. |
Content processing
| Attribute | Type | Required | Default | Description |
|---|---|---|---|---|
content-html-to-text | bool | No | false | Strip HTML tags from the content/snippet field before display. Useful for APIs like StackExchange that return HTML in JSON. |
Lifecycle
| Attribute | Type | Required | Default | Description |
|---|---|---|---|---|
disabled | bool | No | false | If true, the engine is parsed but not registered. Use to temporarily disable an engine without removing its config. |
Federation
| Attribute | Type | Required | Default | Description |
|---|---|---|---|---|
share | string | No | auto-detected | Controls whether results from this engine are shared with peer instances. Values: "none" (never share – used for engines with API keys to avoid leaking quota), "cache" (share cached results but don’t proxy live queries – the default), "proxy" (full proxy: friends can trigger live queries through us). When omitted, the engine factory auto-detects: if the URL or headers contain an API key pattern (api_key=, Authorization, etc.), defaults to "none"; otherwise defaults to "cache". |
tor-outbound | bool | No | false | Route this engine’s HTTP requests through the local Tor SOCKS5 proxy (127.0.0.1:9050). Use for engines that aggressively IP-ban search aggregators. Requires a running Tor daemon. |
Examples
JSON API engine (MDN)
engine "mdn" {
name "MDN Web Docs"
categories "web"
search-url "https://developer.mozilla.org/api/v1/search?q={query}&page={page}"
method "GET"
result-type "json"
results-path "documents"
title-path "title"
url-path "mdn_url"
url-prefix "https://developer.mozilla.org"
content-path "summary"
page-size 10
paging true
first-page 1
}
HTML scraping engine (Anaconda)
engine "anaconda" {
name "Anaconda"
categories "web"
search-url "https://anaconda.org/search?q={query}&page={page}"
method "GET"
result-type "html"
url-prefix "https://anaconda.org"
results-selector "tbody tr"
title-selector "td h5"
url-selector "td h5 a:last-of-type"
content-selector "td p"
page-size 10
paging true
first-page 0
}
Engine with custom headers (Crossref)
engine "crossref" {
name "Crossref"
categories "web"
search-url "https://api.crossref.org/works?query={query}&rows=20&offset={offset}"
method "GET"
result-type "json"
results-path "message.items"
title-path "title"
url-path "URL"
content-path "abstract"
content-html-to-text true
headers {
"User-Agent" "4got/1.0 (https://4got.org; mailto:admin@4got.org)"
}
page-size 20
paging true
first-page 0
}
POST engine with body template (ArtStation)
engine "artstation" {
name "www.artstation.com"
categories "images"
search-url "https://www.artstation.com/api/v2/search/projects.json"
method "POST"
result-type "json"
results-path "data"
title-path "title"
url-path "permalink"
content-path "description"
thumbnail-path "smaller_square_cover_url"
body-template "{\"query\":\"{query}\",\"page\":{page},\"per_page\":20,\"sorting\":\"relevance\",\"pro_first\":\"1\",\"filters\":[],\"additional_fields\":[\"description\"]}"
page-size 20
paging true
first-page 1
}
URL template engine (Art Institute of Chicago)
engine "artic" {
name "www.artic.edu"
categories "images"
search-url "https://api.artic.edu/api/v1/artworks/search?q={query}&page={page}&limit=20"
method "GET"
result-type "json"
results-path "data"
title-path "title"
url-path "id"
url-template "https://www.artic.edu/artworks/{id}"
content-path "thumbnail.alt_text"
page-size 20
paging true
first-page 1
}
How the generic engine factory works
At startup, LoadAndRegisterGenericEngines() in generic_engine.go:
- Reads
data/engines.kdlviaresolveKDLSection()(which supports both standalone files andengines { ... }sections inside a unified KDL file). - Parses the raw bytes into
[]GenericEngineConfigviaparseEngineKDL(), a line-by-line KDL parser that handles engine blocks, headers sub-blocks, and all property types. - Skips entries with empty ID, no categories, or
disabled true. - Sets defaults:
page-sizedefaults to 10,namedefaults to the ID. - Creates a
GenericEnginestruct (which implements theEngineinterface) for each config and registers it in the global engine registry. - Determines the share policy: explicit
shareattribute wins; otherwise auto-detects based on whether the URL or headers contain API key patterns.
When Search() is called on a GenericEngine:
- Substitutes
{query},{page},{offset},{lang}in the URL template (and body template if POST). - Makes the HTTP request, routing through Tor if
tor-outboundis set. - Records the raw HTML for drift detection (
drift.go). - Dispatches to
parseHTML()(goquery CSS selectors) orparseJSON()(dot-path navigation) based onresult-type. - Returns
[]WebResultwith each result’sEnginefield set to the engine ID.
Testing a new engine via the admin playground
The admin playground at /admin/engines/new (owner-only) provides a web form for configuring and testing generic engines without editing KDL files directly.
- Log in as owner at
/admin/login. - Navigate to
/admin/engines/new. - Fill in the engine identity (ID, display name, categories).
- Set the search URL template with
{query}and{page}placeholders. - Choose HTML or JSON result type and fill in the corresponding selectors/paths.
- Enter a test query and click “Test Engine” – this POSTs to
/admin/engines/test, which creates a temporaryGenericEnginein memory, runs a search, and returns results as JSON displayed in the form. - Iterate on selectors until results look correct.
- Click “Save Engine” to append the generated KDL block to
data/engines.kdland register it in memory immediately (no restart needed).
The test endpoint does not persist anything – only “Save Engine” writes to disk.
Engine Catalog
4got ships with 274 engine configurations in data/engines.kdl plus 18 handwritten engines in Go. Engines are grouped by category. Most require no API key; exceptions are noted.
Disabled engines are included but inactive by default. Enable them by setting disabled false in engines.kdl.
Handwritten engines (Go)
These are implemented directly in Go for speed, reliability, or because they need custom scraping logic.
| Engine | ID | Categories | Notes |
|---|---|---|---|
| Brave | brave | web, images, news | HTML scraping |
| DuckDuckGo | duckduckgo | web | HTML scraping |
google | web | HTML scraping | |
| Startpage | startpage | web | HTML scraping |
| Wikipedia | wikipedia | web | MediaWiki API |
| Mojeek | mojeek | web | HTML scraping |
| Marginalia | marginalia | web | JSON API |
| Wiby | wiby | web | JSON API |
| Hacker News | hackernews | news | Algolia API |
| YouTube | youtube | videos | HTML scraping |
| SoundCloud | soundcloud | music | JSON API |
| Flickr | flickr | images | JSON API |
| Pexels | pexels | images | JSON API |
| Yandex | yandex | web, images | HTML scraping |
| Qwant | qwant | web, images, news | JSON API |
| Bing | bing | web, images, news, videos | HTML scraping |
| Baidu | baidu | web | HTML scraping |
| Yahoo | yahoo | web, news | HTML scraping |
Sidecar engines (Python + Playwright)
These require the Python sidecar for JavaScript rendering:
| Engine | What it handles |
|---|---|
| Google (sidecar) | JS-heavy Google SERPs |
| Qwant (sidecar) | Qwant’s SPA rendering |
| Yandex (sidecar) | Yandex captcha bypass |
KDL-configured engines by category
Web (159 engines)
General web search. The largest category.
| Engine | ID | Notes |
|---|---|---|
| MDN Web Docs | mdn | JSON API |
| StackExchange | stackexchange | API, Q&A results |
| Open Library | openlib | Book search |
| npm | npm | Package registry |
| Anaconda | anaconda | HTML scraping |
| Arch Wiki | archwiki | REST API |
| Wikidata | wikidata | JSON API |
| Crossref | crossref | Academic DOIs |
| Apple App Store | apple_app_store | iTunes API |
| Boardreader | boardreader | Forum search |
| bpb.de | bpb | German civic education |
| CachyOS | cachy_os | |
| crates.io | crates | Rust packages |
| Docker Hub | docker_hub | |
| GitHub | github | Repo search |
| GitHub Code | github_code | Code search |
| GitHub Issues | github_issues | |
| GitHub Topics | github_topics | |
| GitLab | gitlab | |
| GitLab Snippets | gitlab_snippets | |
| Grokipedia | grokipedia | |
| Hacker News | hackernews | Algolia API |
| hex.pm | hex | Elixir packages |
| Hugging Face | huggingface | AI models |
| Lemmy | lemmy | Fediverse forums |
| Mastodon | mastodon | Fediverse posts |
| Microsoft Learn | microsoft_learn | |
| mwmbl | mwmbl | Open-source index |
| NVD | nvd | CVE/vulnerability DB |
| OpenAlex | openalex | Academic |
| OpenLibrary | openlibrary | |
| Pirate Bay | piratebay | Torrents |
reddit | ||
| tagesschau.de | tagesschau | German news (web+news) |
| Tootfinder | tootfinder | Mastodon search |
| Void Linux | voidlinux | |
| Wikipedia | wikipedia | |
| Simple Wikipedia | simple_wikipedia | |
| Wikibooks | wikibooks | |
| Wikinews | wikinews | |
| Wikiquote | wikiquote | |
| Wikisource | wikisource | |
| Wikivoyage | wikivoyage | |
| Wiktionary | wiktionary | |
| Wikimedia Commons | wikicommons | |
| Internet Archive | internet_archive | |
| LibreTranslate | libre_translate | |
| PubMed | pubmed | |
| RubyGems | rubygems | |
| CTAN | ctan | LaTeX packages |
| cdnjs | cdnjs | JS CDN |
| pub.dev | pub_dev | Dart packages |
| CocoaPods | cocoapods | iOS packages |
| OpenBSD Ports | openbsd_ports | |
| TVmaze | tvmaze | TV shows |
| TMDB | tmdb | Movies |
| OpenLibrary Books | openlibrary_search | |
| LibriVox | librivox | Audiobooks |
| Project Gutenberg | gutenberg | Free ebooks |
| Giphy | giphy | GIFs; uses public API key |
| OpenStreetMap | openstreetmap | Nominatim |
| Photon | photon | Geocoder |
| Open-Meteo | open_meteo | Weather data |
| Alpine Packages | alpine_packages | |
| Homebrew | homebrew | Disabled |
| Flathub | flathub | |
| Snapcraft | snapcraft | |
| YTS | yts | Movie torrents |
| Discogs | discogs | |
| Last.fm | lastfm | |
| Urban Dictionary | urban_dictionary | |
| DDG Definitions | duckduckgo_definitions | |
| SVGRepo | svgrepo | |
| Steam | steam | Games |
| PyPI | pypi | Python packages |
| 1337x | 1337x | Torrents |
| APKMirror | apkmirror | Android APKs |
| eBay | ebay | |
| F-Droid | fdroid | FOSS Android apps |
| Solid Torrents | solidtorrents | |
| Wordnik | wordnik | Dictionary |
| fyyd | fyyd | Podcasts; disabled |
| Podcast Index | podcast_index | |
| Semantic Scholar | semantic_scholar | Academic |
| SourceHut | sourcehut | |
| Gitea | gitea | |
| lib.rs | librs / lib_rs | Rust crate search |
| Lobsters | lobste_rs | |
| Packagist | packagist | PHP packages |
| Radio Browser | radio_browser | |
| Dailymotion | dailymotion | |
| Bandcamp | bandcamp | |
| Stack Overflow Tags | stackoverflow_main | |
| Anaconda Cloud | anaconda_cloud | |
| arXiv | arxiv | Academic preprints |
| DBLP | dblp | CS bibliography |
| IA Scholar | internet_archive_scholar | |
| Nyaa | nyaa | Anime torrents |
| Library Genesis | libgen | |
| Codeberg | codeberg | |
| pkg.go.dev | pkg_go_dev | Go packages |
| Hackage | hackage | Haskell packages |
| NuGet | nuget | .NET packages |
| Maven | maven | Java packages |
| pkgs.org | pkgs_org | Cross-distro package search |
| Libraries.io | libraries_io | Multi-registry |
| Anna’s Archive | annas_archive | |
| Z-Library | z_library | |
| Metacritic | metacritic | |
| Rotten Tomatoes | rotten_tomatoes | |
| Letterboxd | letterboxd | |
| AniDB | anidb | |
| VNDB | vndb | Visual novels |
| PCGamingWiki | pcgamingwiki | |
| IsThereAnyDeal | isthereanydeal | Game prices |
| BoardGameGeek | boardgamegeek | |
| WordReference | wordreference | |
| Linguee | linguee | |
| DeepL | deepl | Disabled |
| Brave Goggles | brave_goggles | |
| SearX | searx | Meta-meta-search |
| Whoogle | whoogle | |
| Stract | stract | |
| Yep | yep | |
| Rightdao | right_dao | |
| Presearch | presearch | |
| Alexandria | alexandria | |
| Petal Search | petalsearch | |
| Swisscows | swisscows | |
| MetaGer | metager | |
| eTools | etools | |
| Dogpile | dogpile | |
| Gibiru | gibiru | |
| searx.space | searx_space | Instance list |
| Monocles | monocles | |
| Mullvad Leta | mullvad_leta | |
| 4get.ca | 4get_ca | Sibling metasearch |
| Emojipedia | emojipedia | |
| Color Hex | colorhex | |
| AlternativeTo | alternativeto | |
| Goodreads | goodreads | |
| IMDB | imdb_search | |
| itch.io | itch_io | Indie games |
| Are.na | are_na | |
| Kickstarter | kickstarter | |
| Product Hunt | product_hunt | |
| IEEE Xplore | ieee | |
| Springer | springer_open | |
| CORE | core | Academic |
| DOAJ | doaj | Open access journals |
| Zenodo | zenodo | Research data |
| ORCID | orcid | |
| Unpaywall | unpaywall | Open access finder |
| OSF | osf | Preprints |
| ANN Anime | ann | Disabled |
| Kitsu | kitsu | Anime/manga |
| MyAnimeList | jikan | Via Jikan API |
| RAWG | rawg | Games |
| MangaDex | mangadex | |
| Open Food Facts | openfoodfacts | |
| MusicBrainz | musicbrainz | |
| OpenAIRE | openairedatasets | |
| Exchange Rate | exchangerate | |
| Ecosia | ecosia | Disabled |
| Mojeek | mojeek | Disabled |
| Marginalia | marginalia | Disabled |
| Etsy | etsy | |
| CRAN | cran | R packages; disabled |
| CPAN | cpan | Perl packages |
| Conda-Forge | conda_forge | Disabled |
| Repology | repology | Disabled |
| HowLongToBeat | howlongtobeat | Disabled |
| IGDB | igdb | Disabled |
| BASE Academic | base_search | |
| Fatcat Scholar | fatcat | |
| Google Scholar | google_scholar | Disabled |
Images (15 engines)
| Engine | ID | Notes |
|---|---|---|
| Art Institute of Chicago | artic | |
| ArtStation | artstation | POST API |
| findthatmeme | findthatmeme | Meme search |
| Openverse | openverse | CC-licensed |
| Pixabay | pixabay | |
| Adobe Stock | adobe_stock | |
| Unsplash | unsplash | |
| Wallhaven | wallhaven | |
| Pexels | pexels | |
| Flickr | flickr | |
| Wikimedia Commons | wikicommons_images | |
| Giphy | giphy | |
| SVGRepo | svgrepo | Vector graphics |
| DeviantArt | deviantart | Disabled |
| Pixiv | pixiv | Disabled |
Videos (13 engines)
| Engine | ID | Notes |
|---|---|---|
| Bilibili | bilibili | Chinese video |
| BitChute | bitchute | |
| CCC Media | ccc_media | Chaos Computer Club |
| Dailymotion | dailymotion | |
| IMDb | imdb | |
| Internet Archive Video | archive_video | |
| Invidious | invidious | YouTube frontend |
| Odysee | odysee | LBRY |
| PeerTube | peertube | Fediverse video |
| Vimeo | vimeo | |
| YouTube (no API) | youtube_noapi | Disabled |
| Rumble | rumble | Disabled |
| 360kan | 360search_videos | Disabled |
News (10 engines)
| Engine | ID | Notes |
|---|---|---|
| Yahoo News | yahoo_news | |
| Hacker News | hackernews_news | |
| Reuters | reuters | |
| tagesschau.de | tagesschau | German; also in web |
| Wikinews | wikinews | |
| The Guardian | the_guardian | Uses test API key |
| ANSA News | ansa | Italian wire service |
| AP News | ap_news | Disabled |
| Bing News | bing_news | Disabled |
| Google News | google_news | Disabled |
Music (13 engines)
| Engine | ID | Notes |
|---|---|---|
| Bandcamp | bandcamp | |
| Deezer | deezer | |
| Discogs | discogs | |
| Genius | genius | |
| Jamendo | jamendo | CC music |
| Last.fm | lastfm | |
| LibriVox | librivox | Audiobooks |
| Mixcloud | mixcloud | |
| MusicBrainz | musicbrainz | |
| Radio Browser | radio_browser | Internet radio |
| SoundCloud | soundcloud | |
| YouTube (no API) | youtube_noapi | Also videos; disabled |
| Freesound | freesound | Disabled; needs API key |
Science (12 engines)
| Engine | ID | Notes |
|---|---|---|
| arXiv | arxiv | |
| BASE | base_search_sci | Bielefeld Academic |
| CORE | core_sci | |
| Crossref | crossref_sci | |
| dblp | dblp | CS papers |
| DOAJ | doaj_sci | Open access |
| IA Scholar | internet_archive_scholar | |
| OpenAlex | openalex_sci | |
| PubMed | pubmed_sci | |
| Semantic Scholar | semantic_scholar_sci | |
| Zenodo | zenodo_sci | |
| Google Scholar | google_scholar | Disabled |
IT (15 engines)
| Engine | ID | Notes |
|---|---|---|
| Codeberg | codeberg | |
| crates.io | crates_io | Rust |
| Docker Hub | docker_hub_it | |
| GitHub Repos | github_repos_it | |
| GitLab | gitlab_it | |
| hex.pm | hex_it | Elixir |
| MDN | mdn_it | |
| npm | npm_search | |
| NuGet | nuget_it | .NET |
| Packagist | packagist_it | PHP |
| pkg.go.dev | pkg_go_dev | Go |
| pub.dev | pub_dev_it | Dart |
| PyPI | pypi_packages | Python |
| RubyGems | rubygems_it | |
| Stack Overflow | stackoverflow |
Files (9 engines)
| Engine | ID | Notes |
|---|---|---|
| 1337x | 1337x_files | |
| Anna’s Archive | annas_archive_files | |
| Library Genesis | libgen | |
| Nyaa | nyaa | Anime torrents |
| Pirate Bay | piratebay_files | |
| Solid Torrents | solidtorrents_files | |
| YTS | yts_files | Movie torrents |
| BT4G | bt4g | Disabled |
| TorrentGalaxy | torrentgalaxy | Disabled |
Social (6 engines)
| Engine | ID | Notes |
|---|---|---|
| Hacker News | hackernews_social | |
| Lemmy | lemmy_social | |
| Lobsters | lobsters_social | |
| Mastodon | mastodon_social | |
reddit_social | ||
| Tootfinder | tootfinder_social |
Books (4 engines)
| Engine | ID | Notes |
|---|---|---|
| Goodreads | goodreads_books | |
| Internet Archive | internet_archive_books | |
| Open Library | openlibrary_books | |
| Project Gutenberg | gutenberg_books |
Q&A (6 engines)
| Engine | ID | Notes |
|---|---|---|
| Ask Ubuntu | askubuntu_qa | |
| Math SE | math_se_qa | |
| Server Fault | serverfault_qa | |
| StackExchange | stackexchange_qa | |
| Super User | superuser_qa | |
| Unix & Linux SE | unix_se_qa |
Packages (9 engines)
| Engine | ID | Notes |
|---|---|---|
| Alpine Packages | alpine_packages_pkg | |
| Arch Wiki | archwiki_packages | |
| AUR | aur | |
| Flathub | flathub_pkg | |
| Gentoo Wiki | gentoo_wiki | |
| NixOS Packages | nixos_packages | |
| Void Linux | void_packages_pkg | |
| Homebrew | homebrew_pkg | Disabled |
| Repology | repology | Disabled |
Maps (2 engines)
| Engine | ID | Notes |
|---|---|---|
| OpenStreetMap (Nominatim) | nominatim_maps | |
| Photon Geocoder | photon_maps |
Weather (2 engines)
| Engine | ID | Notes |
|---|---|---|
| wttr.in | wttr_in | |
| Open-Meteo | open_meteo_weather |
Lyrics (1 engine)
| Engine | ID | Notes |
|---|---|---|
| Genius Lyrics | genius_lyrics |
Oracles
Oracles are instant-answer widgets shown above search results. They trigger on specific query patterns and provide rich, interactive answers without requiring a click-through. Oracles are checked in CheckOracles() in oracles.go. Access can be restricted per-group via oracle-permissions.kdl.
Local oracles (no external calls)
These run entirely on the server with no network requests. They are mutually exclusive — the first match wins and skips all remaining oracles.
| Oracle | Permission key | Trigger patterns | What it shows |
|---|---|---|---|
| Calculator | calculator | 2+2, sqrt(16), 15% of 200, sin(1.5), 2^8 | Arithmetic, trig, logarithms, percentage calculations |
| Unit Converter | unit_converter | 10 km to miles, 72 fahrenheit to celsius, 5 gallons to liters, 100 fl oz to ml | Length, weight, volume, speed, data storage, cooking units, temperature, force/mass |
| Base Converter | base_converter | 0xff in decimal, 255 in hex, 255 in binary, 0b1010 in decimal, 0o77 in hex | Decimal, hex, binary, octal conversions |
| Color | color | #FF5500, #f50, rgb(255,85,0), hsl(20,100%,50%) | Color swatch with hex/RGB/HSL conversions |
| Private IP | private_ip | Bare IPv4/IPv6 addresses in private ranges (192.168.1.1, 10.0.0.1) | Identifies the private range and offers HTTP/HTTPS links |
| Timezone | timezone | time in tokyo, time in new york, time in UTC+5 | Current time in 70+ cities and IANA timezones |
| UUID | uuid | Any UUID string (e.g. 550e8400-e29b-41d4-a716-446655440000) | Version, variant, and embedded timestamp (v1/v7) |
| Timestamp | timestamp | 10-digit or 13-digit numbers, now, current timestamp, unix timestamp | Converts Unix timestamps to human dates, or shows current timestamp |
| JWT | jwt | Any header.payload.signature JWT token | Decodes header and payload, shows algorithm and expiry |
| CIDR | cidr | 192.168.1.0/24, 10.0.0.0/8 | Network, broadcast, host range, usable hosts, netmask |
| Hash/Base64 | hash | MD5 (32 hex), SHA-1 (40 hex), SHA-256 (64 hex), SHA-512 (128 hex), or base64 strings (40+ chars) | Identifies hash type, or decodes base64 to text |
| Cron | cron | 5-field cron expressions like 0 */2 * * 1-5 | Human-readable schedule description with per-field explanation |
| DOI | doi | 10.1000/xyz123 or doi.org/10.1000/xyz123 | Links to publisher (doi.org) and Sci-Hub |
| Timer | timer | timer 5 minutes, timer 30s, countdown 10m, stopwatch | Interactive countdown timer or stopwatch widget |
| Random | random | flip a coin, roll a die, roll 3d20, random 1 to 100 | Coin flip, dice rolls (NdX), random number in range (crypto/rand) |
| Password | password | password, generate password, password 32 chars | Cryptographically random password with copy button (4-256 chars) |
| QR Code | qr | qr https://example.com, qr code hello world | Inline SVG QR code (up to ~271 bytes, versions 1-10) |
| Chmod | chmod | chmod 755, chmod rwxr-xr-x | Numeric-to-symbolic and symbolic-to-numeric conversion with explanation |
| Roman Numeral | roman | XIV in decimal, 42 in roman | Roman numeral to/from decimal conversion (1-3999) |
| Date Arithmetic | date_arithmetic | days between Jan 1 and Dec 31, 30 days from now, days until Christmas, 90 days ago | Day counts, future/past dates, with IFC calendar dates |
Network oracles (external API calls, no key required)
These make external network requests but do not require any API key.
| Oracle | Permission key | Trigger patterns | What it shows | API used |
|---|---|---|---|---|
| Reverse DNS | reverse_dns | rdns 8.8.8.8, reverse dns 1.1.1.1, nslookup 8.8.4.4, ptr <IP> | PTR records for an IP address | System DNS resolver |
| DNS Lookup | dns_dig | example.com, dig example.com, dns example.com | A, AAAA, MX, TXT records | System DNS resolver |
| IP Geolocation | ip_geo | Bare public IPv4 like 8.8.8.8, where is 1.1.1.1, geolocate <IP> | City, region, country, ISP, ASN, PTR | ipapi.co |
| My IP | my_ip | my ip, whats my ip, ip, show my ip | User’s public IP with geolocation and reverse DNS | ipapi.co |
| Currency | currency | 100 usd to eur, 50 gbp in jpy, usd to eur | Exchange rate with forward and inverse rates | fawazahmed0/exchange-api (jsDelivr CDN) |
| WHOIS | whois | whois example.com | Domain registrar, creation/expiry dates, nameservers | RDAP (rdap.org) |
| Speedtest | speedtest | speedtest, speed test, internet speed, bandwidth test | Interactive download/upload speed test with latency and jitter | Server’s own /api/v1/speedtest/* endpoints |
| Benchmark | benchmark | benchmark, browser benchmark, system benchmark | JS/DOM/Crypto/Canvas/Memory benchmark with scores; includes speedtest if permitted | Client-side JS + server speedtest |
| Translation | translation | translate hello to french, translate X to Y | Text translation with language detection | Configured translation backends |
| Auto-translate | translation | (Automatic on non-native-language queries) | Detects foreign language queries and translates to user’s known language | Configured translation backends |
| Map | map | map of paris, map tokyo, directions to berlin | OpenStreetMap tile with links to OSM, Apple, Google, HERE, Bing, OsmAnd, Waze, Yandex Maps | Nominatim geocoding + OSM tiles |
| Wikipedia | wikipedia | Entity-like queries (1-3 capitalized words), who is X, what is X | Infobox card with image, summary, and Wikipedia link | Wikipedia REST API |
| DDG Instant Answers | ddg_ia | (Any query — lowest priority among knowledge oracles) | Abstract, definition, or answer card from DuckDuckGo | DuckDuckGo Instant Answer API |
| Package Version | packages | latest lodash, npm lodash, pypi requests, pip flask version | Latest version with link to registry | npm Registry / PyPI |
| Radio | radio | radio new york, fm tokyo, radio stations in berlin | Up to 5 radio stations with inline audio players | radio-browser.info |
| Sports Scores | sports | nfl scores, nba scores, lakers scores, premier league scores | Live/recent scores for NFL, NBA, MLB, NHL, MLS, EPL, La Liga, Bundesliga, Serie A, Ligue 1, Champions League, WNBA, NCAA | ESPN API |
| SSL Certificate | ssl_cert | ssl example.com, cert example.com, certificate example.com | Certificate issuer, validity dates, SANs, protocol version, cipher suite | Direct TLS connection |
| Earthquake | earthquake | earthquakes, earthquakes today, recent earthquakes, earthquakes in california | Recent significant earthquakes with magnitude, location, depth, and time | USGS Earthquake API |
| Flights | flights | flight UA123, flight status AA456, flights from JFK to LAX | Live flight tracking by callsign; route planning with bearing and distance | OpenSky Network API |
| Recipe | recipe | recipe for pasta, how to make brownies, how to cook risotto | Step-by-step recipe with ingredients, thumbnail, and source link | TheMealDB |
API-keyed oracles (require configuration)
These oracles require API keys configured in config.kdl under api-keys { ... }. Without the key, the oracle silently returns nothing.
| Oracle | Permission key | Trigger patterns | What it shows | API key config | API used |
|---|---|---|---|---|---|
| Wolfram Alpha | wolfram | (Any query — lowest priority knowledge oracle) | Computational answers, facts, math | wolframalpha "..." | Wolfram Alpha Short Answers API |
| Weather | weather | weather, weather in paris, forecast tokyo, temperature in london | Current temperature, humidity, wind, conditions with icon | Optional: openweathermap "..." | Open-Meteo (free, always); OpenWeatherMap (if key set, tabbed multi-source) |
| Dictionary | dictionary | define serendipity, meaning of ubiquitous, what does ephemeral mean, urban slang | Multi-source tabbed definitions | Optional: mw_collegiate, mw_thesaurus, mw_learners, mw_medical, mw_spanish, mw_school, mw_elementary, mw_intermediate, mw_intermediate_thesaurus, wordnik | Free Dictionary API + Wiktionary + Urban Dictionary (always); up to 9 Merriam-Webster products + Wordnik (with keys) |
| Stock | stock | $AAPL, TSLA stock, stock price MSFT | Price, change, open/high/low, previous close | Optional: alpha_vantage "...", twelvedata "...", finnhub "..." | Alpha Vantage, Twelve Data, Finnhub (multi-source tabbed if multiple keys) |
| Crypto | crypto | $BTC, bitcoin price, price of ethereum, crypto solana | USD/EUR/BTC price, 24h change, market cap | None (free API) | CoinGecko |
| Inflation / CPI | inflation | inflation, cpi, consumer price index, dollar inflation | Annual inflation rate with 12-month sparkline chart | None (free API) | Bureau of Labor Statistics |
| Purchasing Power | inflation | purchasing power of dollar, dollar worth in 1970 | Historical purchasing power table or year-specific comparison | None (free API) | Bureau of Labor Statistics CPI-U data |
| Air Quality | air_quality | air quality, aqi, pollution, air quality in beijing | AQI value, category, dominant pollutant, color-coded display | Optional: aqicn "..." | Open-Meteo Air Quality (free, always); AQICN (with key, tabbed multi-source) |
| News | news | news about AI, latest news on climate, headlines technology | Multi-source news headlines with thumbnails | Optional: currentsapi "...", newsdata "...", gnews "..." | Google News RSS (free, always); Currents API, Newsdata.io, GNews (with keys, tabbed) |
| Music / Spotify | music | play bohemian rhapsody, listen to radiohead, song despacito, Artist — Track | Spotify embed player with album info and alternative service links | Optional: spotify-client-id "..." + spotify-client-secret "..." | Spotify Web API (with keys); links-only fallback without keys |
Knowledge oracles (fire concurrently)
Three knowledge oracles fire simultaneously on every query and the best result wins. Priority order: Wikipedia > DDG Instant Answers > Wolfram Alpha.
| Priority | Oracle | Permission key | API |
|---|---|---|---|
| 0 (highest) | Wikipedia Infobox | wikipedia | Wikipedia REST API (no key) |
| 1 | DDG Instant Answers | ddg_ia | DuckDuckGo IA API (no key) |
| 2 (lowest) | Wolfram Alpha | wolfram | Wolfram Alpha Short Answers API (requires key) |
Smart tab oracles
When smart_tab_oracles is enabled (default: on), certain oracles auto-fire on category-specific tabs even without explicit trigger words:
| Tab | Auto-fired oracles |
|---|---|
| Maps | Map + Weather (for the displayed location), IP Geolocation |
| Music | Music (with forced “play” prefix), Radio |
| Science | Calculator, Unit Converter |
| IT | My IP, DNS Lookup, CIDR Calculator, Base Converter |
| Weather | Weather (with “weather” prefix) |
AI Research tools
When AI mode is enabled (lite or full), the AI research agent has access to these tools. These are not query-triggered oracles but tools the AI can invoke during research:
| Tool | Description |
|---|---|
fetch | Fetches and extracts text from URLs; supports depth=2 for same-domain crawling |
psychoscrape | Citation-aware web scraper that extracts structured claims with source attribution |
youtube_transcript | Extracts timestamped transcripts/captions from YouTube videos |
lua | Sandboxed Lua execution environment with HTTP, JSON, math, and string libraries |
send | Shows intermediate findings to the user during research |
eta | Updates the user-facing ETA countdown |
friction | Logs improvement suggestions for the server developer |
pin | Pins a result to the top of the search results |
forget | Removes a result from the search results |
friend_search | Searches federated peer instances for additional results |
API keys
Most engines scrape public pages and need no key. Engines that benefit from or require API keys:
| Service | Config key | Required? |
|---|---|---|
| Wolfram Alpha | api-keys { wolframalpha "..." } | Optional; enables Wolfram oracle |
| DeepL | api-keys { deepl "..." } | Optional; translation backend |
| Freesound | api-keys { freesound "..." } | Required for freesound engine |
| Spotify | api-keys { spotify-client-id / spotify-client-secret } | Optional; enables Spotify embed in music oracle |
| OpenWeatherMap | api-keys { openweathermap "..." } | Optional; adds second weather source |
| Alpha Vantage | api-keys { alpha_vantage "..." } | Optional; stock quotes |
| Twelve Data | api-keys { twelvedata "..." } | Optional; stock quotes |
| Finnhub | api-keys { finnhub "..." } | Optional; stock quotes |
| AQICN | api-keys { aqicn "..." } | Optional; adds second air quality source |
| Currents API | api-keys { currentsapi "..." } | Optional; news oracle source |
| Newsdata.io | api-keys { newsdata "..." } | Optional; news oracle source |
| GNews | api-keys { gnews "..." } | Optional; news oracle source |
| MW Collegiate | api-keys { mw_collegiate "..." } | Optional; dictionary source |
| MW Thesaurus | api-keys { mw_thesaurus "..." } | Optional; dictionary source (synonyms/antonyms) |
| MW Learner’s | api-keys { mw_learners "..." } | Optional; dictionary source |
| MW Medical | api-keys { mw_medical "..." } | Optional; dictionary source |
| MW Spanish | api-keys { mw_spanish "..." } | Optional; bilingual dictionary source |
| MW School | api-keys { mw_school "..." } | Optional; dictionary source |
| MW Elementary | api-keys { mw_elementary "..." } | Optional; dictionary source |
| MW Intermediate | api-keys { mw_intermediate "..." } | Optional; dictionary source |
| MW Int. Thesaurus | api-keys { mw_intermediate_thesaurus "..." } | Optional; dictionary source |
| Wordnik | api-keys { wordnik "..." } | Optional; dictionary source |
| Giphy | Hardcoded public key | None needed |
| The Guardian | Hardcoded test key | None needed |
| Pexels | Per-engine header in engines.kdl | Optional |
Adding a new engine
Add an engine block to data/engines.kdl:
engine "my_engine" {
name "My Engine"
categories "web"
search-url "https://api.example.com/search?q={query}&page={page}"
method "GET"
result-type "json"
results-path "results"
title-path "title"
url-path "url"
content-path "snippet"
page-size 20
paging true
first-page 1
}
Restart 4got. The engine registers automatically.
Template variables: {query}, {page}/{pageno}, {offset}, {lang}.
For HTML scraping, set result-type "html" and use CSS selectors (results-selector, title-selector, url-selector, content-selector).
Architecture
Request flow
User query
│
▼
┌─────────────────────────────────────────────────────────┐
│ handleSearch() │
│ │
│ 1. Parse bangs ──── !g, !w, etc. ──→ 302 redirect │
│ │ (no match) │
│ ▼ │
│ 2. Auth check │
│ │ │
│ ├── No session, first visit ──→ /captcha (IQ test) │
│ ├── No session, returning ───→ PoW challenge page │
│ │ (client-side mining) │
│ ▼ │
│ 3. Parse search params (query, category, cookies) │
│ │ │
│ ▼ │
│ 4. Local cache lookup ── hit ──→ skip to step 8 │
│ │ (miss) │
│ ▼ │
│ 5. Friend cache lookup ── hit ──→ store locally, │
│ │ (miss) skip to step 8 │
│ ▼ │
│ 6. Fanout (singleflight-coalesced) │
│ ├── Engine 1 ──→ goroutine ──→ HTTP scrape/API │
│ ├── Engine 2 ──→ goroutine ──→ HTTP scrape/API │
│ ├── Engine N ──→ goroutine ──→ HTTP scrape/API │
│ └── Friends ──→ goroutine ──→ peer cache query │
│ │ │
│ │ hard window (1000ms default) or │
│ │ idle timeout (500ms) with must-include gate │
│ ▼ │
│ 7. Merge + deduplicate + rank │
│ ├── URL normalization + dedup │
│ ├── Cross-engine consensus scoring │
│ ├── Per-engine weight × position decay │
│ ├── Intent-based boost (code, academic, images...) │
│ ├── Tracking param stripping │
│ └── Store merged results in cache │
│ │ │
│ ▼ │
│ 8. Render │
│ ├── Check oracles (calculator, unit, timezone, etc.)│
│ ├── URL rewriting (YouTube→Invidious, etc.) │
│ ├── Auto-translate if foreign query detected │
│ ├── Log search (if history enabled) │
│ └── Render HTML template with results + timeline │
│ │
│ 9. Push to peer caches (background goroutine) │
└─────────────────────────────────────────────────────────┘
Auth flow
New visitor Returning visitor
│ │
▼ ▼
┌────────────┐ ┌─────────────┐
│ IQ Test │ │ PoW WASM │
│ (captcha) │ │ (SHA-256 │
│ Image grid │ │ mining) │
└────┬───────┘ └──────┬──────┘
│ correct │ valid nonce
▼ ▼
┌──────────────────────────────────────┐
│ Ed25519-signed session cookie │
│ (AuthCaptcha or AuthPow) │
│ TTL: captcha=24h, pow=5min │
└──────────────────────────────────────┘
Owner: POST /admin/login with owner_secret
→ AuthOwner session cookie
Groups: cookie hash → group lookup (groups.kdl)
→ permissions: sidecar, server_api_keys,
friend_proxy, priority_fanout
Federation protocol
Instance A Instance B
│ │
│ GET /api/v1/cache │
│ ?q=QUERY&cat=web │
│ Authorization: Bearer PEER_SEC │
│ ─────────────────────────────→ │
│ │
│ 200 OK (cached results) │
│ or 204 No Content │
│ ←───────────────────────────── │
│ │
│ POST /api/v1/cache │
│ (push fresh results to friend) │
│ ─────────────────────────────→ │
│ │
│ POST /api/v1/proxy │
│ {"service":"translation", │
│ "params":{...}} │
│ ─────────────────────────────→ │
│ │
│ 200 OK (translation result) │
│ ←───────────────────────────── │
│ │
│ GET /ami4got │
│ ─────────────────────────────→ │
│ {"version":"2.0", │
│ "engine_count":15, ...} │
│ ←───────────────────────────── │
Proxyable services: translation, wolfram, autocomplete
Auth: shared peer_secret as Bearer token
Wait policy: configurable (all friends, N friends, miss-ok)
Failure tracking: auto-mark down at threshold (e.g. 1% over 1d)
AI research agent loop
User searches same query twice
│
▼
GET /api/v1/research?q=QUERY (SSE stream)
│
▼
┌─────────────────────────────────────────┐
│ RunAIResearch() │
│ │
│ System prompt (data/metaprompt.md) │
│ + top search result URLs │
│ │ │
│ ▼ │
│ LLM generates tool calls: │
│ ┌──────────────────────────────────┐ │
│ │ fetch(url) → HTTP GET + strip │ │
│ │ lua(code) → sandboxed Lua VM │ │
│ │ send(text) → SSE "message" │ │
│ │ eta(s,job) → SSE "eta" │ │
│ │ pin(text) → permanent context │ │
│ │ forget(id) → evict old result │ │
│ │ friction(s) → operator feedback │ │
│ └──────────────────────────────────┘ │
│ │ │
│ │ (loop until done or timeout) │
│ ▼ │
│ SSE "done" event with final answer │
│ │
│ Cache result for replay │
└─────────────────────────────────────────┘
Provider: any OpenAI-compatible API
(Cerebras, OpenRouter, Ollama, custom)
Sandbox: Lua (recommended) or bash (containers only)
Timeout: max-runtime-seconds (default 60)
MCP: optional external tool servers via stdio/HTTP
File organization
Go server (go-server/)
| File | Lines | Purpose |
|---|---|---|
main.go | 503 | Entry point, config loading, startup sequence |
routes.go | 364 | HTTP route registration table |
handlers_search.go | 455 | Search and landing page handlers |
handlers_settings.go | 281 | Settings page GET/POST |
handlers_auth.go | 280 | Captcha and PoW handlers |
handlers_admin.go | ~200 | Admin log and login |
handlers_api.go | ~150 | JSON API endpoint |
handlers_misc.go | ~100 | About, health, robots |
admin_users.go | 210 | User group management |
engine.go | 483 | Engine interface, types, registry, HTTP client |
generic_engine.go | 453 | KDL-driven engine factory |
engine_brave.go | 377 | Brave scraper |
engine_google.go | 301 | Google scraper |
engine_bing.go | 405 | Bing scraper |
engine_yahoo.go | 351 | Yahoo scraper |
engine_baidu.go | 307 | Baidu scraper |
engine_duckduckgo.go | 230 | DDG scraper |
engine_startpage.go | ~170 | Startpage scraper |
engine_yandex.go | ~170 | Yandex scraper |
engine_qwant.go | ~150 | Qwant API client |
engine_wikipedia.go | ~120 | MediaWiki API |
engine_youtube.go | ~130 | YouTube scraper |
engine_hackernews.go | ~100 | HN Algolia API |
engine_marginalia.go | ~100 | Marginalia API |
engine_mojeek.go | ~100 | Mojeek scraper |
engine_wiby.go | ~80 | Wiby API |
engine_flickr.go | ~80 | Flickr API |
engine_pexels.go | ~80 | Pexels API |
engine_soundcloud.go | ~100 | SoundCloud API |
fanout.go | 988 | Concurrent engine dispatch, scoring, dedup, merge |
cache.go | 346 | In-memory LRU + TTL cache |
peercache.go | 249 | Friend cache query/push |
peers.go | 268 | Peer config loading, failure tracking |
friendproxy.go | 327 | Service proxy between friends |
auth.go | 316 | Ed25519 sessions, PoW verification |
captcha_prerender.go | 688 | Pre-rendered captcha pool (IQ test grid) |
captcha_image.go | 307 | Captcha image compositing |
oracles.go | 893 | Calculator, unit, timezone, base, translate, DDG IA, Wolfram |
oracles_extra.go | 852 | Currency, IP geo, DNS, color, packages, radio |
airesearch.go | 1917 | AI agent: LLM loop, tool dispatch, Lua sandbox |
airesearch_sse.go | 187 | SSE event streaming |
airesearch_cache.go | ~100 | Research result caching |
mcp_client.go | 195 | MCP tool server integration |
translation.go | 614 | Multi-backend translation with routing |
langdetect.go | ~100 | Language detection for auto-translate |
autocomplete.go | 302 | Multi-backend autocomplete merger |
classifier.go | ~100 | Query intent classification |
imageproxy.go | 433 | Image proxy with SSRF protection |
templates.go | 975 | HTML template rendering |
db.go | 304 | SQLite schema, logging, queries |
groups.go | 263 | User tier/group loading from KDL |
tiers.go | 222 | Permission checking |
rate_limit.go | 202 | Per-IP rate limiting |
rewrite.go | ~100 | URL hostname rewriting |
i18n.go | ~100 | Internationalization |
instances.go | 288 | Instance browser + ami4got |
plugins.go | 214 | Lua plugin system |
selfheal.go | ~100 | Autonomous engine repair agents |
sidecar.go | ~100 | Python sidecar integration |
rss.go | ~100 | Atom feed generation |
openapi.go | 530 | OpenAPI 3.0 spec generation |
api_compat.go | ~100 | 4get-compatible API |
api_doc.go | ~100 | Plain-text API docs |
metrics.go | ~100 | Prometheus counters |
counters.go | ~50 | Atomic hit/miss counters |
config_validate.go | 186 | Config validation at startup |
kdl_utils.go | ~80 | KDL parsing helpers |
kdlresolver.go | ~80 | KDL section resolution |
helpers.go | ~80 | Cookie reading, IP extraction |
stubs.go | 199 | Compile-time stubs for optional features |
types.go | ~50 | Shared type definitions |
Python sidecar (python-sidecar/)
| File | Lines | Purpose |
|---|---|---|
server.py | 96 | Flask HTTP server |
google.py | 245 | Google Playwright scraper |
yandex.py | 219 | Yandex Playwright scraper |
qwant.py | 259 | Qwant Playwright scraper |
KDL configs (data/)
| File | Controls |
|---|---|
config.kdl | Main server configuration (port, auth, cache, API keys) |
config.default.kdl | Default config template |
engines.kdl | All 274 generic engine definitions |
peers.kdl / peers.default.kdl | Friend network topology and wait policy |
groups.kdl | User tiers and permissions |
captcha.kdl | Captcha pool sizes and generation cadence |
autocomplete.kdl | Autocomplete backend weights |
translation.kdl | Translation backend priority and rate limits |
instances.kdl | Instance browser entries |
ai-research.kdl / ai-research.default.kdl | AI agent provider, model, access control |
metaprompt.md | AI agent system prompt |
Static assets (static/)
CSS, JavaScript (PoW WASM worker, autocomplete, SSE client), and fonts.
Templates
HTML templates rendered server-side by templates.go. No client-side framework.
Key design decisions
Single binary. The Go server compiles to one static binary (plus CGO for SQLite). No runtime dependencies except the data/ and static/ directories.
KDL over JSON for subsystems. Human-readable, comment-friendly config. The resolver supports both per-file (peers.kdl) and unified (conf.kdl with sections) layouts.
Singleflight coalescing. Concurrent identical queries share one fanout, preventing thundering herd on popular searches.
Ed25519 session cookies. Session tokens are signed, not stored server-side. No session table, no Redis.
Dual-layer anti-abuse. First-time visitors solve a visual captcha (pre-rendered, no JS required for the test itself). Returning visitors solve a WASM proof-of-work (SHA-256 mining). Both issue signed session cookies.
Oracle-first. Queries that match oracle patterns (math, units, DNS, etc.) return instant answers without waiting for engine fanout.
Friend-first cache. Before fanning out to upstream engines, 4got checks local cache, then friend caches. This minimizes upstream load across the friend network.
Design Decisions
Why federation over Tor outbound for IP-ban avoidance. Tor exit nodes are heavily flagged by upstream engines, leading to high captcha rates and slow responses. Federation distributes queries across multiple real IPs (each friend instance has its own), so no single IP gets rate-limited. Tor outbound is still available per-engine (tor-outbound true) for engines that block datacenter IPs but not Tor, but the primary strategy is the friend network.
Why low comment density is intentional. The codebase prefers well-named functions, types, and variables over inline commentary. Comments are reserved for non-obvious architectural decisions, protocol boundaries, and “why” explanations – not restating what the code already says. This keeps the signal-to-noise ratio high during code review and reduces stale-comment drift.
Why KDL over YAML/TOML/JSON for config. KDL supports inline comments, nested blocks without deep indentation, and space-separated multi-value attributes (ideal for categories "web" "news"). YAML is whitespace-sensitive and error-prone; TOML doesn’t handle nested structures cleanly; JSON forbids comments. KDL’s document-oriented model maps naturally to engine definitions and peer configs.
Why SQLite over Redis/Postgres. SQLite is embedded (no external daemon), survives restarts without a separate persistence layer, and handles 4got’s write volume (search logs, cache, stats) comfortably on a single core. The entire database is one file, trivially backed up. Redis would add an operational dependency for the cache layer alone; Postgres is overkill for a single-instance search engine.
Why single-binary Go over microservices. A metasearch engine’s subsystems (fanout, caching, auth, federation) share in-process state (the engine registry, in-memory cache, rate limiter). Splitting them into services would add network hops on the hot path and complicate deployment for self-hosters. The Python sidecar exists only for Playwright-dependent engines that need a real browser, and communicates over localhost HTTP.
Why no BM25. BM25 was implemented and tested but removed. The marginal ranking improvement over the existing position-weighted cross-engine consensus scoring was negligible, and BM25 is vulnerable to keyword stuffing in snippets – an attacker can inflate term frequency in a single engine’s results to dominate rankings. The current scoring (inverse position * engine weight * engine count * authority domain multiplier) is simpler and harder to game.
Why no click tracking. 4got is privacy-first: no result click is recorded, no outbound redirect URL wraps results, and no client-side beacon fires on click. This means ranking cannot incorporate click-through signals, which is an accepted trade-off. The honeypot link (/static/assets/beacon.js) exists solely to detect bots, not to track users.
Why engine goroutines are not cancelled after the user gets their response. When the fanout time budget expires and results are served to the user, the remaining engine goroutines continue running in the background. This is intentional: late-arriving results are merged into the cache, so the next user who searches the same query gets richer results. Cancelling goroutines would save a few connections but would mean cached results are permanently limited to whatever came back within the first time window. The cache is the product; the user’s wait time is the budget.
Why engine health monitoring is notification-based, not autonomous. Engine health snapshots (engine_snapshots.go) store the last working and last error HTML for each engine, and the notification system (notify.go) dispatches structured events to ntfy push notifications, webhooks, and a JSONL event log. Actual remediation happens outside the program — the operator hooks up whatever they want (a Claude agent, a cron job, a human reading notifications). This keeps the search engine’s scope clean: it detects problems and tells you, it doesn’t try to fix them itself. The earlier “self-healing” framing was retired in favor of this more honest and composable design.
Why adaptive PoW difficulty, not static. AdaptivePowDifficulty in auth.go scales the PoW difficulty based on current server load: +1 bit above 100 RPM, +2 above 500, +3 above 1000. This means legitimate users during low-traffic periods solve a ~1 second puzzle, while during attacks the difficulty automatically ramps up. Combined with the image captcha (first visit) and Tor-aware exemptions, this creates a graduated defense that doesn’t punish normal users for attacker behavior.
Deploying 4got
Quick Start
git clone https://github.com/your-user/4got.git && cd 4got
cp data/config.default.kdl data/config.kdl # edit owner-secret!
docker compose up -d
Open http://localhost:8888 — you have a working search engine. See First-run configuration to set your admin secret and API keys.
Docker Compose
The recommended way to run 4got. The docker-compose.yml builds both the Go server and the optional Python sidecar.
# Start the server only (no browser-backed engines)
docker compose up -d
# Start with the Python sidecar for Google/Qwant/Yandex JS scraping
docker compose --profile sidecar up -d
# View logs
docker compose logs -f server
# Stop everything
docker compose down
Configuration lives in data/config.kdl (mounted as a volume). The SQLite database (data/4got.db) is created automatically on first run.
To customise ports, API keys, or peer settings, copy the override example and edit it:
cp docker-compose.override.yml.example docker-compose.override.yml
# edit docker-compose.override.yml, then:
docker compose up -d
First-run configuration
- Copy
data/config.default.kdltodata/config.kdlif you haven’t already - Set
owner-secretto a strong random string — this is your admin password:python3 -c "import secrets; print(secrets.token_urlsafe(32))" - Add API keys in the
api-keysblock (all optional):wolframalpha— instant answersdeepl— translation widgetspotify-client-id/spotify-client-secret— music results
- Configure engine groups in
data/groups.kdl(the defaults are sensible) - Start the server and visit
/admin/loginto authenticate as owner - Optionally set
peer-secretand configuredata/peers.kdlfor federated search across multiple instances
Prerequisites
- Go 1.23+ (the
go.modtargets 1.25 but 1.23+ should work) - SQLite3 development headers (e.g.
libsqlite3-devon Debian,sqlite-devon Alpine) - GCC or a C compiler (required by go-sqlite3’s CGO binding)
- cjxl (optional) for JPEG XL captcha and image proxy reencoding — part of libjxl
- Playwright (optional) for the Python sidecar that handles JS-rendered engine scraping
Building from source
cd go-server
go build -o 4got .
CGO must be enabled (it is by default). If cross-compiling, set CGO_ENABLED=1 explicitly:
CGO_ENABLED=1 go build -o 4got .
The binary is fully self-contained. Copy it alongside the data/, static/, and templates/ directories.
Directory layout
4got/
go-server/
4got # the binary
data/
config.kdl # main configuration
*.kdl # subsystem configs (peers, groups, captcha, etc.)
4got.db # SQLite database (created automatically)
static/ # CSS, JS, fonts
templates/ # HTML templates
The binary looks for data/config.kdl (relative to CWD) or ../data/config.kdl (when running from go-server/). Override with FGOT_CONFIG=/path/to/config.kdl.
Running directly
cd /path/to/4got
./go-server/4got
By default it listens on 0.0.0.0:8888. Override via config.kdl or environment variables:
FGOT_PORT=9999 FGOT_HOST=127.0.0.1 ./go-server/4got
Running with systemd
A production-ready unit file ships in the repo at 4got.service. Install it:
sudo cp 4got.service /etc/systemd/system/4got.service
sudo systemctl daemon-reload
sudo systemctl enable --now 4got
sudo journalctl -u 4got -f
The unit file targets network-online.target, runs as user/group 4got, applies security hardening (ProtectSystem=strict, ProtectHome=yes, NoNewPrivileges, PrivateTmp), and sets LimitNOFILE=65536 for high-concurrency deployments. Edit ReadWritePaths if your data directory is not /opt/4got/data.
Reverse proxy
nginx
server {
listen 443 ssl http2;
server_name search.example.com;
ssl_certificate /etc/letsencrypt/live/search.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/search.example.com/privkey.pem;
location / {
proxy_pass http://127.0.0.1:8888;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# SSE support (AI research agent)
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 120s;
}
# Optional: serve static files directly
location /static/ {
alias /opt/4got/static/;
expires 7d;
}
}
Caddy
search.example.com {
reverse_proxy 127.0.0.1:8888
}
Caddy handles TLS automatically via Let’s Encrypt.
Docker / Podman
A multi-stage Dockerfile is provided in the project root, based on Debian bookworm-slim (glibc required for CGO/SQLite):
docker build -t 4got .
docker run -d -p 8888:8888 -v 4got-data:/data 4got
Podman is a drop-in replacement – the same commands work with podman instead of docker.
Mount a volume at /data to persist the SQLite database, configuration, and cache across restarts. The docker-compose.yml handles this automatically:
docker compose up -d
Codebase size comparison
Lines of code as of May 2026. Measured with wc -l against source files, excluding vendored dependencies.
| Component | Lines |
|---|---|
| 4got Go source (minus tests) | 20,866 |
| 4got test code | 5,324 |
| 4got Go total | 26,190 |
| 4got KDL configs (engine defs, etc.) | 10,682 |
| 4got Python sidecar | 819 |
| 4got templates (HTML) | 3,326 |
| 4got static (JS + CSS) | 3,299 |
| 4get PHP (reference implementation) | 35,480 |
| SearXNG Python (minus tests) | 47,425 |
| SearXNG test code | 3,764 |
| SearXNG Jinja templates | 2,263 |
4got’s Go core is ~40% smaller than 4get’s PHP while covering more categories (15 vs 4get’s handful) and adding federation, AI research, oracles, and multi-backend translation. The engine count difference is largely engines.kdl (274 engines declared in config, not code).
PaaS providers
SQLite needs persistent storage on every PaaS. Ephemeral filesystems lose your cache, stats, and search history on every redeploy. Always mount a volume at /data.
Fly.io
A fly.toml is included in the repo root. Create a volume before deploying:
fly launch # creates the app, detects the Dockerfile
fly volumes create 4got_data --region iad --size 1
fly deploy
The fly.toml mounts the volume at /data and exposes port 8888 internally. Fly handles TLS and routing automatically.
Railway
Railway auto-detects the Dockerfile. Set the PORT environment variable to 8888 in your service settings. Attach a persistent volume mounted at /data to keep SQLite state across deploys.
Render
Create a new Web Service with Docker deployment. Point it at the repo, then add a persistent disk mounted at /data (minimum 1 GB). Set the HTTP port to 8888 in the service configuration.
Coolify
Self-hosted PaaS. Point Coolify at the git repo, it will build from the Dockerfile. Add a persistent volume mapped to /data. No special configuration needed beyond that.
LXC / bare metal
Install Go 1.25+ and a C compiler (GCC), then build from source:
cd go-server
CGO_ENABLED=1 go build -o 4got .
Copy the binary alongside the required directories to the target machine:
rsync -a go-server/4got go-server/templates/ static/ data/ docs/ target:/opt/4got/
The binary runs from the project root and expects data/, static/, and go-server/templates/ as siblings (it searches multiple template paths automatically). See Running with systemd above for a production service file.
Install tesseract-ocr on the host if you want file analysis OCR support.
DNS privacy
To prevent search query leakage to your ISP via DNS resolution, run dnscrypt-proxy in front of your system resolver (Technitium, Unbound, or systemd-resolved). This encrypts outgoing DNS queries to upstream resolvers.
Recommended setup:
- Install
dnscrypt-proxyand point it at trusted DoH/DoT servers (e.g. Quad99.9.9.9, Cloudflare1.1.1.1). - Configure your local resolver (Technitium/Unbound) to forward to
127.0.0.1:5353(dnscrypt-proxy’s default listen port). - Ensure 4got’s host uses this resolver chain so engine queries don’t leak plaintext DNS to your ISP.
Or just point Claude Code at this repo and ask it to help you deploy
claude "Deploy 4got to my server at /opt/4got. Set up systemd, nginx reverse proxy on search.example.com, and configure a Wolfram Alpha API key."
Runtime dependencies for banner uploads
Banner transcoding shells out to cjxl (stills → progressive JXL, metadata stripped) and ffmpeg with libaom (animated GIF → animated AVIF). On Debian: apt install libjxl-tools ffmpeg. Without them, banner uploads fail with “Transcoding failed” — searches are unaffected.
Scaling plan
Every server owner needs one. This is the capacity model for a 4got instance, worked for the reference deployment (1 vCPU, 1 GB RAM, 300 mbps, 1 TB/month), and the API-key policy that survives growth.
The surprise: bandwidth is the binding constraint, not compute
Go handles thousands of concurrent connections on one core; a cached search serve is sub-millisecond work. The ceilings, in the order you will actually hit them:
- Monthly traffic cap. 1 TB/month ≈ 33 GB/day ≈ 385 KB/s sustained. A results page is ~100 KB raw, ~30 KB with compression, before proxied images. That prices the box at roughly ~1M page views/month as an absolute ceiling — and proxied images eat it far faster, which is why the traffic guardrails throttle heavy transfers first while search stays fast. Watch the Traffic line in settings; the knobs are
traffic-*in config.kdl. - Upstream engine tolerance. Every UNCACHED search fans out to dozens–hundreds of engines from one IP. Public engines throttle single IPs long before your hardware blinks. The realistic uncached-fanout budget is hundreds to low thousands per day per engine — the communal cache’s hit rate is therefore the real multiplier of the whole system. Everything that raises hit rate (shared anonymous cache, prefetch, peers) is scaling work.
- Compute/RAM. Last by a wide margin: ~50–100 simultaneous uncached fanouts before TLS handshake latency degrades on one core; cached serves in the hundreds per second; SQLite’s single writer is nowhere near stressed at these rates.
Net: thousands of daily active users are realistic on the reference box if the cache hit rate is healthy; millions of monthly uniques require more egress — a bigger pipe, and more usefully, peers: federated instances sharing caches are the horizontal-scaling story, and each peer brings its own upstream IP budget.
API keys: the tiered policy
Server-covered keys do not scale — a 25-req/day API handed to the public is exhausted by breakfast and everyone’s annoyed. The policy that scales:
- Trusted groups (deluxe: friends, family, beta testers) — keys covered by the server, limits shown in settings.
- Everyone else — bring your own key. BYO keys can opt into the pool: draws are randomly scrambled across contributed keys, which (a) gives contributors anonymity-by-randomness against the API provider — the provider sees the server’s IP either way, but key-scrambling breaks the which-human-asked-what linkage that identity-tied keys otherwise leak — and (b) absorbs surges, since most people use a fraction of their quota on most days, so a member can burst far beyond their own key some days (their agents too) against the pool’s slack.
- Grace budget (Monero era): pool members may run negative within a configured grace; the deeper negative, the harder the scheduler works to lend their key out. Balances settle when the ledger lands.
- ToS note for owners: some API providers prohibit key pooling. Enabling the pool is the owner’s call per API; the per-key limit fields and group gating already let you keep sensitive keys unpooled.
Changing server defaults without fighting users’ cookies
Revoking a covered key, changing a default, renaming a setting — all must be supported behavior, not a migration crisis. Mechanism (planned): a settings epoch — the server stamps an epoch into served settings; cookies carry the epoch they were written under; on mismatch the server migrates known-renamed keys, drops known-dead ones, and (where a change is user-visible) says so once. Until the epoch machinery lands, the rule of thumb: additive changes are free; renames keep a legacy-read fallback (as ai_mode → ai_mode_single/double does today); revocations of covered keys degrade to “bring your own” messaging rather than errors.
Friend Access with mTLS
4got uses mutual TLS (mTLS) client certificates to control who can access your instance. This is a deliberate choice over OAuth, SSO, or any third-party identity provider — your friends prove their identity with a cryptographic certificate that you issued, and no external service is involved in the handshake.
Why mTLS instead of OAuth?
OAuth solves “let strangers log in with their existing accounts.” You’re solving “let my friends in and everyone else out.” Different problem, different tool.
With mTLS:
- No third party knows who uses your search engine. There’s no Google/GitHub/Discord OAuth callback leaking the fact that your friend searched for something. The authentication is a TLS handshake between the client and your server — no one else is in the loop.
- No tokens to steal. A stolen OAuth token gives an attacker your session. A stolen client cert without the private key is useless. The private key never leaves your friend’s device.
- No session hijacking. The TLS handshake happens before HTTP. There are no cookies, headers, or query parameters that a man-in-the-middle or XSS attack could extract.
- No infrastructure dependency. Your auth doesn’t break when GitHub is down, when Google changes their OAuth scopes, or when Discord decides to deprecate an API version.
- Revocation is simple. Remove a friend’s certificate serial number from the trust list. Done. No “invalidate all sessions” dance.
The tradeoff is that cert distribution requires a one-time manual step per friend. For a friends-and-family instance, this is a feature, not a bug — it means access can’t be self-service, which is exactly what you want.
How it works
Friend's browser Your server
| |
|------- TLS ClientHello -------------->|
|<------ TLS ServerHello + ServerCert --|
|<------ CertificateRequest ------------|
|------- ClientCert (signed by your CA) |
|------- CertificateVerify ------------>|
| |
| (server checks: is this cert signed |
| by my CA? is it not revoked?) |
| |
|<------ TLS Finished ------------------|
|------- HTTP request ----------------->|
|<------ Search results ----------------|
If the client doesn’t present a valid certificate, the TLS handshake fails. The connection is refused before any HTTP traffic is exchanged. Unauthorized users never reach 4got — they get a TLS error, not a login page.
Setup
1. Create your Certificate Authority
You are the CA. This is a one-time operation.
mkdir -p /etc/4got/certs && cd /etc/4got/certs
# Generate CA private key
openssl genrsa -out ca.key 4096
# Generate CA certificate (valid 10 years)
openssl req -new -x509 -key ca.key -out ca.crt -days 3650 \
-subj "/CN=4got Friend CA/O=4got"
Guard ca.key with your life. Anyone with this file can issue client certs that your server will trust. Store it offline when not issuing new certs.
2. Issue a client certificate for a friend
Run this for each friend. Replace alice with their name.
# Generate friend's private key
openssl genrsa -out alice.key 2048
# Generate a certificate signing request
openssl req -new -key alice.key -out alice.csr \
-subj "/CN=alice/O=4got Friends"
# Sign with your CA (valid 2 years)
openssl x509 -req -in alice.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out alice.crt -days 730
# Package into a .p12 file (this is what your friend installs)
openssl pkcs12 -export -out alice.p12 \
-inkey alice.key -in alice.crt -certfile ca.crt \
-name "4got (alice)"
You’ll be prompted for an export password. Pick something simple — your friend needs to type it once during import. You can use an empty password if you trust the delivery channel.
Clean up the intermediate files:
rm alice.key alice.csr alice.crt
# Keep alice.p12 — this is what you give to your friend
3. Give the .p12 to your friend
Preferably via sneakernet — hand them a USB drive, AirDrop it across the table, or NFC beam it from your phone. Physical proximity is the strongest delivery channel because there’s no server, no log, no metadata trail, and no possibility of interception.
If physical handoff isn’t possible, send the .p12 over an end-to-end encrypted channel (Signal, Matrix with verification, encrypted email). Communicate the export password through a different channel than the file itself (e.g., file over Signal, password spoken over a phone call).
Never send client certificates over unencrypted email, Discord DMs, Slack, or any platform that stores messages server-side. The .p12 contains your friend’s private key — if it leaks, you need to revoke and reissue.
4. Configure Caddy
Caddy handles the public-facing TLS (automatic Let’s Encrypt) and verifies client certificates against your CA.
search.yourdomain.com {
tls {
client_auth {
mode require_and_verify
trusted_ca_cert_file /etc/4got/certs/ca.crt
}
}
reverse_proxy localhost:8888
}
That’s the entire Caddyfile. Caddy:
- Obtains and renews a Let’s Encrypt certificate for
search.yourdomain.comautomatically - Requires every connecting client to present a certificate signed by your CA
- Rejects unauthorized connections at the TLS layer (before HTTP)
- Forwards authenticated requests to 4got on localhost
5. Your friend installs the certificate
Android
Tap the .p12 file (from Files, a download, or a file manager). Android prompts to install it as a user certificate. Enter the export password. Chrome and other browsers will present it automatically when connecting to your instance.
iOS / iPadOS
Tap the .p12 file to open it. Go to Settings → General → VPN & Device Management and install the profile. Enter the export password. Safari and all apps using the system keychain will present it automatically.
macOS
Double-click the .p12 file. Keychain Access opens and imports it. Enter the export password. Safari and Chrome use it automatically. Firefox users: Preferences → Privacy & Security → Certificates → View Certificates → Import.
Windows
Double-click the .p12 file. The Certificate Import Wizard opens. Follow the prompts, enter the export password, and import into the “Personal” store. Edge and Chrome use it automatically. Firefox users: Settings → Privacy & Security → Certificates → View Certificates → Import.
Linux
Firefox: Preferences → Privacy & Security → Certificates → View Certificates → Your Certificates → Import. Select the .p12 file and enter the export password. Chromium-based browsers: use the system NSS certificate store or import via chrome://settings/certificates.
None of this touches VPN settings, proxy configuration, or network setup. The certificate is a TLS identity credential — it lives in the browser/OS certificate store alongside everything else and is presented during the TLS handshake when the server requests it.
Revoking a friend’s access
Quick method: reissue the CA
If you only have a handful of friends and need to revoke one, the simplest approach is to regenerate the CA and reissue certificates to everyone except the revoked friend. This is practical when you have fewer than ~20 friends.
Proper method: Certificate Revocation List (CRL)
For larger deployments or when you don’t want to bother everyone:
# Create a CRL (first time)
openssl ca -gencrl -out /etc/4got/certs/crl.pem \
-keyfile /etc/4got/certs/ca.key \
-cert /etc/4got/certs/ca.crt
# Revoke a specific certificate
openssl ca -revoke alice.crt \
-keyfile /etc/4got/certs/ca.key \
-cert /etc/4got/certs/ca.crt
# Regenerate the CRL
openssl ca -gencrl -out /etc/4got/certs/crl.pem \
-keyfile /etc/4got/certs/ca.key \
-cert /etc/4got/certs/ca.crt
Note: CRL-based revocation with Caddy requires checking the tls directive documentation for your Caddy version, as CRL support has varied across releases. An alternative is to maintain a list of allowed certificate fingerprints and check them in a Caddy handler.
FAQ
Why not Tailscale / WireGuard? Those are great for private access but they require your friends to install a VPN client and join your network. mTLS works in any browser on any device with zero additional software. It’s also invisible to the user after the one-time cert install — they just open the URL and search.
Why not Cloudflare Access? Cloudflare terminates TLS at their edge and inspects all HTTP traffic in plaintext. There is no Cloudflare configuration where they cannot read your search queries. If you’re building a privacy-first search engine, routing all traffic through a surveillance company defeats the purpose.
Why not HTTP basic auth? Passwords get shared, written down, reused, and phished. Client certificates are cryptographic keys bound to a device. Basic auth is also transmitted on every request (Base64 encoded, not encrypted beyond TLS), while client cert auth happens once during the TLS handshake.
Why not OAuth2 Proxy / Authelia / Authentik? These are good tools for multi-tenant SaaS applications. For a friends-only search engine, they add a database, a login page, a session store, password reset flows, and a dependency on an external identity provider — all to solve a problem that mTLS solves with zero moving parts.
Can my ISP see that my friends are connecting? Your ISP can see that an IP address connected to your server on port 443. They cannot see the client certificate, the search queries, or any HTTP content — that’s all inside the TLS tunnel. The mTLS handshake itself is encrypted.
What if a friend loses their device? Revoke their certificate and issue a new one. The old cert becomes useless immediately (once the CRL is updated or the CA is reissued).
Federation: The Friend Network
4got instances can form a peer-to-peer friend network for cache sharing, service proxying, and instance discovery. This is how multiple 4got deployments cooperate to improve speed and resilience.
Philosophy
The 4got protocol is a result-sharing protocol for self-contained queries — queries that are small, universal, and should have roughly the same answer regardless of who asks or where they ask from. Search engines are just one type of intelligence backend that can feed results into this protocol. A friend’s instance might answer a query from Google, from a custom scraper, from an AI agent, from a local database, or from another friend’s cache — the protocol doesn’t care. It exists to share answers between friends, not to replace any particular search provider. There is room for the protocol to be used with all kinds of intelligence backends beyond traditional search engines.
Overview
The friend network provides three capabilities:
- Peer cache sharing — friends share cached search results, reducing redundant upstream queries
- Service proxy — friends proxy expensive services (translation, Wolfram Alpha, autocomplete) for each other
- Instance discovery — the
/ami4gotendpoint and instance browser let users find other instances
Peer cache sharing protocol
When a search is performed, 4got queries all configured friends in parallel via GET /api/v1/cache?q=<query>&cat=<category>. Friends respond with cached results if available.
Request flow
- User searches for “quantum computing”
- 4got sends the query to all configured friends simultaneously
- Friends check their local cache and respond with results (or 204 No Content)
- 4got merges friend results with its own engine results using the standard scoring/deduplication algorithm
- If
store-friend-cacheis enabled, results from friends are stored in the local cache
Wait policy
The wait-for setting in peers.kdl controls how many friends must respond before proceeding:
| Value | Meaning |
|---|---|
1.0 | Wait for all friends (proportion: 100%) |
0.5 | Wait for half of the online friends |
3 | Wait for exactly 3 friends |
-2 | OK to miss up to 2 friends |
-0.1 | OK to miss up to 10% of friends |
The wait policy interacts with the fanout hard window (fanout-hard-window-ms in config.kdl). Friends that don’t respond within the hard window are skipped regardless of the wait policy.
Failure tracking
The failure-threshold setting tracks friend reliability:
failure-threshold 0.01 1d
This means: if a friend fails more than 1% of requests over a 1-day window, consider them “down” and don’t wait for them. They’ll be retried periodically but won’t block search results.
Always-wait and preferred friends
// Always wait for this friend's cache (even if slow)
always-wait-for "alice"
// Prefer this friend for specific services
prefer-friend "alice" for="translation,wolfram"
Service proxy
Friends can proxy expensive API-backed services for each other. This lets a single Wolfram Alpha API key serve multiple instances, or a single translation backend serve the whole friend network.
How it works
- Instance A needs to translate text but has no DeepL API key
- Instance A sends
POST /api/v1/proxyto friend Instance B with{"service": "translation", "params": {"text": "hello", "target": "fr"}} - Instance B checks its
allow-proxylist, finds “translation” is allowed, performs the translation, and returns the result - Instance A uses the result as if it performed the translation locally
Supported services
translation— text translation (DeepL, Papago, Lingva, NLLB, Microsoft)wolfram— Wolfram Alpha short answers APIautocomplete— search suggestion backends
Configuration
On the provider instance (the one that actually has the API keys):
allow-proxy "translation" "wolfram" "autocomplete"
On the consumer instance (the one that wants to use the services):
prefer-friend "provider-name" for="translation,wolfram"
The consumer will try the preferred friend first, then fall back to local backends.
Authentication
Service proxy requests are authenticated using the peer-secret from config.kdl, sent as a Bearer token.
Instance discovery
The ami4got endpoint
Every 4got instance exposes GET /ami4got (inspired by 4get’s /ami4get.php). This returns a JSON object with instance metadata:
{
"version": "2.0",
"api_version": "1",
"engine_count": 15,
"stats": {
"hits": 12345,
"real_searches": 6789
}
}
Instance browser
The /instances page shows a list of known 4got instances. It fetches info from configured friends and public instances periodically.
Configure in instances.kdl:
fetch-interval-min 60
friends {
instance "https://search.friend.example.com"
}
public {
instance "https://public.4got.example.com"
}
Friend-of-friend discovery
When Instance A fetches /ami4got from Instance B, it can discover Instance B’s own friends. This creates an organic network where adding a single friend connection transitively discovers the broader network.
Setting up two instances as friends
Step 1: Choose a shared secret
Both instances need the same peer-secret in their config.kdl:
peer-secret "a_strong_random_string_shared_between_friends"
Step 2: Configure peers on each instance
On Instance A (data/peers.kdl):
wait-for 1.0
store-friend-cache true
allow-proxy "translation" "wolfram"
peer "instance-b" {
url "https://search-b.example.com"
}
On Instance B (data/peers.kdl):
wait-for 1.0
store-friend-cache true
allow-proxy "translation" "wolfram"
peer "instance-a" {
url "https://search-a.example.com"
}
Step 3: Add to instance browser (optional)
On both instances, add each other to instances.kdl:
friends {
instance "https://search-b.example.com"
}
Step 4: Test
- Restart both instances
- Search on Instance A — check the logs for “cached on 1 friend” in the results
- Visit
/admin/logto see engine stats (friend results appear as(friend)entries)
Step 5: Optional mTLS
For additional security, configure mutual TLS between friends:
peer "instance-b" {
url "https://search-b.example.com"
cert "/path/to/client.crt"
key "/path/to/client.key"
}
Both instances need to trust each other’s client certificates.
AI Research Agent
The AI research agent is a tool-using LLM that synthesizes search results into a comprehensive answer. It is triggered when a user searches the same query twice (the “double-search” trigger), indicating they want deeper analysis than raw search results can provide.
How it works
- User searches “quantum error correction”
- 4got returns normal search results
- User searches “quantum error correction” again (or clicks the “Research” button)
- 4got starts the AI research agent as a Server-Sent Events (SSE) stream
- The agent fetches top search result URLs, reads them, and synthesizes an answer
- Intermediate findings stream to the user in real time via the oracle box
Configuration
Edit data/ai-research.kdl:
enabled true
mode "lua"
provider "cerebras"
model "llama3.1-8b"
api-key "csk-..."
base-url "https://api.cerebras.ai/v1"
max-runtime-seconds 60
require "deluxe"
Fields
| Field | Description |
|---|---|
enabled | Master switch (true/false) |
mode | Execution mode: “lua” (sandboxed, recommended) or “bash” (dangerous, host access) |
provider | LLM provider: “cerebras”, “openrouter”, “ollama”, “openai”, or any OpenAI-compatible endpoint |
model | Model name (provider-specific) |
api-key | API key for the provider |
base-url | Override the API endpoint URL. Defaults are inferred from provider name. |
max-runtime-seconds | Kill the agent after this many seconds |
require | Group permission required to use the agent (e.g. “deluxe”). Empty = everyone. |
Providers
Cerebras (recommended for free tier):
provider "cerebras"
model "llama3.1-8b"
api-key "csk-..."
base-url "https://api.cerebras.ai/v1"
Cerebras offers free API access for Llama 3.1 8B with extremely fast inference (~1000 tok/s). Good enough for research synthesis.
OpenRouter (access to many models):
provider "openrouter"
model "anthropic/claude-sonnet-4-20250514"
api-key "sk-or-..."
Ollama (local, no API key needed):
provider "ollama"
model "llama3.1:8b"
base-url "http://localhost:11434/v1"
Any OpenAI-compatible API:
provider "custom"
model "your-model"
api-key "your-key"
base-url "https://your-api.example.com/v1"
Metaprompt customization
The agent’s system prompt is loaded from data/metaprompt.md. This file controls the agent’s personality, tool usage patterns, and output format.
Key sections in the default metaprompt:
- Context: explains to the LLM where it is and what 4got does
- Tools: describes each tool and when to use it
- Behavior rules: stream findings via send(), always use eta() before slow ops, fetch 2-3 URLs minimum
- Context management: how to use pin() and forget() to manage the limited context window
You can customize the metaprompt to change the agent’s behavior. For example, to make it more concise:
Add to metaprompt.md:
## Output length
Keep your final answer under 200 words. Be direct.
Or to make it focus on academic sources:
## Source priority
Prefer academic sources (arxiv.org, scholar.google.com, .edu domains) over news articles.
Always check if there's a relevant arXiv paper.
Tool reference
send(text)
Shows text to the user immediately as a streaming update. The user sees a live feed of the agent’s findings. Use frequently for intermediate results.
send("Found that Rust 1.84 adds async trait support. Checking release notes...")
eta(seconds, job_name)
Updates the ETA countdown timer shown to the user. Call before any slow operation (fetching URLs, running code).
eta(5, "fetching rust-lang.org blog")
fetch(url)
Fetches a URL and returns the page text with HTML tags stripped. Max 100KB response, 10-second timeout. Supports chunked reading for large pages.
fetch("https://blog.rust-lang.org/")
For large pages, request subsequent chunks:
fetch("https://example.com/long-article", chunk=2)
lua(code)
Executes Lua code in a sandboxed environment. Available standard libraries:
- string:
string.find,string.sub,string.gsub,string.format, etc. - table:
table.insert,table.remove,table.sort,table.concat - math:
math.floor,math.ceil,math.random,math.pi, etc. - json_decode(s): Parse a JSON string into a Lua table
- json_encode(t): Serialize a Lua table to JSON
- http_get(url): Fetch a URL and return the body as a string (uses the Go HTTP client with timeouts)
Example:
local data = http_get("https://api.example.com/data.json")
local parsed = json_decode(data)
for _, item in ipairs(parsed.results) do
print(item.title .. ": " .. item.score)
end
pin(text)
Pins a short note to the agent’s context that will never be evicted during context window management. Use to preserve key findings across a long research session. Keep pins to 1-2 sentences.
pin("Key finding: the paper by Smith et al. (2026) shows 99.9% error correction rate using surface codes.")
forget(tool_call_id)
Removes a previous tool result from the agent’s context to free up space. Use after extracting what you need from a large fetch result.
forget("call_abc123")
friction(description)
Silently logs an improvement suggestion. These are recorded for the server operator to review. The user never sees them.
friction("No academic search engine configured - adding arXiv would improve research quality")
Double-search trigger UX
The research agent activates when a user searches the same query twice. The UI flow:
- First search: normal results displayed
- Second search (same query): a “Researching…” panel appears above results with a live SSE stream
- The agent’s intermediate send() calls appear as streaming text
- The eta() timer shows in the corner
- When complete, the final answer replaces the streaming panel
The SSE endpoint is GET /api/v1/research?q=<query>. It returns events:
event: message— intermediate text from send()event: eta— ETA update from eta()event: done— final answer, stream endsevent: error— error message, stream ends
Lua sandbox capabilities
The Lua sandbox (mode "lua") provides a safe execution environment:
Allowed:
- All standard string/table/math operations
json_decode()andjson_encode()for data processinghttp_get()for fetching URLs (through Go’s HTTP client with timeout and SSRF protection)print()for output (captured as the tool result)tostring(),tonumber(),type(),pairs(),ipairs(),unpack()
Blocked:
- File system access (
io,os) require(),dofile(),loadfile()debuglibrary- Arbitrary package loading
- Long-running loops (killed by max-runtime-seconds)
Bash mode warning: Setting mode "bash" gives the agent full shell access on the host. Only use this if 4got is running inside an isolated container (LXC, Docker, VM).
Friction feedback system
The friction() tool lets the agent report issues it encounters during research. These are logged to the database and can be reviewed by the server operator.
Common friction reports:
- Missing engine types (e.g. “no academic engine for arXiv queries”)
- Poor result quality for specific query types
- Timeout issues with upstream services
- Suggestions for new oracle types
Friction logs are stored in the ai_research table and can be queried via the admin panel.
Plugin System
The plugin system supports Lua plugins and MCP tool integration.
Lua plugins
Lua plugins are the primary extension mechanism. Drop a .lua file in the plugins/ directory (relative to the data directory) and it will be loaded at startup.
Hooks
Plugins can register handlers for these hooks:
| Hook | Arguments | Return | When |
|---|---|---|---|
on_search(query, category, params) | query string, category string, search params table | modified query string (or nil to keep original) | Before search is dispatched to engines |
on_result(result, engine_id) | result table, engine ID string | modified result table, or nil to filter it out | After each engine returns results, before merging |
on_render(html, query, category) | HTML string, query, category | modified HTML string | After the results page HTML is generated, before sending to client |
Plugin structure
-- plugins/example.lua
-- Plugin metadata
plugin = {
name = "Example Plugin",
version = "1.0",
author = "your name",
description = "An example plugin"
}
-- Called before search dispatch
function on_search(query, category, params)
-- Modify query, return nil to keep original
return nil
end
-- Called for each result from each engine
function on_result(result, engine_id)
-- Return the result to keep it, nil to filter it out
return result
end
-- Called after HTML is generated
function on_render(html, query, category)
-- Inject custom HTML, modify existing HTML, etc.
return html
end
Available Lua APIs
Plugins have access to the same sandbox as the AI research agent:
- string, table, math standard libraries
json_decode(s)/json_encode(t)for JSON processinghttp_get(url)for fetching URLslog(message)for logging to the server logconfig_get(key)for reading plugin-specific config values
Example: Ad domain blocker
-- plugins/block_ads.lua
plugin = {
name = "Ad Blocker",
version = "1.0",
description = "Filters results from known advertising domains"
}
local ad_domains = {
["doubleclick.net"] = true,
["googleadservices.com"] = true,
["facebook.com/ads"] = true,
["amazon.com/gp/slredirect"] = true,
["taboola.com"] = true,
["outbrain.com"] = true,
["criteo.com"] = true,
}
-- Extract domain from URL
local function get_domain(url)
local domain = url:match("https?://([^/]+)")
if domain then
domain = domain:gsub("^www%.", "")
return domain
end
return ""
end
function on_result(result, engine_id)
local domain = get_domain(result.url or "")
if ad_domains[domain] then
log("Blocked ad domain: " .. domain)
return nil -- filter out
end
return result
end
Example: Result enrichment
-- plugins/enrich_github.lua
plugin = {
name = "GitHub Enricher",
version = "1.0",
description = "Adds star count and language to GitHub repo results"
}
function on_result(result, engine_id)
local url = result.url or ""
local owner, repo = url:match("github%.com/([^/]+)/([^/]+)")
if owner and repo then
-- Fetch repo info from GitHub API
local api_url = "https://api.github.com/repos/" .. owner .. "/" .. repo
local data = http_get(api_url)
if data then
local info = json_decode(data)
if info and info.stargazers_count then
result.snippet = result.snippet ..
" [★" .. tostring(info.stargazers_count) ..
" | " .. (info.language or "unknown") .. "]"
end
end
end
return result
end
MCP for AI tools
The AI research agent can be extended with MCP (Model Context Protocol) servers to give it access to additional tools beyond the built-in set.
How MCP integration works
- Configure MCP servers in
ai-research.kdl(or theai-researchsection of your unified KDL config) - At startup, 4got connects to each MCP server and discovers available tools
- These tools are added to the AI research agent’s tool list alongside the built-in tools
- When the agent calls an MCP tool, 4got proxies the call to the appropriate MCP server
Configuration
// In ai-research.kdl
mcp-server "local-kb" {
command "python3" "-m" "knowledge_base_mcp"
env "KB_PATH=/path/to/knowledge-base"
}
mcp-server "calculator" {
url "http://localhost:3001/mcp"
}
MCP servers can be launched as subprocesses (command) or connected to over HTTP (url).
Example: Local knowledge base MCP server
Create an MCP server that gives the AI agent access to a local knowledge base:
# knowledge_base_mcp.py
import json
import sys
import os
from pathlib import Path
KB_PATH = Path(os.environ.get("KB_PATH", "./kb"))
def search_kb(query):
"""Search local markdown files for relevant content."""
results = []
for md_file in KB_PATH.glob("**/*.md"):
content = md_file.read_text()
if query.lower() in content.lower():
# Extract surrounding context
idx = content.lower().index(query.lower())
start = max(0, idx - 200)
end = min(len(content), idx + 200)
results.append({
"file": str(md_file.relative_to(KB_PATH)),
"excerpt": content[start:end]
})
return results[:5]
# MCP protocol handler
def handle_request(request):
if request["method"] == "tools/list":
return {
"tools": [{
"name": "search_knowledge_base",
"description": "Search the local knowledge base for information",
"inputSchema": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"}
},
"required": ["query"]
}
}]
}
elif request["method"] == "tools/call":
tool_name = request["params"]["name"]
if tool_name == "search_knowledge_base":
query = request["params"]["arguments"]["query"]
results = search_kb(query)
return {"content": [{"type": "text", "text": json.dumps(results, indent=2)}]}
return {"error": "unknown method"}
# stdio transport
for line in sys.stdin:
request = json.loads(line)
response = handle_request(request)
response["id"] = request.get("id")
print(json.dumps(response), flush=True)
Example: Giving the AI agent calculator access
If the built-in calculator oracle isn’t sufficient, you could connect a Wolfram Language MCP server:
mcp-server "wolfram-lang" {
command "wolframscript" "-code" "MCPServer[]"
}
This would let the agent evaluate arbitrary Wolfram Language expressions during research.
Plugin loading order
- All
.luafiles inplugins/are loaded alphabetically - MCP servers are connected in the order they appear in config
- Plugins are loaded after all engines are registered but before the HTTP server starts
- Plugin hooks run in load order; the first
on_searchreturn value that is non-nil wins
Plugin safety
- Lua plugins run in the same sandbox as the AI research agent (no file/OS access)
http_get()goes through Go’s HTTP client with SSRF protection- Plugins that throw errors are logged and disabled, they don’t crash the server
- MCP servers run as separate processes with their own permissions
- Plugin execution has a per-call timeout (configurable, default 5 seconds)
Planned but not yet implemented
- Plugin hot-reloading (SIGHUP or admin endpoint to reload without restart)
- Per-plugin configuration in KDL
- Plugin marketplace/registry
- TypeScript/WASM plugin support
- Plugin-specific rate limiting
- Plugin metrics in the admin dashboard
Adding a Language
4got ships with 62 locale files. To add a new language:
Steps
-
Copy
data/locales/en.kdltodata/locales/{code}.kdlwhere{code}is the ISO 639-1 language code (e.g.fr,de,zh). -
Translate every key. The format is
key "translated value"— one per line. -
Do NOT include
about_*keys in non-English locales. The about page renders in English only. -
RTL languages (
ar,fa,he,ur,yi,dv) get automaticdir="rtl"on the HTML tag. No extra work needed. -
Restart the server or send SIGUSR1 to hot-reload.
File naming
Use standard ISO 639-1 codes. For regional variants, use underscore: pt_BR.kdl, zh_TW.kdl.
Testing
Set the language cookie manually: document.cookie = "language=fr; path=/; max-age=34560000" and refresh. Or use the language setting in /settings if it exists for your locale.
Contributing to 4got
Dev environment setup
- Clone the repo and
cdinto it. - Install Go 1.25+ (the
go.modrequires 1.25.0). - Build:
cd go-server && CGO_ENABLED=1 go build -o 4got . - Copy
data/config.default.kdltodata/config.kdland set yourowner-secret. - Run:
cd .. && go-server/4got
CGO is required for the SQLite driver.
Adding a new KDL engine
Generic engines are defined in data/engines.kdl and parsed by go-server/generic_engine.go. See engine-spec.md for the full KDL schema. The fastest workflow is the admin playground at /admin/engines/new (owner-only): fill in the form, test against a live query, then click “Save Engine” to append the KDL block to disk and register it without a restart.
Adding a new oracle
Oracles are instant-answer widgets shown above search results (calculator, unit converter, currency, etc.).
- Write your oracle function in
go-server/oracles_extra.go. Follow the existing pattern: the function takes a query string (and optionally*http.Request,http.ResponseWriter,Config) and returns*OracleAnsweror nil. - Register it in
go-server/oracles.goinsideCheckOracles(). Add a call to your function in the appropriate section (local oracles, API oracles, or knowledge oracles). Order matters: the first match in the local oracle section wins.
Running tests
cd go-server && CGO_ENABLED=1 go test ./...
Code style
- No comments unless the WHY is non-obvious. Do not explain what the code does if it is clear from reading it.
- AI-generated files get a header:
// AI-generated: <model> - Keep functions short. If a function needs a comment block explaining it, it should probably be smaller.
PR expectations
CGO_ENABLED=1 go build -o /dev/null .must pass.go vet ./...must be clean.- Do not introduce new linter warnings.