Federation: The Friend Network

4got instances can form a peer-to-peer friend network for cache sharing, service proxying, and instance discovery. This is how multiple 4got deployments cooperate to improve speed and resilience.

Philosophy

The 4got protocol is a result-sharing protocol for self-contained queries — queries that are small, universal, and should have roughly the same answer regardless of who asks or where they ask from. Search engines are just one type of intelligence backend that can feed results into this protocol. A friend’s instance might answer a query from Google, from a custom scraper, from an AI agent, from a local database, or from another friend’s cache — the protocol doesn’t care. It exists to share answers between friends, not to replace any particular search provider. There is room for the protocol to be used with all kinds of intelligence backends beyond traditional search engines.

Overview

The friend network provides three capabilities:

1. Peer cache sharing — friends share cached search results, reducing redundant upstream queries
2. Service proxy — friends proxy expensive services (translation, Wolfram Alpha, autocomplete) for each other
3. Instance discovery — the /ami4got endpoint and instance browser let users find other instances

Peer cache sharing protocol

When a search is performed, 4got queries all configured friends in parallel via GET /api/v1/cache?q=<query>&cat=<category>. Friends respond with cached results if available.

Request flow

1. User searches for “quantum computing”
2. 4got sends the query to all configured friends simultaneously
3. Friends check their local cache and respond with results (or 204 No Content)
4. 4got merges friend results with its own engine results using the standard scoring/deduplication algorithm
5. If store-friend-cache is enabled, results from friends are stored in the local cache

Wait policy

The wait-for setting in peers.kdl controls how many friends must respond before proceeding:

The wait policy interacts with the fanout hard window (fanout-hard-window-ms in config.kdl). Friends that don’t respond within the hard window are skipped regardless of the wait policy.

Failure tracking

The failure-threshold setting tracks friend reliability:

failure-threshold 0.01 1d

This means: if a friend fails more than 1% of requests over a 1-day window, consider them “down” and don’t wait for them. They’ll be retried periodically but won’t block search results.

Always-wait and preferred friends

// Always wait for this friend's cache (even if slow)
always-wait-for "alice"

// Prefer this friend for specific services
prefer-friend "alice" for="translation,wolfram"

Service proxy

Friends can proxy expensive API-backed services for each other. This lets a single Wolfram Alpha API key serve multiple instances, or a single translation backend serve the whole friend network.

How it works

1. Instance A needs to translate text but has no DeepL API key
2. Instance A sends POST /api/v1/proxy to friend Instance B with {"service": "translation", "params": {"text": "hello", "target": "fr"}}
3. Instance B checks its allow-proxy list, finds “translation” is allowed, performs the translation, and returns the result
4. Instance A uses the result as if it performed the translation locally

Supported services

• translation — text translation (DeepL, Papago, Lingva, NLLB, Microsoft)
• wolfram — Wolfram Alpha short answers API
• autocomplete — search suggestion backends

Configuration

On the provider instance (the one that actually has the API keys):

allow-proxy "translation" "wolfram" "autocomplete"

On the consumer instance (the one that wants to use the services):

prefer-friend "provider-name" for="translation,wolfram"

The consumer will try the preferred friend first, then fall back to local backends.

Authentication

Service proxy requests are authenticated using the peer-secret from config.kdl, sent as a Bearer token.

Instance discovery

The ami4got endpoint

Every 4got instance exposes GET /ami4got (inspired by 4get’s /ami4get.php). This returns a JSON object with instance metadata:

{
  "version": "2.0",
  "api_version": "1",
  "engine_count": 15,
  "stats": {
    "hits": 12345,
    "real_searches": 6789
  }
}

Instance browser

The /instances page shows a list of known 4got instances. It fetches info from configured friends and public instances periodically.

Configure in instances.kdl:

fetch-interval-min 60

friends {
    instance "https://search.friend.example.com"
}

public {
    instance "https://public.4got.example.com"
}

Friend-of-friend discovery

When Instance A fetches /ami4got from Instance B, it can discover Instance B’s own friends. This creates an organic network where adding a single friend connection transitively discovers the broader network.

Setting up two instances as friends

Step 1: Choose a shared secret

Both instances need the same peer-secret in their config.kdl:

peer-secret "a_strong_random_string_shared_between_friends"

Step 2: Configure peers on each instance

On Instance A (data/peers.kdl):

wait-for 1.0
store-friend-cache true
allow-proxy "translation" "wolfram"

peer "instance-b" {
    url "https://search-b.example.com"
}

On Instance B (data/peers.kdl):

wait-for 1.0
store-friend-cache true
allow-proxy "translation" "wolfram"

peer "instance-a" {
    url "https://search-a.example.com"
}

Step 3: Add to instance browser (optional)

On both instances, add each other to instances.kdl:

friends {
    instance "https://search-b.example.com"
}

Step 4: Test

1. Restart both instances
2. Search on Instance A — check the logs for “cached on 1 friend” in the results
3. Visit /admin/log to see engine stats (friend results appear as (friend) entries)

Step 5: Optional mTLS

For additional security, configure mutual TLS between friends:

peer "instance-b" {
    url "https://search-b.example.com"
    cert "/path/to/client.crt"
    key "/path/to/client.key"
}

Both instances need to trust each other’s client certificates.