AI Research Agent

The AI research agent is a tool-using LLM that synthesizes search results into a comprehensive answer. It is triggered when a user searches the same query twice (the “double-search” trigger), indicating they want deeper analysis than raw search results can provide.

How it works

1. User searches “quantum error correction”
2. 4got returns normal search results
3. User searches “quantum error correction” again (or clicks the “Research” button)
4. 4got starts the AI research agent as a Server-Sent Events (SSE) stream
5. The agent fetches top search result URLs, reads them, and synthesizes an answer
6. Intermediate findings stream to the user in real time via the oracle box

Configuration

Edit data/ai-research.kdl:

enabled true
mode "lua"
provider "cerebras"
model "llama3.1-8b"
api-key "csk-..."
base-url "https://api.cerebras.ai/v1"
max-runtime-seconds 60
require "deluxe"

Fields

Providers

Cerebras (recommended for free tier):

provider "cerebras"
model "llama3.1-8b"
api-key "csk-..."
base-url "https://api.cerebras.ai/v1"

Cerebras offers free API access for Llama 3.1 8B with extremely fast inference (~1000 tok/s). Good enough for research synthesis.

OpenRouter (access to many models):

provider "openrouter"
model "anthropic/claude-sonnet-4-20250514"
api-key "sk-or-..."

Ollama (local, no API key needed):

provider "ollama"
model "llama3.1:8b"
base-url "http://localhost:11434/v1"

Any OpenAI-compatible API:

provider "custom"
model "your-model"
api-key "your-key"
base-url "https://your-api.example.com/v1"

Metaprompt customization

The agent’s system prompt is loaded from data/metaprompt.md. This file controls the agent’s personality, tool usage patterns, and output format.

Key sections in the default metaprompt:

• Context: explains to the LLM where it is and what 4got does
• Tools: describes each tool and when to use it
• Behavior rules: stream findings via send(), always use eta() before slow ops, fetch 2-3 URLs minimum
• Context management: how to use pin() and forget() to manage the limited context window

You can customize the metaprompt to change the agent’s behavior. For example, to make it more concise:

Add to metaprompt.md:

## Output length
Keep your final answer under 200 words. Be direct.

Or to make it focus on academic sources:

## Source priority
Prefer academic sources (arxiv.org, scholar.google.com, .edu domains) over news articles.
Always check if there's a relevant arXiv paper.

Tool reference

send(text)

Shows text to the user immediately as a streaming update. The user sees a live feed of the agent’s findings. Use frequently for intermediate results.

send("Found that Rust 1.84 adds async trait support. Checking release notes...")

eta(seconds, job_name)

Updates the ETA countdown timer shown to the user. Call before any slow operation (fetching URLs, running code).

eta(5, "fetching rust-lang.org blog")

fetch(url)

Fetches a URL and returns the page text with HTML tags stripped. Max 100KB response, 10-second timeout. Supports chunked reading for large pages.

fetch("https://blog.rust-lang.org/")

For large pages, request subsequent chunks:

fetch("https://example.com/long-article", chunk=2)

lua(code)

Executes Lua code in a sandboxed environment. Available standard libraries:

• string: string.find, string.sub, string.gsub, string.format, etc.
• table: table.insert, table.remove, table.sort, table.concat
• math: math.floor, math.ceil, math.random, math.pi, etc.
• json_decode(s): Parse a JSON string into a Lua table
• json_encode(t): Serialize a Lua table to JSON
• http_get(url): Fetch a URL and return the body as a string (uses the Go HTTP client with timeouts)

Example:

local data = http_get("https://api.example.com/data.json")
local parsed = json_decode(data)
for _, item in ipairs(parsed.results) do
    print(item.title .. ": " .. item.score)
end

pin(text)

Pins a short note to the agent’s context that will never be evicted during context window management. Use to preserve key findings across a long research session. Keep pins to 1-2 sentences.

pin("Key finding: the paper by Smith et al. (2026) shows 99.9% error correction rate using surface codes.")

forget(tool_call_id)

Removes a previous tool result from the agent’s context to free up space. Use after extracting what you need from a large fetch result.

forget("call_abc123")

friction(description)

Silently logs an improvement suggestion. These are recorded for the server operator to review. The user never sees them.

friction("No academic search engine configured - adding arXiv would improve research quality")

Double-search trigger UX

The research agent activates when a user searches the same query twice. The UI flow:

1. First search: normal results displayed
2. Second search (same query): a “Researching…” panel appears above results with a live SSE stream
3. The agent’s intermediate send() calls appear as streaming text
4. The eta() timer shows in the corner
5. When complete, the final answer replaces the streaming panel

The SSE endpoint is GET /api/v1/research?q=<query>. It returns events:

• event: message — intermediate text from send()
• event: eta — ETA update from eta()
• event: done — final answer, stream ends
• event: error — error message, stream ends

Lua sandbox capabilities

The Lua sandbox (mode "lua") provides a safe execution environment:

Allowed:

• All standard string/table/math operations
• json_decode() and json_encode() for data processing
• http_get() for fetching URLs (through Go’s HTTP client with timeout and SSRF protection)
• print() for output (captured as the tool result)
• tostring(), tonumber(), type(), pairs(), ipairs(), unpack()

Blocked:

• File system access (io, os)
• require(), dofile(), loadfile()
• debug library
• Arbitrary package loading
• Long-running loops (killed by max-runtime-seconds)

Bash mode warning: Setting mode "bash" gives the agent full shell access on the host. Only use this if 4got is running inside an isolated container (LXC, Docker, VM).

Friction feedback system

The friction() tool lets the agent report issues it encounters during research. These are logged to the database and can be reviewed by the server operator.

Common friction reports:

• Missing engine types (e.g. “no academic engine for arXiv queries”)
• Poor result quality for specific query types
• Timeout issues with upstream services
• Suggestions for new oracle types

Friction logs are stored in the ai_research table and can be queried via the admin panel.