Engine KDL Specification

This document specifies the KDL format for defining generic engines in data/engines.kdl. Generic engines are parsed by go-server/generic_engine.go and registered alongside hand-written engines (Brave, Google, etc.) at startup. A single engines.kdl file defines all ~274 generic engines.

Engine block structure

Each engine is a top-level engine node with a quoted string ID and a child block of properties:

engine "my_engine" {
    name "My Engine Display Name"
    categories "web" "news"
    search-url "https://example.com/search?q={query}&page={page}"
    method "GET"
    result-type "json"
    results-path "data.items"
    title-path "title"
    url-path "link"
    content-path "description"
    page-size 20
    paging true
    first-page 1
}

The engine ID (the string after engine) is the unique identifier used in logs, cache keys, weight cookies, and the admin UI. It must be a short slug (lowercase, underscores allowed).

Attributes reference

Identity

Request

Response parsing – HTML mode

Used when result-type is "html" (or "HTML" or "xpath").

Response parsing – JSON mode

Used when result-type is "json".

URL construction

Pagination

Content processing

Lifecycle

Federation

Examples

JSON API engine (MDN)

engine "mdn" {
    name "MDN Web Docs"
    categories "web"
    search-url "https://developer.mozilla.org/api/v1/search?q={query}&page={page}"
    method "GET"
    result-type "json"
    results-path "documents"
    title-path "title"
    url-path "mdn_url"
    url-prefix "https://developer.mozilla.org"
    content-path "summary"
    page-size 10
    paging true
    first-page 1
}

HTML scraping engine (Anaconda)

engine "anaconda" {
    name "Anaconda"
    categories "web"
    search-url "https://anaconda.org/search?q={query}&page={page}"
    method "GET"
    result-type "html"
    url-prefix "https://anaconda.org"
    results-selector "tbody tr"
    title-selector "td h5"
    url-selector "td h5 a:last-of-type"
    content-selector "td p"
    page-size 10
    paging true
    first-page 0
}

Engine with custom headers (Crossref)

engine "crossref" {
    name "Crossref"
    categories "web"
    search-url "https://api.crossref.org/works?query={query}&rows=20&offset={offset}"
    method "GET"
    result-type "json"
    results-path "message.items"
    title-path "title"
    url-path "URL"
    content-path "abstract"
    content-html-to-text true
    headers {
        "User-Agent" "4got/1.0 (https://4got.org; mailto:admin@4got.org)"
    }
    page-size 20
    paging true
    first-page 0
}

POST engine with body template (ArtStation)

engine "artstation" {
    name "www.artstation.com"
    categories "images"
    search-url "https://www.artstation.com/api/v2/search/projects.json"
    method "POST"
    result-type "json"
    results-path "data"
    title-path "title"
    url-path "permalink"
    content-path "description"
    thumbnail-path "smaller_square_cover_url"
    body-template "{\"query\":\"{query}\",\"page\":{page},\"per_page\":20,\"sorting\":\"relevance\",\"pro_first\":\"1\",\"filters\":[],\"additional_fields\":[\"description\"]}"
    page-size 20
    paging true
    first-page 1
}

URL template engine (Art Institute of Chicago)

engine "artic" {
    name "www.artic.edu"
    categories "images"
    search-url "https://api.artic.edu/api/v1/artworks/search?q={query}&page={page}&limit=20"
    method "GET"
    result-type "json"
    results-path "data"
    title-path "title"
    url-path "id"
    url-template "https://www.artic.edu/artworks/{id}"
    content-path "thumbnail.alt_text"
    page-size 20
    paging true
    first-page 1
}

How the generic engine factory works

At startup, LoadAndRegisterGenericEngines() in generic_engine.go:

1. Reads data/engines.kdl via resolveKDLSection() (which supports both standalone files and engines { ... } sections inside a unified KDL file).
2. Parses the raw bytes into []GenericEngineConfig via parseEngineKDL(), a line-by-line KDL parser that handles engine blocks, headers sub-blocks, and all property types.
3. Skips entries with empty ID, no categories, or disabled true.
4. Sets defaults: page-size defaults to 10, name defaults to the ID.
5. Creates a GenericEngine struct (which implements the Engine interface) for each config and registers it in the global engine registry.
6. Determines the share policy: explicit share attribute wins; otherwise auto-detects based on whether the URL or headers contain API key patterns.

When Search() is called on a GenericEngine:

1. Substitutes {query}, {page}, {offset}, {lang} in the URL template (and body template if POST).
2. Makes the HTTP request, routing through Tor if tor-outbound is set.
3. Records the raw HTML for drift detection (drift.go).
4. Dispatches to parseHTML() (goquery CSS selectors) or parseJSON() (dot-path navigation) based on result-type.
5. Returns []WebResult with each result’s Engine field set to the engine ID.

Testing a new engine via the admin playground

The admin playground at /admin/engines/new (owner-only) provides a web form for configuring and testing generic engines without editing KDL files directly.

1. Log in as owner at /admin/login.
2. Navigate to /admin/engines/new.
3. Fill in the engine identity (ID, display name, categories).
4. Set the search URL template with {query} and {page} placeholders.
5. Choose HTML or JSON result type and fill in the corresponding selectors/paths.
6. Enter a test query and click “Test Engine” – this POSTs to /admin/engines/test, which creates a temporary GenericEngine in memory, runs a search, and returns results as JSON displayed in the form.
7. Iterate on selectors until results look correct.
8. Click “Save Engine” to append the generated KDL block to data/engines.kdl and register it in memory immediately (no restart needed).

The test endpoint does not persist anything – only “Save Engine” writes to disk.