Engines & Providers¶
AtDork searches through two kinds of engines:
- Built-in scraping engines — provided by the
ddgslibrary, no account needed (default behaviour). - Third-party API providers — official paid/free search APIs
(Serper.dev, SerpApi, Brave Search API), configured under
engines:inatdork.yml. Deterministic results, higher reliability, but they consume your API credits.
Both kinds are selected the same way — with --backend, search.backend, or
a profiles: chain — and both feed the identical downstream pipeline
(validation, filtering, dedupe, cache, resume, output formats).
Built-in engines (scraping)¶
| Backend | Notes |
|---|---|
auto |
Default: ddgs picks its best source per query. |
google |
Google via ddgs. |
bing |
Bing via ddgs. |
duckduckgo |
DuckDuckGo (first entry of the default fallback chain). |
brave |
Brave scraping backend — unless overridden by engines.brave (API). |
mojeek |
Mojeek — light scraping footprint. |
startpage |
Startpage. |
wikipedia |
Wikipedia (last default fallback). |
yandex, yahoo |
Available via ddgs. |
The default fallback chain (when search.fallback is not configured) is:
atdork -q "inurl:admin" # auto
atdork -q "inurl:admin" --backend google # force one scraping backend
atdork -q "test" --no-fallback-backends # single backend only, no chain
No API key, no account, no config required.
API providers¶
Add them under engines: in atdork.yml (see
Configuration):
version: 2
engines:
serper:
type: api
api_key: ${SERPER_API_KEY}
qps: 1
serpapi:
type: api
api_key: ${SERPAPI_KEY}
qps: 1
brave:
type: api
api_key: ${BRAVE_API_KEY}
qps: 1
export SERPER_API_KEY=... # or SERPAPI_KEY / BRAVE_API_KEY
atdork -q "site:github.com password" --backend serper
Supported providers at a glance¶
| Engine | Endpoint | Key env var | Free tier* | Per-request cap | qps default |
|---|---|---|---|---|---|
serper |
POST google.serper.dev/search |
SERPER_API_KEY |
one-time credits (≈2,500) | 100 | 1.0 |
serpapi |
GET serpapi.com/search |
SERPAPI_KEY |
100 searches/month | 100 | 1.0 |
brave |
GET api.search.brave.com/res/v1/web/search |
BRAVE_API_KEY |
2,000 queries/month | 20 | 1.0 |
* Free-tier figures are what the providers published last we checked —
always confirm on their pricing pages. qps is AtDork's local limiter
(minimum interval between your own requests); raise or lower it to match
your plan (qps: 0 disables it).
Parameter support matrix¶
How AtDork's common search parameters map per provider:
| Parameter | serper | serpapi | brave |
|---|---|---|---|
query + max_results |
✅ q, num |
✅ q, num |
✅ q, count (max 20) |
region (us-en) |
✅ gl/hl |
✅ gl/hl |
✅ country/search_lang |
safesearch |
⚪ ignored | ⚪ ignored (engine-dependent) | ✅ safesearch off/moderate/strict |
timelimit (d/w/m/y) |
⚪ ignored | ✅ tbs=qdr:d… |
✅ freshness=pd/pw/pm/py |
| proxy + custom UA | ✅ applied per request | ✅ | ✅ |
Provider-specific options can be passed through engines.<name> and land in
the request (e.g. engine: google-news for serpapi, location: Paris, France
for serper, offset: 20 for brave):
engines:
serpapi:
type: api
api_key: ${SERPAPI_KEY}
engine: google
tbs: "" # extra query params via options
Result normalisation¶
Every provider response is normalised to AtDork's internal shape —
{title, href, body} — so filters, dedupe, cache, resume and all output
formats behave identically to scraping results.
Backend resolution & override rules¶
For every backend name AtDork tries, resolution order is:
- Registry — is the name configured under
engines:? → use the API provider. A configured name overrides a built-in backend with the same name (engines.brave→ Brave Search API instead of ddgs scraping). - Otherwise → built-in
ddgsscraping path.
# API directly
atdork -q "inurl:test" --backend serper
# explicit chain: scraping first, API last
atdork -q "inurl:test" --backend google,serper
# profile from atdork.yml
atdork -q "inurl:test" --backend resilient
The fallback chain¶
Behaviour:
| Situation | Queue |
|---|---|
backend: auto + chain configured |
auto first, then the chain |
| single backend + chain | that backend, then the chain |
comma list (google,serper) |
used as-is |
| profile name | profile list as-is (even with --no-fallback-backends) |
| no chain configured | built-in ddgs default chain |
What happens on provider errors:
| Error kind | Detected from | Behaviour |
|---|---|---|
quota (credits exhausted) |
HTTP 402/403, body keywords, serpapi error field |
provider skipped, next backend immediately, no retry |
rate_limit (429) |
HTTP 429 | same as quota (treated as rate limit) |
auth (bad key) |
HTTP 401/403 | skipped, clear log line |
config (key not set) |
empty api_key |
skipped; raised at the end if nothing was found |
network / parse / api |
timeouts, bad JSON, other 4xx/5xx | retried on the same backend with backoff, then next |
Credit caution: a provider in the chain is queried for every query that reaches it — including queries that legitimately have zero results (an empty scraper result advances to the next backend). If API credits are precious, either keep API engines out of
fallbackand invoke them explicitly with--backend serper, or make them the first entry so you pay only one API call per query instead of chain-tail calls after every empty scrape.
Configuration reference for engines.<name>¶
| Key | Default | Description |
|---|---|---|
type |
api |
Only api is supported (validated at load). |
provider |
(name) | Implementation when the key differs from the class name (e.g. two serper accounts: myserper: {provider: serper, ...}). |
api_key |
— | ${ENV} interpolation strongly recommended; unset ${ENV} fails at load, empty value fails at query time with a clear error. |
qps |
1.0 |
Local rate limit (requests/second, 0 = unlimited). Shared across worker threads. |
timeout |
10 |
HTTP timeout in seconds. |
| (extra) | — | Passed to the provider as a request option (engine, gl, hl, location, country, search_lang, offset, ...). |
Troubleshooting¶
| Symptom | Cause / fix |
|---|---|
Config error: ... engines.serper.api_key: environment 'SERPER_API_KEY' is not set |
Export the variable, use ${SERPER_API_KEY:-}, or remove the engine. |
engines.serper.api_key is not set — export SERPER_API_KEY... at query time |
Engine configured with an empty/literal-empty key; set the key. |
API quota exceeded (rate limit): ... in logs |
Out of credits — AtDork moves to the next backend automatically. Check your provider dashboard. |
Rate limited: ... |
Provider throttling — lower qps, or let the chain continue. |
Authentication failed: ... |
Key invalid/rotated — regenerate and update the env var. |
unknown provider 'x' (supported: brave, serpapi, serper) |
provider: / engine name typo in engines:. |
| Brave results stop at 20 | Free-tier per-request cap (count ≤ 20); paginate via offset option or use another provider for larger max_results. |
Keys never appear in logs: requests/responses are passed through the
redactor (api_key=***, ***last4, masked URL credentials), so it is safe to
run with --debug when diagnosing provider issues.