{
  "markdown": "# crawl-census-client\n\n**Ask before you fetch.** A drop-in client that stops your crawler spending requests on doors\nthat are shut, and stops it routing around content someone is trying to sell.\n\nReading robots.txt answers one question and hides two others. Measured across **23,482 domains**\nby [Crawl Census](https://crawlcensus.com):\n\n- **2,874 domains permit AI agents in robots.txt and then refuse those same agents at the\n  network edge.** A parser sees permission; the fetch returns 403. You pay for the round trip\n  and get nothing.\n- **208 domains answer an AI user agent with `HTTP 402 Payment Required`.** That is a price,\n  not a refusal. Treating it as a block walks away from content the operator wants to sell you.\n  Retrying around it takes something they are charging for.\n\nNo dependencies. No key required.\n\n## MCP server\n\nThe same measurement is exposed as a remote MCP server, so an agent can ask before it fetches\nrather than after it fails. Listed in the\n[official MCP registry](https://registry.modelcontextprotocol.io) as\n`io.github.taylorsmithgg/crawl-census`.\n\n```json\n{ \"mcpServers\": { \"crawl-census\": { \"url\": \"https://crawlcensus.com/mcp\" } } }\n```\n\n| Tool | Answers |\n|---|---|\n| `crawl_preflight` | will these domains serve my agent, refuse it, or charge it? |\n| `agent_profile` | what does this census publish about my crawler, and how do I correct it? |\n| `census_facts` | the headline findings as dated records with denominators and citation lines |\n| `site_report` | the stored audit for one domain |\n| `scan_site` | measure a domain now |\n| `census_stats` | corpus-level totals |\n\nNo authentication for read tools. Streamable HTTP.\n\n## Install\n\n```bash\nnpm i github:taylorsmithgg/crawl-census-client\npip install git+https://github.com/taylorsmithgg/crawl-census-client\n```\n\n## Use\n\n```js\nimport { politeFetch } from \"crawl-census-client\";\n\nconst r = await politeFetch(\"https://example.com/\", { agent: \"gptbot\" });\nif (r.skipped) console.log(r.verdict, r.reason);   // disallow | refuse | pay\nelse           process(await r.response.text());\n```\n\n```python\nfrom crawl_census import polite_fetch\n\nr = polite_fetch(\"https://example.com/\", agent=\"gptbot\")\nif r.skipped:\n    print(r.verdict, r.reason)\nelse:\n    process(r.body)\n```\n\nSkipping is returned, not raised. It is the normal outcome for a large share of the web, and a\ncrawl loop should be able to count skips without a try/except around every URL.\n\n## Split a queue before crawling it\n\nOne call per 1,000 domains instead of one per host:\n\n```js\nconst { crawl, skip, pay, unknown } = await partition(urls, { agent: \"gptbot\" });\n```\n\n```python\np = partition(urls, agent=\"gptbot\")\np.crawl, p.skip, p.pay, p.unknown\n```\n\n## Or just take the file\n\nFor a fetcher that only needs a deny list in memory, skip the per-domain calls entirely:\n\n```bash\ncurl https://crawlcensus.com/agents/gptbot/blocklist.txt   # one domain per line, commented header\n```\n\n```js\nconst sync = await syncBlocklist(\"gptbot\");   // full list once\nif (sync.blocked.has(host)) skip();\nsetInterval(() => sync.refresh(), 3600_000);  // then deltas only, a few hundred bytes\n```\n\n```python\nsync = BlocklistSync(\"gptbot\")\nif host in sync: skip()\nsync.refresh()          # {'added': 3, 'removed': 1, 'size': 3310, 'cursor': ...}\n```\n\nThe delta feed is `https://crawlcensus.com/agents/<agent>/changes.json?since=<unix>` and each\nresponse carries `next_since`, so a long-running crawler stays current on a few hundred bytes\nan hour instead of re-downloading the list.\n\nThat file covers **robots.txt only**. Edge refusal and HTTP 402 are per-request behaviours and\nstill need `preflight` or `politeFetch`.\n\n## What a crawl costs the census\n\nMeasured, not asserted. Twenty hosts fetched concurrently used to cost twenty preflight calls\ncarrying one domain each; the same host requested three times at once cost three, because the\ncache only helps after the first lookup resolves. The anonymous allowance is 240 calls an hour,\nso a crawler hit its ceiling at 240 hosts when one call covers twenty-five.\n\n`politeFetch` now shares work automatically: lookups issued in the same tick leave as one\nbatched call, and concurrent lookups for the same host await a single request.\n\n| pattern | before | now |\n|---|---|---|\n| 20 hosts, concurrent | 20 calls | 1 call of 20 |\n| 1 host, 3 URLs, concurrent | 3 calls | 1 call |\n| 60 hosts, concurrent | 60 calls | 3 calls (25 / 25 / 10) |\n| `partition` then fetch | 2 calls | 1 call |\n\n`batchSize` defaults to 25, the per-call cap without a key. Raise it with a Pro or Data key.\n`batchWaitMs` widens the coalescing window for concurrency that arrives in waves rather than\nall at once; the default of zero flushes on the next tick.\n\n## Paying, when an origin quotes a price\n\nA `pay` verdict carries the amount when the origin named one:\n\n```js\nconst r = await politeFetch(url, { agent: \"claudebot\" });\nif (r.verdict === \"pay\") console.log(r.price);   // \"USD 0.5\", or null if none was quoted\n```\n\nTwo things worth knowing. Most origins answering HTTP 402 name no amount at all, so `price`\nis usually null and the arrangement has to be made out of band. And pricing is per crawler:\nacross the measured corpus, 78 of 213 charging origins charge some agents and serve others\nfree, so ask with your own token rather than assuming a domain on the list will charge you.\n\n## Two kinds of unknown\n\n`partition` splits a work queue into `crawl`, `pay`, `skip`, `unknown` and `undecidable`.\n\nThe last two look alike and are not. `unknown` means the census has not measured that domain\nyet: submit it and the next pass gets a real verdict. `undecidable` means the site's robots.txt\ndisallows CrawlCensusBot, so this census will never measure it — retrying is guaranteed waste,\nand a loop that resubmits its unknowns each pass would resubmit those forever. The server marks\nthe difference with a `measurable` boolean; read that, never the `reason` prose.\n\n```js\nconst p = await partition(urls, { agent: \"gptbot\" });\nawait Promise.all(p.crawl.map(politeFetchOne));\nif (p.unmeasured.length) await submitUnmeasured(p, { agent: \"gptbot\" });\n// p.undecidable: read their robots.txt yourself. Asking us again cannot help.\n```\n\nSubmission is a separate call on purpose. A library that quietly POSTs during what reads as a\nlookup is a bad citizen, and you should choose when your queue positions are spent.\n\n## Skipping everything that will not serve you\n\nA deny list is the smaller half. Measured against the live census, a crawler that skips only\nrobots disallows still spends around 2,900 requests a pass on domains that permit it in\nrobots.txt and refuse it at the edge, or that answer HTTP 402 — for PerplexityBot that set is\nlarger than its deny list. Those fetches return nothing and cost a round trip each.\n\n```js\nconst skip = await syncSkipList(\"gptbot\");\nif (skip.has(host)) continue;        // disallowed, refused at the edge, or priced\nskip.why(host);                      // \"disallow\" | \"pay\" | \"refuse\" | null\nsetInterval(() => skip.refresh(), 3600_000);\n```\n\n| agent | deny list | also skippable | total |\n|---|---|---|---|\n| GPTBot | 3,542 | 2,944 | 6,486 |\n| ClaudeBot | 3,169 | 3,194 | 6,363 |\n| PerplexityBot | 1,128 | 3,658 | 4,786 |\n\nThe three sets are kept apart internally, so a change moves the one it belongs to. `why()`\nfollows the same precedence as preflight: a disallow outranks a price, because a price is not\npermission.\n\n## Keeping a deny list current\n\n`syncBlocklist` / `BlocklistSync` download the list once, then apply only what changed.\n\nThe list is served with the exact position in the change feed it was built at, in an\n`x-cursor` header and a `# cursor:` comment. The clients read it and resume from there, so\nthere is no gap between the snapshot and the first poll, and no reliance on your clock being\nin step with the server's. Polling by second cannot express a position inside a second, and a\ncrawl batch writes dozens of events into one, so a second-granularity resume can drop the\nremainder of it: measured live, resuming after the first of three same-second changes\nrecovered both siblings by cursor and neither by second.\n\n```js\nconst sync = await syncBlocklist(\"gptbot\");   // cursor comes from the list itself\nif (sync.blocked.has(host)) skip();\nsetInterval(() => sync.refresh(), 3600_000);  // a few hundred bytes per poll\n```\n\n`refresh()` applies only robots transitions to the list, because that is what the list is made\nof. Edge refusals, new prices and llms.txt changes come back in `other` for you to act on\nseparately — an earlier version deleted those domains from the deny list, so a crawler resumed\nfetching exactly what had just started refusing it.\n\n## Verdicts\n\nThe authoritative definition of each verdict — what it means, what it obliges a crawler to do,\nand whether asking again could change it — is published as data at\n[`/api/v1/verdicts`](https://crawlcensus.com/api/v1/verdicts). The list below is a summary; if\nthe two ever disagree, the endpoint is right and this file is stale.\n\n`politeFetch` skips `disallow`, `refuse` and `pay` by default, which is the endpoint's derived\n`do_not_fetch` set. The copy here is deliberate — a crawl loop should not need a network call to\ndecide — and a test compares the two so it cannot drift unnoticed.\n\n| Verdict | Meaning | Default behaviour |\n|---|---|---|\n| `allow` | robots.txt permits this agent, and a live request carrying its user agent was served | fetch |\n| `disallow` | robots.txt forbids this agent at the site root | skip |\n| `refuse` | robots.txt permits it; the edge refused it anyway. The allowance is not real | skip |\n| `pay` | the origin answered HTTP 402. It will serve this agent on commercial terms | skip |\n| `unknown` | not measured recently enough to answer | fetch |\n\n`onPay: \"fetch\"` (`on_pay=\"fetch\"`) overrides the paywall default. It is an explicit opt-in and\nis recorded on the result as `paidRouteOverridden` so it shows up in your logs.\n\n## It degrades, it does not fail\n\nIf the census is unreachable every verdict becomes `unknown` and your crawl proceeds as it\nnormally would. A third-party outage must never stop your pipeline. There is a live test for\nexactly this.\n\n## What we publish about your agent\n\n```js\nconst p = await agentProfile(\"claudebot\");\n// robots disallow rate, edge refusal rate, operator page, correction channel\n```\n\nIf a figure is wrong, the correction channel is in that response and on your\n[operator page](https://crawlcensus.com/operators). Registry facts are corrected without\nargument; disputed measurements are published alongside the dispute with the underlying scan\nrecords, rather than quietly amended.\n\n## Limits\n\n25 domains per preflight call anonymously, 200 with a Pro key, 1,000 with a Data key. Pass\n`apiKey`. Details at <https://crawlcensus.com/for-crawlers>.\n\n## Tests\n\n`node test.mjs` runs against the live census on purpose. The value of this client is whether\nits verdicts match reality, and a mocked test would assert only that the mock agrees with itself.\n\nMIT. Data is CC BY 4.0, attribute as \"Source: Crawl Census (crawlcensus.com)\".\n",
  "bytes": 11089,
  "sha": "e46933acbce8f83531daf932095faac25ed44f8bc775ff21d127a9e27882af29",
  "repo_slug": "taylorsmithgg/crawl-census-client",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_taylorsmithgg_crawl_census_280122b2/readme"
}