{ "markdown": "# ABS Data API — cartography and front door\n\nThe Australian Bureau of Statistics Data API is valuable and badly described.\nThis project measured what it actually serves — every dataflow, by retrieval —\nand exposes the result as a typed front door: an MCP server and an HTTP API\nthat only ever offer what exists.\n\n![The ABS Data API cartography: unmasking the 4.21% reality — a comic-style map of the findings and the front door](docs/cartography-comic.png)\n\n| | |\n|---|---|\n| Front door | |\n| Add to claude.ai / any MCP client as a connector | `https://abs-data-front-door.aicolab.workers.dev/mcp` (registry name `io.github.AI-CoLab/abs-data`) |\n| MCP (2026-07-28, stateless) — 4 fixed verbs + Code Mode `search`/`execute`, resources, prompts | `https://abs-data-front-door.aicolab.workers.dev/mcp` |\n| RPC door (Cap'n Web; what `@abs/sdk` speaks — HTTP batch or WebSocket) | `https://abs-data-front-door.aicolab.workers.dev/rpc` |\n| HTTP API | `https://abs-data-front-door.aicolab.workers.dev/api/tables` |\n| Scalar reference | |\n| Corrected OpenAPI | |\n| The menu as one document | |\n| Live browsable catalogue | |\n| Static snapshot (claude.ai artifact) | |\n\n## What was found\n\n- **621,536,714 series** exist across 1,227 dataflows. ABS's metadata implies\n 14,761,557,818 — a density of **4.21%**, overstated **23.8×**.\n- The per-dimension metadata is **exact**: every advertised option exists, and\n nothing real is unadvertised. The entire overstatement is combinatorial.\n- **81.0%** of series sit in flows whose sparsity is principled and verified in\n the keys — almost entirely hierarchical geography encoded as separate\n dimensions (`STATE` is a function of `REGION` in 664 flows). The remaining\n 19% is not reducible to a single-dimension dependency.\n- `detail=serieskeysonly` — the endpoint that should answer \"what exists\" — is\n broken (HTTP 200, malformed JSON). Eight of fifteen documented behaviours do\n not conform. See `reports/`.\n\n`DESIGN.md` is the full record: verified API findings, every decision and why,\nand the corrections made along the way.\n\n## Layout\n\n```\npackages/schema Drizzle schema (declared vs observed, never joined), shared CSV/period parsing\npackages/crawler structural crawl, observed probe, conformance suite, sizing\npackages/catalogue delta report, sparsity analysis, contract generator, OpenAPI emitter, artifact\npackages/contract the corrected contract: stable Zod verbs + generated tables/options (per crawl)\npackages/worker the front door: MCP door (SDK v2) + HTTP door (oRPC + Scalar) + RPC door (Cap'n Web) on one Worker\npackages/sdk @abs/sdk — typed TypeScript client over the RPC door (connect(), invalidSelectionFromError())\nreports/ delta.md/json, sparsity.md/json, openapi.json, abs-report-draft.md\ndata/ local SQLite catalogue (~400GB), raw archive, D1 import SQL — not committed\n```\n\n## Running it\n\n```sh\npnpm install\npnpm db:migrate # local SQLite at data/abs-catalogue.sqlite\n\npnpm crawl:structural # 7 bulk requests, ~1 minute\npnpm crawl:probe -- '--key-values=*' # every dataflow by retrieval; ~24h, 20GB heap for the giants\npnpm report:delta && pnpm report:sparsity\npnpm generate:contract # -> packages/contract/src/generated, data/d1/*.sql\npnpm --filter @abs/catalogue emit:openapi\n\ncd packages/worker\nnpx wrangler d1 execute abs-catalogue --local --file=../../data/d1/00-schema.sql # then each data file\nnpx wrangler dev\nABS_MCP_URL=http://localhost:8787/mcp npx tsx test/protocol.ts # ~50 checks, all doors\npnpm --filter @abs/sdk example # the SDK against production\nABS_URL=http://localhost:8787 npx tsx test/limits.ts # rate limits (spends a minute's budget)\n```\n\nRate limits, per client address: 600 requests/min overall; 120/min for data\npulls (each is a live ABS call); 20/min for Code Mode `execute`. Over the limit,\nHTTP doors answer 429 with `Retry-After`; an MCP tool call gets a tool error.\n\nUsing the SDK from TypeScript:\n\n```ts\nimport { connect, invalidSelectionFromError } from \"@abs/sdk\";\n\nconst abs = connect(); // https://abs-data-front-door.aicolab.workers.dev/rpc\nconst [hits, cpi] = await Promise.all([ // one HTTP batch\n abs.searchTables({ query: \"cpi rent\", limit: 3 }),\n abs.describeTable(\"CPI\"),\n]);\ntry {\n const data = await abs.getData({ table: \"CPI\", select: { REGION: \"Sydney\", INDEX: \"10001\" }, lastN: 4 });\n} catch (err) {\n const invalid = invalidSelectionFromError(err); // typed: reason, dimension, validOptions from observation\n}\n```\n\nThe probe is resumable: rerunning skips flows already recorded as complete. For\nthe largest census flows run with `NODE_OPTIONS=--max-old-space-size=20480`.\n\nGotcha: wrangler keys the *local* D1 state by `database_id`. Changing the id in\n`wrangler.toml` (e.g. after `d1 create`) gives `wrangler dev` a fresh, empty\nlocal database — re-run the local import.\n\n## Principles\n\n- **Observed-only.** Declared metadata steers the crawler and feeds the defect\n report; it never enters what a caller sees. Availability is empirical: observed\n options from the catalogue, exact combination checks live against ABS's\n `availableconstraint` (which agrees with the probe on every dimension of every\n flow).\n- **One contract, derived doors.** Every crawl regenerates one contract; the MCP\n verbs, the HTTP routes, the OpenAPI document and the docs all read or render\n it. Nothing hardcodes the surface, so nothing drifts.\n- **Only calls that return data.** A selection is verified before it is\n forwarded; an invalid one comes back with the observed valid options — as an\n MRTR `input_required` on MCP, a typed 422 on HTTP.\n", "bytes": 6151, "sha": "2aa6c5bbd7dd720427f2c9688da5de9c270b57e9c497d19f56324ccd2e2f42bb", "repo_slug": "ai-colab/abs-data-mcp", "fonte": "repo", "truncated": false, "api": "https://agentalog.com/api/listings/mcp_io_github_samjb123_abs_data_8f45aaab/readme" }