{ "markdown": "# Inferrail\n\n\n\nKnow what your AI work costs.\n\nInferrail turns supported OpenAI chat-completion traffic into local,\nattributable economic receipts. Give related requests a customer-defined\n`work_id`, declare an outcome when your application knows one, and inspect the\nknown inference economics associated with that work without storing prompts,\nresponses, or tool payloads in Inferrail's own records.\n\n[![CI](https://github.com/domondi1/inferrail/actions/workflows/ci.yml/badge.svg)](https://github.com/domondi1/inferrail/actions/workflows/ci.yml)\n[![PyPI](https://img.shields.io/pypi/v/inferrail.svg)](https://pypi.org/project/inferrail/)\n[![License: Apache-2.0](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)\n\nFor the supported chat-completions surface, Inferrail records known cost when\nmeasured usage and a verified price are available. Otherwise it reports\n`unknown`, never a fabricated `$0`.\n\n## 30-second demo\n\n**Current main / upcoming Work Economics release.** Work Economics was added\nafter the current PyPI release. To try the current product before the next\nrelease, install from `main`:\n\n```bash\npip install \"inferrail @ git+https://github.com/domondi1/inferrail.git@main\"\ninferrail demo\n```\n\nThe demo needs no API key, no network call, and no provider billing. It runs\ncanned requests through Inferrail's real engine with made-up prices labeled\n`DEMO`, then shows receipts, attribution, work-level economics, and explicit\nunknown evidence.\n\n**Stable PyPI release.** `pip install inferrail` currently installs `0.1.2`.\nIt includes the gateway, receipts, reports, and `TaskTransaction`, but not the\nnew `work` commands. It remains the stable released install until the next\npackage publication.\n\n## What just happened?\n\n```text\nAI request\n -> InferenceReceipt\n -> caller-supplied attribution\n -> related requests share work_id\n -> customer-declared outcome\n -> Work Economics\n```\n\n- **Receipt:** one inference request produced payload-free economic evidence.\n- **Attribution:** the caller can attach identifiers such as customer,\n workflow, or project.\n- **Work:** several requests can share a `work_id` that your application\n defines.\n- **Outcome:** your application can append a declaration of what happened to\n that work.\n- **Work Economics:** Inferrail joins that declaration with matching receipts\n and reports known attributed inference economics for the work.\n\nYou decide what a unit of work means: a contract review, support resolution,\ncoding task, research run, or document-processing job. Inferrail associates\neconomic evidence with the identifier your application supplies; it does not\ninterpret the business meaning of that identifier or its outcome.\n\n### Request economics vs. Work Economics\n\n**Request economics:** what known inference economics belong to one request?\n\n**Work Economics:** what known inference economics belonged to the\ncustomer-defined unit of work those requests were performing?\n\nThis is not a full cost of work, COGS, margin, or business-value calculation.\n\n## Track a unit of work\n\nThe following uses real provider requests and requires `OPENAI_API_KEY`:\n\n```bash\nexport OPENAI_API_KEY=\n\ninferrail try \"Review this contract clause\" \\\n -a work_id=contract_review_42\n\ninferrail try \"Identify remaining risks\" \\\n -a work_id=contract_review_42\n\ninferrail work outcome contract_review_42 --status completed\ninferrail work contract_review_42\ninferrail work --all\n```\n\nFor a gateway client, the equivalent generic attribution header is:\n\n```text\nX-Inferrail-Attribute-Work-Id: contract_review_42\n```\n\nThe deterministic offline demo includes this synthetic example:\n\n```text\nwork-contract-1\n 2 inference receipts\n customer-declared outcome: resolved\n known attributed inference cost: $0.000483\n```\n\n`resolved` is only the demo application's own outcome meaning. Inferrail does\nnot treat any outcome status as universally successful.\n\nIf Inferrail cannot verify the price for an observed inference event, its cost\nremains `unknown` rather than being treated as zero. No receipt evidence is\nalso not the same thing as known zero cost.\n\n## First real request and reports\n\n`inferrail try` is the shortest route to one real receipt. It uses your\nexisting `OPENAI_API_KEY`; if it is not set, Inferrail prints what is required.\nIt prints the response, receipt, measured tokens, known cost or `unknown`, the\nlocal receipt path, and the next report command.\n\n```bash\ninferrail try \"Reply with one word: ready\" --customer acme\ninferrail report\ninferrail report --by customer\ninferrail report --by workflow\ninferrail report --by provider\n```\n\n## What a receipt contains\n\nOne payload-free JSON receipt per supported request:\n\n```json\n{\n \"receipt_id\": \"ir_1e6c916bac8940ca8a85\",\n \"provider\": \"openai\",\n \"model\": \"gpt-4o-mini\",\n \"prompt_tokens\": 842,\n \"completion_tokens\": 191,\n \"estimated_cost_usd\": \"0.000241\",\n \"attributes\": { \"customer\": \"acme\", \"workflow\": \"contract-review\" }\n}\n```\n\n(Trimmed — the full record also carries pricing provenance, status,\nroute, timestamp, latency, and retry count. See\n[Privacy boundary](#privacy-boundary) below for the complete shape.)\n\n## TaskTransaction: receipt-only task grouping\n\nOne task is rarely one call. Tag every request belonging to one unit of\nwork with the same attribution value, then ask Inferrail what the task\ncost:\n\n```bash\nexport OPENAI_API_KEY=\ninferrail try \"Reply with one word: ready\" -a task_id=bug_9281\ninferrail try \"Summarize: the retry patch is deployed\" -a task_id=bug_9281\ninferrail transaction bug_9281\n```\n\n```\nTask: bug_9281\nTransaction: tx_72fcfcca9ede9d2facc3\nStatus: success\n\nEVENT TYPE EVENT ID STATUS COST\ninference ir_f6fb6403d5324ea0acf9 success $0.000003\ninference ir_756cc072a27f42f4a2ea success $0.000007\n\nKnown total cost: $0.00001\n```\n\nThis TaskTransaction example uses real provider requests and a `task_id`.\nThe offline demo instead correlates requests with `work_id` and shows Work\nEconomics. Over HTTP, an\n`X-Inferrail-Attribute-Task-Id: bug_9281` header does the same thing;\n`inferrail.track_task(task_id=...)` (see [Attribute spend](#attribute-spend)\nbelow) attaches it automatically to every nested call in an agent run, no\nheader-threading required. See\n[docs/adr/0008](docs/adr/0008-task-transactions.md).\n\n## Use it as a gateway\n\nFor a long-running application, start the separate gateway process. The\ngateway process must have access to the provider credential through the\nconfigured environment variable; a key held only inside application memory is\nnot automatically transferred to the gateway.\n\n```bash\ninferrail serve --quickstart\n```\n\n```bash\ncurl http://127.0.0.1:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -H \"X-Inferrail-Attribute-Customer: acme\" \\\n -d '{\n \"model\": \"default\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Say hello in five words.\"}]\n }'\n```\n\nThe response is standard OpenAI `choices`/`usage` plus a non-standard\n`inferrail` block (route, provider, latency, retries) any OpenAI client\nalready ignores. `X-Inferrail-Attribute-*` headers are optional\nattribution — never forwarded upstream. See\n[examples/basic_chat_request.py](examples/basic_chat_request.py) for a\nminimal Python client, or point a supported OpenAI-compatible chat client at\n`http://127.0.0.1:8000/v1`. An OpenAI SDK client that does not set `base_url`\ncan use its existing `OPENAI_BASE_URL` environment mechanism instead.\n\nThe default receipt is one JSONL line per supported request in\n`./inferrail-receipts.jsonl`, relative to the gateway's working directory.\nTreat that file as machine/audit evidence; use `inferrail report` for the\nhuman aggregate, `inferrail transaction ` for receipt-only task\ngrouping, and `inferrail work ` for work-attributed inference\neconomics plus a customer-declared outcome.\n\n
\nFramework examples (LangChain, LlamaIndex, CrewAI)\n\n```python\n# LangChain\nfrom langchain_openai import ChatOpenAI\n\nllm = ChatOpenAI(\n base_url=\"http://127.0.0.1:8000/v1\",\n api_key=\"not-needed\", # or your INFERRAIL_GATEWAY_TOKEN if auth is enabled\n model=\"default\",\n)\n```\n\n```python\n# LlamaIndex\nfrom llama_index.llms.openai_like import OpenAILike\n\nllm = OpenAILike(\n model=\"default\",\n api_base=\"http://127.0.0.1:8000/v1\",\n api_key=\"not-needed\",\n is_chat_model=True,\n context_window=8192,\n)\n```\n\n```python\n# CrewAI\nfrom crewai import LLM\n\nllm = LLM(\n model=\"openai/default\", # \"openai/\" prefix required by CrewAI\n base_url=\"http://127.0.0.1:8000/v1\",\n api_key=\"not-needed\",\n)\n```\n\n
\n\n`\"model\"` normally selects a named route from `inferrail.yaml` (e.g.\n`\"default\"`), which maps to a provider + underlying model. If\n`default_provider` is set in your config, a `model` that matches no route\nis instead forwarded to that provider unchanged — so `\"model\":\n\"gpt-5.6-sol\"` works with no route pre-registered for it. Named routes\nalways take priority. This passthrough is on by default for the\nzero-config quickstart path, off by default otherwise. Full design:\n[docs/adr/0007](docs/adr/0007-model-passthrough-routing.md).\n\n## Attribute spend\n\nThree ways to attach business context to a request, all landing in the\nsame `attributes: dict[str, str]` on its receipt:\n\n- **HTTP header** (gateway): `X-Inferrail-Attribute-: `, e.g.\n `X-Inferrail-Attribute-Task-Id: bug_9281`.\n- **CLI flag** (`inferrail try`): `--customer`/`--workflow` shorthand, or\n generic `-a =` for anything else, including `task_id`.\n- **Ambient, for nested agent calls**: `inferrail.track_task` attaches\n `X-Inferrail-Attribute-Task-Id` to every outgoing request for the\n duration of a `with` block or decorated function — no threading a\n `task_id` parameter through nested function signatures by hand.\n\n```python\nimport inferrail\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://127.0.0.1:8000/v1\",\n api_key=\"not-needed\",\n # also accepted by LangChain's ChatOpenAI, CrewAI's LLM, etc. via\n # their own http_client= argument\n # base_url must match the client's own base_url above — the header is\n # only ever attached to requests going to that destination.\n http_client=inferrail.attributed_http_client(base_url=\"http://127.0.0.1:8000/v1\"),\n)\n\n@inferrail.track_task(task_id=\"bug_9281\")\ndef fix_bug():\n client.chat.completions.create(...) # tagged automatically\n run_subagent() # nested calls too — no task_id parameter needed\n```\n\n`with inferrail.track_task(task_id=\"...\"):` works the same way. Sync and\nasync are both supported (`attributed_async_http_client(base_url=...)` for\n`AsyncOpenAI`/async frameworks); concurrent tasks never cross-contaminate.\nThis is a small client-side convenience over the HTTP header above — no\ngateway or schema change, `task_id` only, no public API stability\ncommitment yet. See\n[docs/adr/0009](docs/adr/0009-ambient-task-tracking.md).\n\nOnce tagged, `inferrail report` shows the all-up aggregate, while\n`inferrail report --by `\naggregates receipts by any of these dimensions —\n`customer`, `workflow`, `task_id`, or anything else you've attached.\n\n## Referral early access\n\nReferral access is opening soon. Planned early-access rewards are based on\nverified routed usage, not signup:\n\n```text\n1 verified referral\n→ +90 days of cost history for both sides\n\n3 verified referrals\n→ Pro for one year + unlimited seats\n\n10 verified referrals\n→ Founding Operator\n→ permanent Pro\n→ logo on the site\n→ roadmap vote\n→ private channel\n\n25 verified referrals\n→ Inferrail free for life\n→ 20% recurring on additional teams referred\n```\n\nProgram terms will be published when referral access opens.\n\nSee the current program presentation at [tryinferrail.com](https://tryinferrail.com).\n\n## How it works\n\n`InferenceEngine` normalizes the request, resolves `model` to a route in\n`inferrail.yaml` (a pure config lookup — no cost/latency-aware\nselection in v0.1), calls the one provider adapter in this version\n(`OpenAIProvider`, generic over `base_url` — OpenAI itself, Azure\nOpenAI's compatible surface, vLLM, llama.cpp-server, or anything else\nspeaking the same wire format), and emits a telemetry event and a\nreceipt for every supported request, success or failure. Full lifecycle, package\nlayout, and the streaming/retry boundaries:\n[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).\n\n## Privacy boundary\n\nInferrail's own local receipt, telemetry, and outcome records contain\neconomic metadata and caller-supplied identifiers, not persisted prompts,\nresponses, tool payloads, or free-form business outcome payloads.\nStructurally, the receipt and telemetry schemas have no field capable of\nholding message content, and\n`test_inference_receipt_has_no_payload_fields` enforces it. This is a\nclaim about **Inferrail's own local records**, not about the request path as\na whole — your configured provider still receives the real prompt either\nway; Inferrail is a pass-through gateway to it, not a privacy boundary\nagainst the provider.\n\nInferrail currently measures supported OpenAI chat-completions traffic. It is\nnot a background monitor: it records while requests pass through the running\nprocess and serves nothing when that process is stopped. It does not enforce\nbudgets or control provider spend.\n\n`inferrail try` says this in its own output too, not just in the schema:\n\n```\n Prompt stored no\n Response stored no\n```\n\nThe full receipt shape, all fields:\n\n```json\n{\n \"receipt_id\": \"ir_1e6c916bac8940ca8a85\",\n \"route\": \"default\",\n \"provider\": \"openai\",\n \"model\": \"gpt-4o-mini\",\n \"status\": \"success\",\n \"prompt_tokens\": 842,\n \"completion_tokens\": 191,\n \"pricing\": {\n \"input_usd_per_million\": \"0.15\",\n \"output_usd_per_million\": \"0.60\",\n \"source\": \"https://developers.openai.com/api/docs/pricing\",\n \"verified_date\": \"2026-08-16\"\n },\n \"estimated_cost_usd\": \"0.000241\",\n \"attributes\": { \"customer\": \"acme\", \"workflow\": \"contract-review\" },\n \"total_latency_ms\": 15.96,\n \"retry_count\": 0\n}\n```\n\nIf Inferrail can't verify a price for the (provider, model) pair,\n`pricing` and `estimated_cost_usd` are `null` — never a guessed or\nfabricated cost. You can check the no-payload claim yourself against a\nrunning gateway, not just take it on faith:\n[docs/PRODUCT.md's verification walkthrough](docs/PRODUCT.md#verifying-privacy-claims-yourself).\nDesign rationale:\n[docs/adr/0005](docs/adr/0005-privacy-preserving-economic-receipts.md).\n\n## MCP\n\n```bash\npip install \"inferrail[mcp]\"\n```\n\nAn MCP server (`inferrail-mcp`), published on the MCP registry as\n[`io.github.domondi1/inferrail`](https://registry.modelcontextprotocol.io),\nexposes Inferrail's local receipt ledger to any MCP-aware agent (Claude\nCode, Claude Desktop, Cursor, ...) as two **read-only** tools — neither\nexecutes inference nor spends provider budget:\n\n| Tool | What it does |\n|---|---|\n| `get_spend` | Aggregates local receipts by provider/model/route/attribute (including `task_id`), optional time window |\n| `get_health` | Checks gateway reachability + most recent local receipt |\n\n```json\n{\n \"mcpServers\": {\n \"inferrail\": { \"command\": \"inferrail-mcp\" }\n }\n}\n```\n\nClaude Code: `claude mcp add inferrail -- inferrail-mcp`. Full contract:\n[inferrail-mcp/README.md](inferrail-mcp/README.md).\n\n## Supported today\n\n- `POST /v1/chat/completions`: streaming (`stream: true`, real SSE\n passthrough) and tool/function calling, single string message content,\n no `n != 1`\n- `GET /health`\n- One provider adapter, generic over any OpenAI-compatible HTTP endpoint\n- Named-route + optional passthrough model routing (above)\n- Per-route retry with backoff on transient provider errors\n- Local structured telemetry and payload-free cost receipts for supported\n requests, plus `inferrail report`, grouped reports, and\n `inferrail transaction `\n- Customer-defined `work_id` attribution, append-only outcome declarations,\n and derived Work Economics via `inferrail work outcome`, `inferrail work\n `, and `inferrail work --all`\n- CLI: `inferrail demo`, `try`, `serve` (`--quickstart`), `config check`,\n `report`, `transaction`, `work`\n\n## Not yet\n\nHonest edges, not silent gaps — full list in\n[docs/PRODUCT.md](docs/PRODUCT.md):\n\n- Cost- or latency-aware routing, or automatic failover to a different\n provider/model on error — routing is a static config lookup\n- Budgets, spend limits, or blocking a request based on cost\n- Any provider whose wire protocol isn't OpenAI-compatible (native\n Anthropic, Gemini, Bedrock, ...)\n- The full OpenAI API surface — only `/v1/chat/completions` and\n `/health` exist; no embeddings, assistants, batch, images, or audio\n- Multi-user auth or role-based access control —\n `INFERRAIL_GATEWAY_TOKEN` is one shared secret, not a user system\n- Any hosted or cloud-operated component\n- Non-LLM economic events (browser, search, compute/sandbox, MCP tool\n cost) in a `TaskTransaction` — its only event type today is `inference`\n- Outcome or business-value linkage (success signal, revenue, margin) on\n a `TaskTransaction` — it aggregates cost only\n\n## Deployment boundary\n\n**Single node.** The receipt ledger is a local append-only JSONL file, so\nevery process that should appear in one report must write to one file on\none filesystem.\n\n- Concurrent writers to the same file are safe: each receipt is written\n as a single atomic `O_APPEND` write, so threads *and* multiple\n processes on the same host can share one ledger without interleaving or\n losing records.\n- Not supported: several hosts writing to one ledger, aggregating ledgers\n across machines, or anything resembling a shared/hosted control plane.\n Running Inferrail on N hosts gives you N separate ledgers, and nothing\n in the product merges them.\n- `inferrail report` and `inferrail transaction` read the whole file into\n memory. That is fine for the millions-of-bytes range a developer\n preview produces; it is not a query engine, and there is no retention,\n rotation, or compaction. Rotate the file yourself if it grows.\n\nAnything beyond one host is out of scope for v0.x — see\n[docs/PRODUCT.md](docs/PRODUCT.md).\n\n## Paid capabilities (hosted, separate from the gateway)\n\n**Inferrail Work Economics** is Inferrail's first hosted, paid capability:\ngiven caller-declared economic events for a unit of AI work, it returns a\nnormalized cost receipt — known cost, a breakdown by resource class and\nsupplier, and unit economics for the work — paid for over the\n[x402](https://www.x402.org/) protocol by any agent with its own wallet —\nno Inferrail account required. **Base Sepolia testnet only right now**, not\nmainnet, not real money.\n\nThis is unrelated code, in `hosted/`, not part of the `inferrail` package —\nrunning the gateway above never requires it and never talks to it.\n\n- Canonical endpoint: `https://work.tryinferrail.com` ([manifest](https://work.tryinferrail.com/manifest))\n- Human-readable overview: [tryinferrail.com/work-economics](https://tryinferrail.com/work-economics/)\n- Full contract: [docs/capabilities/work-economics.md](docs/capabilities/work-economics.md)\n- Standalone buyer example: [`examples/work_economics_purchase.py`](examples/work_economics_purchase.py)\n\n## Configuration\n\nFor a real deployment instead of quickstart defaults:\n\n```bash\ncp inferrail.example.yaml inferrail.yaml\ncp .env.example .env # then add a real OPENAI_API_KEY\ninferrail config check # validate without starting a server\ninferrail serve\n```\n\n`inferrail.yaml` only ever holds the *name* of an environment variable\nfor a secret, never the secret itself. Full shape (providers, routes,\ntelemetry, receipts, pricing overrides):\n[inferrail.example.yaml](inferrail.example.yaml).\n\nBy default the gateway binds to `127.0.0.1:8000` with no auth. Set\n`INFERRAIL_GATEWAY_TOKEN` to require callers to send `Authorization:\nBearer ` — see [SECURITY.md](SECURITY.md).\n\n## Documentation\n\n- [docs/PRODUCT.md](docs/PRODUCT.md) — exact current scope\n- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — package layout, request\n lifecycle\n- [docs/adr/](docs/adr/) — why specific structural decisions were made\n- [openapi.json](openapi.json) / [config.schema.json](config.schema.json)\n / [llms.txt](llms.txt) — machine-readable references for tooling and\n agents\n- [SECURITY.md](SECURITY.md)\n\n## Development\n\n```bash\ngit clone https://github.com/domondi1/inferrail.git && cd inferrail\npip install -e \".[dev,mcp]\"\nruff check . && mypy && pytest\n```\n\n`pytest` needs no API key or network access — see\n[CONTRIBUTING.md](CONTRIBUTING.md).\n\n## License\n\nApache License 2.0 — see [LICENSE](LICENSE).\n", "bytes": 20693, "sha": "a1f4077e169645115c96156042207417b845ef7fc5561dc7ee59a5686fb21f99", "repo_slug": "domondi1/inferrail", "fonte": "repo", "truncated": false, "api": "https://agentalog.com/api/listings/mcp_io_github_domondi1_inferrail_9cee2f28/readme" }