{
  "markdown": "<div align=\"center\">\n\n# TokenTrust\n\n<!-- mcp-name: io.github.RudrenduPaul/tokentrust -->\n\n[![CI](https://github.com/RudrenduPaul/TokenTrust-CLI/actions/workflows/ci.yml/badge.svg)](https://github.com/RudrenduPaul/TokenTrust-CLI/actions/workflows/ci.yml)\n[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](./LICENSE)\n[![npm](https://img.shields.io/npm/v/tokentrust-cli.svg)](https://www.npmjs.com/package/tokentrust-cli)\n[![PyPI](https://img.shields.io/pypi/v/tokentrust-cli.svg)](https://pypi.org/project/tokentrust-cli/)\n\n[Install](#install) • [What it measures](#what-it-measures) • [Commands](#commands) • [Agent-native / MCP](#agent-native--mcp) • [FAQ](#faq)\n\nVendor-neutral CLI that independently verifies the token and cost savings AI-coding-agent\ncontext-reduction proxies actually deliver, by running the proxy for real against a labeled\ntask corpus instead of trusting the maintainer's own number.\n\n<img src=\"docs/demo.gif\" alt=\"Terminal recording of npm install -g tokentrust-cli followed by tokentrust verify --proxy rtk, printing claimed vs. measured token and cost savings for rtk 0.43.0 across the bundled 23-task corpus\" width=\"640\">\n\n</div>\n\n## Install\n\nTokenTrust ships as two complementary, equally first-class distributions: an npm package for\nNode.js toolchains and a PyPI package for Python toolchains. Both install a `tokentrust` command\nwith the identical CLI surface, the same TT01-TT05 verification categories, and the same bundled\ntask corpus, so pick whichever matches your existing stack.\n\n**npm (Node.js):**\n\n```sh\nnpx tokentrust-cli verify --proxy rtk\n```\n\nNo clone, no local build. `npx` fetches the published package and runs it directly. To install\nit as a dependency instead: `npm install -g tokentrust-cli`.\n\n**pip (Python):**\n\n```sh\npip install tokentrust-cli\ntokentrust verify --proxy rtk\n```\n\nSee [python/README.md](./python/README.md) for the Python package's full documentation,\nincluding a note on the one real behavioral difference between the two: the npm package's\n`js-tiktoken` dependency bundles its tokenizer data for fully offline use, while the Python\npackage's `tiktoken` dependency fetches and caches that same public data on first use.\n\nReal output from that exact command, run against this repo's own bundled task corpus:\n\n```\n$ npx tokentrust-cli verify --proxy rtk\n\nTokenTrust v0.1 -- Token/Context-Reduction Claims Verification\nProxy: rtk 0.43.0 | Repo: TokenTrust | Task corpus: 23 labeled tasks\n\n[MEASURED] TT01 Compression Ratio\n  Claimed (rtk README): up to 70% context reduction\n  Measured (this repo, this corpus): 60.6% average reduction across 23 tasks\n  Range: 0.0% (\"verify-go-build-filter\") to 95.4% (\"verify-git-log-filter\")\n\n[MEASURED] TT02 Cost-Savings Delta\n  Baseline (uncompressed): $0.02 across 23 tasks @ claude-5-sonnet pricing\n  Compressed (rtk-proxied): $0.00 across 23 tasks\n  Actual savings: 77.0% ($0.01) -- vs. claimed 70% ceiling\n\n[FAIL]  TT03 Never-Worse Output Guard\n  2/23 tasks regressed in task-completion diff vs. uncompressed baseline\n\n[PASS]  TT05 Version-Drift Regression Check\n  No prior verified baseline for rtk on this repo -- this run establishes the first baseline.\n\nSummary: 77.0% measured cost savings (claimed: up to 70%) -- see full report\n```\n\nThat's a real run's output, not a hand-typed example -- `npx tokentrust-cli` invokes the exact\nsame `dist/cli.js` entry point (the `tokentrust` command name is unchanged), so it reproduces on\nyour machine with no clone required.\n\n## What it measures\n\n- **TT01: Compression Ratio.** Actual token reduction, measured with a local tokenizer\n  (`js-tiktoken`), against every task in the corpus.\n- **TT02: Cost-Savings Delta.** Dollar-cost savings computed from TT01's measured token delta at\n  published model pricing. Optional `--live` mode verifies the estimate against a real,\n  provider-billed sample (opt-in, your own API key, gated behind `--confirm-cost`, capped at 5\n  tasks by default).\n- **TT03: Never-Worse Output Guard.** Checks whether a proxy's compressed output dropped content\n  a task marks as required to survive compression.\n- **TT04: Cross-Tool Comparative Benchmark.** Pass `--proxy` more than once and TokenTrust runs\n  the identical task corpus through every named proxy side by side.\n- **TT05: Version-Drift Regression Detection.** Compares a run's measured savings against the\n  last-verified baseline for the same proxy/repo pair, so a silent regression across a version\n  bump (like `rtk#582`) gets caught automatically.\n\n## Commands\n\n```\ntokentrust verify --proxy <name> [options]\n```\n\n| Flag | Description |\n|---|---|\n| `--proxy <name>` | Proxy to verify. Repeatable, pass it more than once to run TT04's cross-tool comparison. Supported: `rtk`, `headroom`. Required. |\n| `--repo <path>` | Repo to measure against. Defaults to the current directory. |\n| `--tasks <file>` | Task corpus YAML file. Defaults to the bundled 23-task corpus. |\n| `--live` | Sample real, provider-billed tokens for the first proxy instead of estimating from pricing tables. Requires `--confirm-cost`. |\n| `--confirm-cost` | Confirms the estimated spend `--live` prints before any real API call is made. |\n| `--live-max-tasks <n>` | Max tasks sampled in `--live` mode. Defaults to 5. |\n| `--format <terminal\\|json>` | Report output format. Defaults to `terminal`. |\n| `-h`, `--help` | Show the help message and exit. |\n\nThis table (and the `tokentrust mcp` reference below) is verified against the actual `--help`\noutput of the published `tokentrust-cli` package, not an old copy. One known gap between\nthat live `--help` text and the table above is called out directly in the FAQ, instead of\nsilently repeating it.\n\n`--format json` gives every category's claimed-vs-measured numbers as structured data, so a\nscript or agent can pull the comparison straight out with `jq` instead of parsing terminal text:\n\n<img src=\"docs/demo-3-json-output.gif\" alt=\"Terminal recording of tokentrust verify --proxy rtk --format json piped through jq, printing each TT category's claimed vs. measured savings percentage as structured JSON\" width=\"640\">\n\nExit code is `0` when the run completes with no gated failure, non-zero otherwise. The bundled\nGitHub Action's `--fail-on-regression` maps that straight to a failed CI step, so a version-drift\nregression breaks the build instead of shipping silently.\n\nAdd it to CI with the bundled GitHub Action (`action/action.yml`) so verification reruns\nautomatically whenever a proxy's version bumps:\n\n```yaml\n- uses: RudrenduPaul/TokenTrust-CLI@main\n  with:\n    proxy: rtk\n    fail-on-regression: 'true'\n    cli-version: '<pin to the exact tokentrust-cli version you have verified against>'\n```\n\nPin `cli-version` explicitly rather than relying on the Action's default. The Action deliberately\ndoes not support `latest` for this input: pinning is a supply-chain safeguard, so a compromised\nnpm publish can't reach every workflow using this Action on its very next run without a\ndeliberate version bump on your part. The Action's own default for this input is an early\npre-TT04/TT05/MCP release -- omitting `cli-version` runs that old version, not the one this\nREADME documents.\n\n## Agent-native / MCP\n\nTokenTrust ships in the same dual CLI + MCP-server mode as Semgrep, Trivy, Snyk, and\nSonarQube: one binary, one underlying verification engine, and a second, thin front door for\nagents that speak [MCP (Model Context Protocol)](https://modelcontextprotocol.io) instead of a\nshell. `tokentrust mcp` starts an MCP server over stdio, exposing a single tool,\n`verify_proxy_savings`, that calls straight into the same `runVerify()` engine `tokentrust\nverify` uses -- no verification logic is duplicated, and the tool returns the exact structured\nJSON report `--format json` already produces.\n\n```sh\nnpx tokentrust-cli mcp\n```\n\n<img src=\"docs/usage.gif\" alt=\"Terminal recording of tokentrust mcp --help, showing the MCP server usage text and the verify_proxy_savings tool description\" width=\"640\">\n\n### Register it with an MCP client\n\nPoint the client's server config at this binary with the `mcp` argument. For Claude Code,\nClaude Desktop, or any other client that reads an `mcpServers` block:\n\n```json\n{\n  \"mcpServers\": {\n    \"tokentrust\": {\n      \"command\": \"npx\",\n      \"args\": [\"tokentrust-cli\", \"mcp\"]\n    }\n  }\n}\n```\n\n### The tool\n\n| Field | Description |\n|---|---|\n| `verify_proxy_savings` | Tool name. Mirrors `verify`'s flags one-for-one, minus `--format` -- an MCP call is always machine-facing, so the tool always returns the structured JSON report. |\n| `proxy` (required) | A single proxy name (`\"rtk\"`) or an array (`[\"rtk\", \"headroom\"]`) to run the TT04 cross-tool comparison in one call. Supported: `rtk`, `headroom`. |\n| `repo` | Same as `--repo`. Defaults to the MCP server process's current working directory. |\n| `tasks` | Same as `--tasks`. Defaults to the bundled task corpus. |\n| `live` / `confirmCost` | Same `--live`/`--confirm-cost` safety gate as the CLI: no live, provider-billed API call is made unless BOTH are explicitly `true` in the same call. Neither has a default of `true`. |\n| `liveMaxTasks` | Same as `--live-max-tasks`. Defaults to 5. |\n\nThis is the tool's real, unedited `tools/list` schema, captured from a running `tokentrust mcp`\nserver (`inputSchema` trimmed of per-field descriptions here for length; the live server returns\nthem in full):\n\n```json\n{\n  \"name\": \"verify_proxy_savings\",\n  \"title\": \"Verify proxy token/cost savings\",\n  \"inputSchema\": {\n    \"type\": \"object\",\n    \"properties\": {\n      \"proxy\": { \"anyOf\": [{ \"type\": \"string\", \"enum\": [\"rtk\", \"headroom\"] }, { \"type\": \"array\", \"items\": { \"type\": \"string\", \"enum\": [\"rtk\", \"headroom\"] }, \"minItems\": 1 }] },\n      \"repo\": { \"type\": \"string\" },\n      \"tasks\": { \"type\": \"string\" },\n      \"live\": { \"type\": \"boolean\" },\n      \"confirmCost\": { \"type\": \"boolean\" },\n      \"liveMaxTasks\": { \"type\": \"integer\", \"exclusiveMinimum\": 0 }\n    },\n    \"required\": [\"proxy\"]\n  }\n}\n```\n\nA real `tools/call` against this repo, `{\"name\": \"verify_proxy_savings\", \"arguments\": {\"proxy\":\n\"rtk\"}}`, returns the same shape as the CLI's `--format json` output (trimmed here; the live\ncall returns the full `records` array with TT01/TT02/TT05 entries):\n\n```json\n{\n  \"content\": [\n    {\n      \"type\": \"text\",\n      \"text\": \"{\\n  \\\"run_id\\\": \\\"tt_2026-07-18_f88644\\\",\\n  \\\"repo\\\": \\\"...\\\",\\n  \\\"task_corpus_size\\\": 23,\\n  \\\"proxies\\\": [\\\"rtk\\\"],\\n  \\\"records\\\": [ /* TT01, TT02, TT05 -- same shape as `verify --format json` */ ],\\n  \\\"tt03\\\": { \\\"rtk\\\": { \\\"pass\\\": false, \\\"regressed_count\\\": 2, \\\"task_corpus_size\\\": 23 } },\\n  \\\"tt05\\\": { \\\"rtk\\\": { \\\"pass\\\": true, \\\"message\\\": \\\"No regression vs. last-verified rtk 0.43.0 baseline (stored 2026-07-18).\\\", \\\"prior_run_id\\\": \\\"tt_2026-07-18_608b74\\\", \\\"degraded\\\": false } }\\n}\"\n    }\n  ],\n  \"isError\": false\n}\n```\n\n`isError` is `true` (with no report) whenever the underlying `runVerify()` call itself would\nhave exited non-zero on the CLI -- a missing proxy binary, an invalid task corpus, or the\n`--live` safety gate refusing an under-confirmed call. Progress output and the trace log\n`tokentrust verify` normally prints to stdout are rerouted to stderr in MCP mode, since stdout\nis the live JSON-RPC wire once a stdio transport is connected.\n\n## Proxy support\n\n| Proxy | Status |\n|---|---|\n| `rtk` | Fully supported: real subprocess-based verification (`rtk pipe --filter <name>` for stdin-shaped tasks, `rtk read -l aggressive <files>` for file-based tasks). |\n| `headroom` | Recognized (`--proxy headroom` is a valid flag value), not yet supported. headroom is an HTTP proxy server, not a one-shot compression CLI, so the current subprocess-based harness can't drive it. `tokentrust verify --proxy headroom` prints a message and skips it instead of failing silently. |\n\n## How it compares\n\n| | What it does | Ongoing / self-serve | Verifies a specific claim |\n|---|---|---|---|\n| **TokenTrust** | Runs a named proxy against a labeled task corpus, measures real compression, cost, and output-quality regression, prints claimed vs. measured | Yes, runs in your own CI, on your own repo, every time a proxy version bumps | Yes, that's the whole point |\n| [tokbench](https://github.com/Entelligentsia/tokbench) | Independent pilot benchmark of rtk, headroom, and lean-ctx on real agentic SDLC tasks, with raw transcripts and a pre-registered protocol | No, a single-repo pilot report, replication in progress | Yes, and rigorously: credit where it's due |\n| [Langfuse](https://github.com/langfuse/langfuse), Vantage, Finout, Amnic, Revenium | LLM/AI cost observability and FinOps. Track your actual API spend across models and providers, allocate it across teams | Yes, hosted or self-hosted, ongoing | No, these track what you spent; they don't check whether a specific proxy's specific savings claim holds up |\n\n[tokbench](https://github.com/Entelligentsia/tokbench) is the closest prior art and deserves real\ncredit. It's rigorous and disclosed, but its pilot scope is narrower than a first read suggests:\none repository, one task, N=1 per arm, replication runs in progress. Its own numbers on that\npilot are worth reading directly. Provider-billed input tokens against a 2.28M-token native\nbaseline came in at 2.89M for rtk (+27%) and 3.24M for headroom (+43%, despite headroom\ngenuinely compressing 342K tokens on the wire), because the agent's turn count grew even as the\nper-turn payload shrank. That's a real, independently useful data point, and it's exactly the\nkind of gap between \"compressed\" and \"cheaper\" TokenTrust exists to keep catching, continuously,\nin your own repo rather than a single published pilot.\n\n## Why this exists\n\nContext-reduction proxies (`rtk`, `headroom`, and others) publish compression and\ncost-savings numbers in their own READMEs. Those numbers come from the maintainer's own\nbenchmark, on the maintainer's own workload, with nobody outside the project checking the math.\nThat's not an accusation. It's just how every proxy in this space currently reports its own\nnumbers, and a maintainer benchmarking their own tool isn't running an adversarial test.\n\nThe gap shows up in the proxies' own issue trackers:\n\n- [`rtk#839`](https://github.com/rtk-ai/rtk/issues/839), an open, 5-repo, 2,100-measurement\n  empirical benchmark thread asking how rtk's actual savings compare to what it claims.\n- [`rtk#1935`](https://github.com/rtk-ai/rtk/issues/1935), \"rtk gain hallucinates massive\n  token usage and savings\" (open).\n- [`rtk#582`](https://github.com/rtk-ai/rtk/issues/582), \"RTK Hook Increases Claude Code Costs\n  by 18%,\" a cost regression a maintainer's own test suite didn't catch on its own. TT05 exists\n  specifically to catch this class of regression before a user does.\n\nTokenTrust doesn't compete with these proxies. It verifies them. It has no stake in whether a\nproxy's claimed number holds up, and every category run prints the claimed number right next to\nthe measured one, so the comparison is never hidden or averaged away.\n\nWe also found and fixed a bug in our own measurement: one fixture's baseline had accidentally\nbeen captured with `git log --oneline` instead of a true raw `git log`, which understated rtk's\nreal compression on that task by roughly 42 percentage points. Recapturing it honestly is why\n`verify-git-log-filter` now measures 95.4%, the highest reduction in the corpus, and a real one.\n[Commit e42246c](https://github.com/RudrenduPaul/TokenTrust-CLI/commit/e42246c) has the fix --\nno measurement number ships without a fixture-run behind it. That same commit expanded the\ncorpus from 15 tasks to the current 23, which is why the actual runtime output above says \"23\nlabeled tasks\" even though one leftover help string still mentions the old count (see the FAQ).\n\n## What is TokenTrust, and why does it exist\n\nTokenTrust is a command-line tool that measures whether an AI-coding-agent context-reduction\nproxy's advertised token and cost savings hold up against a real, labeled task corpus, run with a\nlocal tokenizer instead of a spreadsheet estimate. It exists because compression proxies\ncurrently self-report their own savings numbers, and there is no independent, repeatable,\nCI-native way to check one before adopting it. TokenTrust is not a proxy itself and does not\ncompress anything. It verifies proxies that do.\n\n## Real-world validation\n\nTokenTrust's own validation work has already fed back into a real, independently tracked GitHub\nissue: [`rtk-ai/rtk#1313`](https://github.com/rtk-ai/rtk/issues/1313) (filed by @ChrisEdwards,\nasking rtk for a lossless-only mode and an honest account of the silent failures truncation\ncauses in agent contexts) was originally verified as only partially addressed by rtk's existing\nmechanism, because TokenTrust's own fixtures didn't yet carry the quality markers needed to prove\nit either way. Extending three of TokenTrust's `pipe --filter` fixtures with real, verified\nquality markers closed that gap in TokenTrust's own instrumentation, not in rtk, and let the tool\nconfirm, against the real rtk 0.43.0 binary and not a claim, that rtk's existing never-worse guard\nmechanism already does what the issue asked for. The issue's verdict moved from partial to a\ngenuine, re-verified pass as a direct result. TokenTrust never touched rtk's own repository; it\ngot sharp enough to prove what was already true there.\n\n## Python package\n\n`pip install tokentrust-cli` installs the same `tokentrust` CLI as a genuine Python port, not a\nwrapper around the Node binary: real Python source under [python/src/tokentrust/](./python/src/tokentrust/),\nits own pytest suite, and the identical bundled 23-task corpus, copied verbatim into the wheel.\nBoth distributions run the same `cl100k_base` tokenizer encoding, verified to produce identical\ntoken counts on real sample text, and both are maintained together going forward, including\n`tokentrust mcp`: the Python port exposes the same `verify_proxy_savings` MCP tool, with a\nbyte-identical wire schema, as the npm package (see [python/README.md](./python/README.md)'s\n\"Agent-native / MCP\" section). See\n[python/README.md](./python/README.md) for install instructions, [python/docs/getting-started.md](./python/docs/getting-started.md)\nfor a walkthrough, and [python/docs/concepts.md](./python/docs/concepts.md) for the verification\nmethodology shared by both packages.\n\n## FAQ\n\n**What is TokenTrust, and how is it different from a context-reduction proxy like rtk or headroom?**\nTokenTrust is not a proxy itself and does not compress anything. It is a vendor-neutral\nverification layer: it runs a proxy like `rtk` or `headroom` as a real subprocess against a\nfixed, labeled 23-task corpus, measures the actual token and dollar savings with a local\ntokenizer, and prints that measured number next to the number the proxy's own README claims. The\ndifferentiator is independence: TokenTrust has no stake in whether a proxy's claimed number holds\nup, so it never averages the gap away.\n\n**Which platforms does TokenTrust run on, and are the npm and PyPI packages the same tool?**\nBoth are genuine, separately maintained ports of the same tool, not one wrapping the other.\n`npm install -g tokentrust-cli` (or `npx tokentrust-cli`) installs the Node.js build; `pip install\ntokentrust-cli` installs a real Python port under\n[python/src/tokentrust/](./python/src/tokentrust/), with its own pytest suite. Both expose the\nsame `tokentrust` command, the same TT01-TT05 categories, the same bundled task corpus, and the\nsame `cl100k_base` tokenizer encoding.\n\n**Does TokenTrust work with AI agents directly, not just from a shell?**\nYes. `tokentrust mcp` (or `npx tokentrust-cli mcp`) starts an MCP (Model Context Protocol) server\nover stdio that exposes one tool, `verify_proxy_savings`, backed by the same `runVerify()` engine\nthe CLI uses. Any MCP-compatible client, including Claude Code and Claude Desktop, can call that\ntool and get back the same structured JSON report `--format json` produces on the command line.\n\n**How does TokenTrust compare to tokbench, the other independent proxy benchmark?**\n[tokbench](https://github.com/Entelligentsia/tokbench) is real prior art and deserves credit: a\nrigorous, disclosed pilot with raw transcripts and a pre-registered protocol. Its current scope is\nnarrower than a first read suggests, one repository, one task, N=1 per arm, with replication in\nprogress. TokenTrust instead runs a 23-task corpus continuously, in your own CI, on your own repo,\nevery time a proxy version bumps, rather than as a single published pilot report.\n\n**The `tokentrust verify --help` text on the npm package says the default task corpus has 15 tasks. Which is correct, 15 or 23?**\n23 is correct. The bundled corpus was expanded from 15 to 23 tasks in\n[commit e42246c](https://github.com/RudrenduPaul/TokenTrust-CLI/commit/e42246c), and every real\nrun (both npm and PyPI) reports \"Task corpus: 23 labeled tasks\" at the top of its output, matching\nthe actual `fixtures/tasks.yml` file. The npm package's `--tasks` flag help text is a leftover\nstring from before that expansion and hasn't been updated to say 23; the PyPI package's help text\nalready says 23 correctly. This affects only what the `--help` text displays, not what tasks the\nCLI actually runs.\n\n**What if `pip install tokentrust-cli` fails on my Python version?**\nThe PyPI package declares `requires-python = \">=3.10\"` in its `pyproject.toml`, so `pip` will\nrefuse to install it on Python 3.9 or older. Upgrade to Python 3.10, 3.11, 3.12, or 3.13 (the\nversions the package is tested against), or use the npm package instead, which only requires\nNode.js 18 or newer.\n\n**Can I use TokenTrust commercially, and do I need to attribute it?**\nYes. TokenTrust is licensed Apache-2.0 (see [LICENSE](./LICENSE)), which permits commercial use,\nmodification, and distribution, including inside closed-source products, as long as you keep the\nlicense and copyright notice and note any changes you made to the source itself.\n\n**Does TokenTrust modify my code or my proxy's compressed output?**\nNo. It runs the proxy as a real subprocess against fixture tasks, captures the output, and\nmeasures it. Nothing in your repo or the proxy's configuration is changed.\n\n**Can TokenTrust verify a proxy's live, provider-billed cost instead of an estimate?**\nYes, with `--live --confirm-cost`, capped at 5 tasks by default via `--live-max-tasks`. It uses\nyour own API key and never runs a real charge without printing the estimated spend first.\n\n**Is TokenTrust importable as a library, or is it CLI-only?**\nCLI-only today. Neither the npm package nor the PyPI package ships a documented, importable\npublic API: the npm `package.json` lists a `main` entry that points at a file the published\npackage doesn't actually contain, and the PyPI package's top-level module only exports a version\nstring. Use the `tokentrust` command or the `verify_proxy_savings` MCP tool; there is no\nsupported way to `import`/`require` this package's verification logic directly yet.\n\n## Contributing\n\nSee [CONTRIBUTING.md](./CONTRIBUTING.md) for the project layout, how to add a verification\ncategory or fixture task, and the coverage bar every category change is held to, for both the\nnpm and PyPI packages.\n\n## License\n\nApache-2.0. See [LICENSE](./LICENSE).\n",
  "bytes": 23281,
  "sha": "ebecc0dd9b03cc2ef14fb24b0520fbe4c963057e46e1df3ad0185710569425c8",
  "repo_slug": "rudrendupaul/tokentrust",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_rudrendupaul_tokentrust_eb3dc421/readme"
}