{ "markdown": "
\n \"TokenMizer\"/\n\n

TokenMizer

\n\n

Your AI forgets why. TokenMizer remembers.

\n\n

\n An OpenAI-compatible proxy that builds a knowledge graph of your\n session — decisions, files, errors, goals — and replays it when the
\n context window runs out. Not a summary: a queryable graph that knows\n \"we switched from MongoDB to PostgreSQL, and here is why.\"\n

\n\n

\n One line to adopt · works with Claude, GPT, Gemini, Grok, DeepSeek, Mistral, Cohere, Ollama · MIT\n

\n\n

\n \"PyPI\"/\n \"Downloads\"/\n \"CI\"/\n \"MCP\n \n \"Stars\"/\n \"Glama\n \"Sponsor\"/\n

\n\n

\n Quick start ·\n Claude Code & MCP ·\n Architecture ·\n Benchmarks ·\n Configuration ·\n API & CLI ·\n Contributing\n

\n\n \"TokenMizer\n
\n Real run: 25-node graph, checkpoint ckpt_21a0959c3ddf, 233-token resume. Regenerate with python scripts/gen_demo_gif.py.\n
\n\n---\n\n## The problem\n\nEvery AI session has a context limit. When you hit it, the model forgets\nevery decision and every rationale built over hours of work, and you\nspend the first ten minutes of the next session re-explaining the\nproject.\n\nSummarising the history does not fix this. A summary tells you *what*\nwas decided; it loses *why*, and it loses what was rejected — so the\nmodel happily re-proposes the thing you moved off three sessions ago.\n\n## How it works\n\nTokenMizer is a local proxy between your app and any LLM. Every request\npasses through a pipeline that builds a live knowledge graph, compresses\ninputs, caches responses, and checkpoints before the context runs out.\n\n```mermaid\nflowchart LR\n App[\"Your app
OpenAI-compatible client\"]\n subgraph TM[\"TokenMizer :8000\"]\n direction TB\n L0[\"L0 File intelligence\"]\n L1[\"L1 Prompt compression\"]\n L2[\"L2 Terse-output injection\"]\n L4[\"L4 Graph memory
extract → window → inject\"]\n L3[\"L3 Semantic cache\"]\n L5[\"L5 Provider prompt cache\"]\n L0 --> L1 --> L2 --> L4 --> L3 --> L5\n end\n LLM[\"Claude · GPT · Gemini
Grok · DeepSeek · Ollama\"]\n DB[(\"SQLite
graph · checkpoints · ownership\")]\n\n App -->|\"POST /v1/chat/completions\"| TM\n TM --> LLM\n LLM -.->|response| TM\n TM -.->|\"response + savings\"| App\n L4 <-->|\"per-row, locked\"| DB\n```\n\nThe graph is not a summary. It is typed nodes and edges — decisions,\ntasks, files, errors, goals — with a lifecycle, so a decision that gets\nreplaced is marked superseded rather than deleted. The resume block is a\nfiltered projection of it: active decisions, open work, unresolved\nerrors, in a few hundred tokens.\n\n→ [**Architecture**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/architecture.md) — the request sequence, the\ndata model, and the decision lifecycle.\n\n## Quick start\n\n```bash\npip install \"tokenmizer[anthropic,cache]\"\nexport TOKENMIZER_ANTHROPIC_API_KEY=sk-ant-...\ntokenmizer serve\n```\n\nThen change one line in your client:\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n api_key=\"your-key\",\n base_url=\"http://localhost:8000/v1\", # ← only this changes\n)\n\nresp = client.chat.completions.create(\n model=\"claude-sonnet-4-6\",\n messages=[{\"role\": \"user\", \"content\": \"Continue where we left off\"}],\n extra_body={\"session_id\": \"my-project\"}, # ← optional, enables memory\n)\n```\n\nEverything else is unchanged: same request shape, same response shape,\nplus a `tokenmizer` block reporting what was saved.\n\n
\nWindows, Ollama, Docker, and the full step-by-step\n\n**Windows (PowerShell)**\n\n```powershell\n$env:TOKENMIZER_ANTHROPIC_API_KEY = \"sk-ant-...\" # this session\nsetx TOKENMIZER_ANTHROPIC_API_KEY \"sk-ant-...\" # persistent\n```\n\n**No API key?** Ollama runs locally and free:\n\n```bash\nollama pull llama3\npip install tokenmizer\n# then set `provider: ollama` in tokenmizer.yaml\n```\n\n**Docker**\n\n```bash\ndocker compose up -d\n```\n\nFull installation notes, every provider's environment variable, and the\nconfiguration reference are in\n[**docs/configuration.md**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/configuration.md) and\n[**docs/deployment.md**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/deployment.md).\n
\n\n## Use it from your tools\n\nThree ways in, depending on where you work. All three talk to the same\ngraph, so a session checkpointed from Claude Code resumes in the CLI.\n\n### Claude Code — plugin\n\n```\n/plugin marketplace add Shweta-Mishra-ai/tokenmizer\n/plugin install tokenmizer@Shweta-Mishra-ai/tokenmizer\n```\n\nThen, in any session:\n\n```\n/tokenmizer:checkpoint my-project save the session to graph memory\n/tokenmizer:resume my-project load it back (~300 tokens)\n/tokenmizer:analyze data/sales.csv digest a large file\n/tokenmizer:stats token savings report\n```\n\n### Claude Desktop, Cursor, VS Code, Zed — MCP server\n\n\n\n```json\n{\n \"mcpServers\": {\n \"tokenmizer\": {\n \"command\": \"tokenmizer-mcp\",\n \"env\": { \"TOKENMIZER_URL\": \"http://localhost:8000\" }\n }\n }\n}\n```\n\n| Client | Where that goes |\n|---|---|\n| Claude Desktop (macOS) | `~/Library/Application Support/Claude/claude_desktop_config.json` |\n| Claude Desktop (Windows) | `%APPDATA%\\Claude\\claude_desktop_config.json` |\n| Claude Code | `.mcp.json` in the project, or `~/.claude/settings.json` |\n| Cursor | Settings → MCP → Add server, same JSON |\n| VS Code / Zed | their MCP settings, same `command` and `env` |\n| Codex CLI | `~/.codex/config.toml` — TOML, see [docs/api.md](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/api.md) |\n\nRestart the client afterwards. Keep `tokenmizer serve` running for the\ncheckpoint, resume, stats and reasoning tools; file analysis works\nwithout it. If `tokenmizer-mcp` is not on your PATH, use\n`\"command\": \"python\", \"args\": [\"-m\", \"tokenmizer.mcp.server\"]`.\n\n**Six tools:** `checkpoint_session`, `resume_session`, `get_graph_stats`,\n`get_savings_stats`, `analyze_file`, and `why_decision` — ask your agent\n*\"why did we pick X?\"* and it walks the supersession chain with the\nreason and evidence for each hop.\n\n### Anything else — the proxy\n\nAny OpenAI-compatible client works by pointing `base_url` at\n`http://localhost:8000/v1`, as in the quick start above. That covers\nContinue.dev, Aider, LangChain, LlamaIndex, the OpenAI SDKs in every\nlanguage, and `curl`.\n\n→ [**API & CLI reference**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/api.md) — every endpoint, every command,\nevery MCP tool.\n\n## What a resume looks like\n\n```\nGoal: Build FastAPI auth service with JWT + PostgreSQL\nDone: Project setup | User model | Login endpoint | Fix 422 | 18 tests passing\nIn progress: Refresh token rotation\nDecided: PostgreSQL (concurrent writes) | bcrypt | Redis for refresh tokens\nChanged: ~~React~~ → Next.js (better SEO)\nFiles: api/auth.py, api/models.py, config.py\nContinue: Implement token refresh endpoint\n```\n\nA few hundred tokens in place of the whole conversation. The `Changed:`\nline is the part a summary loses — and asking\n`GET /api/graph/{session}/why?q=react` replays the full chain with the\ntrigger, the reason and the evidence for each hop.\n\n## Measured\n\n`python -m benchmarks.eval` scores extraction against a labelled corpus\nof 14 sessions, 6 of them real transcripts:\n\n| Category | Precision | Recall | F1 |\n|---|---|---|---|\n| Files | 98% | 100% | **99%** |\n| Pending tasks | 100% | 90% | **95%** |\n| Errors | 96% | 96% | **96%** |\n| Decisions | 90% | 95% | **92%** |\n| Completed tasks | 92% | 90% | **91%** |\n| | | **macro F1** | **95%** |\n\n**Precision is reported, not just recall.** An extractor that emits the\nwhole transcript as one node scores 100% recall, which is why\nrecall-only extraction numbers should be distrusted — including our own\nearlier ones.\n\nScored separately by origin, because hand-written fixtures are easier\nthan real transcripts and a single headline hides that: **synthetic 96%,\nreal 90%.** Treat 90% as the number that describes real sessions. n=14\nis a small sample and the same person wrote every label.\n\n**Independently verified against 7 other methods.** A separate\n100-session benchmark ([tokenmizer-research](https://github.com/Shweta-Mishra-ai/tokenmizer-research),\na different corpus and scorer than the numbers above) ties TokenMizer\n0.5.4 for first place at **60% macro F1** — level with Mem0-style (60%)\nand Graphiti-style (59%), ahead of GraphRAG-style (44%), MemGPT-style\n(35%), and every naive baseline (≤20%). Decisions and errors are still\nits weakest categories relative to the two methods it ties overall —\n59%/44% F1 against 65%/66% for Graphiti/Mem0-style, up from 50%/36% in\n0.5.3 after a fix (see [CHANGELOG](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/CHANGELOG.md))\ntargeted at the specific gaps that benchmark found.\n\n→ [**Benchmarks**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/benchmarks.md) — memory quality against a\nplain-summary baseline, storage, and how to score your own sessions.\n\n## Why TokenMizer and not X?\n\n**Why not just use Git history?**\nGit stores *what changed*, not *why you decided to change it*. You can't ask Git \"what did we decide about auth?\" or \"why did we switch from MySQL to PostgreSQL?\" TokenMizer stores decisions with trigger, reason, and evidence — not diffs.\n\n**Why not RAG (retrieval-augmented generation)?**\nRAG retrieves *relevant chunks* — it doesn't model *decision state*. If you switched from bcrypt to Argon2 mid-session, RAG might retrieve both and confuse the model about which is current. TokenMizer tracks decision supersession explicitly: the old decision is marked `SUPERSEDED`, the new one `ACTIVE`, and the resume context only includes current state.\n\n**Why not a plain summary at the start of each session?**\nSummaries lose structure. You can't query \"all superseded decisions\" or \"what triggered the auth change\" from a blob of text. Our benchmark shows graph memory preserves **89%** of labelled information against **79%** for a summary baseline — +10 points — and unlike a summary, the graph is queryable, editable, and grows incrementally instead of being re-summarized every turn. See [Benchmarks](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/benchmarks.md#memory-quality--graph-vs-a-plain-summary).\n\n**Why not Mem0 or Zep?**\nMem0 and Zep store *facts* (\"user prefers Python\"). TokenMizer stores *decisions with rationale* — the full causal chain: what was decided, what replaced it, why, what evidence triggered the change. If you need \"remember my name across sessions,\" use Mem0. If you need \"remember that we switched from PostgreSQL to SQLite because of cost, and here's the evidence,\" use TokenMizer.\n\n**Why not just a longer context window?**\nLonger context means higher cost, slower inference, and attention dilution on long histories. TokenMizer compresses a session into a resume block averaging **161 tokens** (measured, n=3 — see [Benchmarks](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/benchmarks.md)) by extracting what actually matters, not by summarizing.\n\n## What is not implemented\n\nTwo settings are accepted by the config and do nothing. They are listed\nhere rather than left to be discovered:\n\n| Setting | Status |\n|---|---|\n| `routing.*` | No implementation. `savings.routing` is always `0`. Enabling it logs a warning and changes nothing. |\n| `state_backend: redis` | Accepted and unused. `tokenmizer/state/backend.py` has no callers; all durable state is SQLite. |\n| `tools` / `tool_choice` (function calling) | Accepted (the request body isn't validated against a strict schema, so a standard OpenAI client sending them never gets a 422) but not forwarded to any provider. The model responds with no knowledge of the tools it was given. Logs a warning server-side per request. |\n\n## Documentation\n\n| | |\n|---|---|\n| [**Architecture**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/architecture.md) | Request pipeline, graph data model, decision lifecycle, file intelligence |\n| [**Configuration**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/configuration.md) | Every setting, environment variables, precedence, providers |\n| [**API & CLI**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/api.md) | Endpoints, commands, MCP tools, Claude Code integration |\n| [**Deployment**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/deployment.md) | Docker, multiple workers, durability, session isolation, security |\n| [**Benchmarks**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/benchmarks.md) | Extraction quality, memory quality, storage, running your own |\n| [**Comparisons**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/docs/comparisons.md) | Mem0, Zep, longer context windows, running alongside other token tools, and the roadmap |\n| [**Contributing**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/CONTRIBUTING.md) | Setup, layer rules, and how to improve extraction |\n| [**Testing**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/TESTING.md) | How to run the suite, the coverage floor, and known limits of the local audit scripts |\n| [**Changelog**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/CHANGELOG.md) · [**Security**](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/SECURITY.md) | Release history and how to report a vulnerability |\n\n## Contributing\n\n```bash\ngit clone https://github.com/Shweta-Mishra-ai/tokenmizer\ncd tokenmizer\npip install -e \".[dev]\"\npytest tests/ -q && ruff check tokenmizer/ # 664 tests, must stay green\n```\n\n**The most valuable contribution is a session where extraction got it\nwrong.** The eval corpus is 14 sessions and the same person wrote every\nlabel in it — that is the honest ceiling on what the numbers above can\ntell you about *your* workload, and the only way past it is transcripts\nnobody here wrote. Label a few of your own in the format documented in\n[`benchmarks/eval/corpus.py`](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/benchmarks/eval/corpus.py) and open a PR,\nor [open an issue](https://github.com/Shweta-Mishra-ai/tokenmizer/issues)\nwith the turn that was missed. Redact freely — the shape of the prose is\nwhat matters, not its content.\n\n[CONTRIBUTING.md](https://github.com/Shweta-Mishra-ai/tokenmizer/blob/main/CONTRIBUTING.md) covers setup, the layer rules, and how\nto run the eval harness.\n\n### Contributors\n\nThanks to everyone who has sent a fix upstream:\n\n- [**@0xfroOty**](https://github.com/0xfroOty) — negated-decision handling in the decision tracker ([#22](https://github.com/Shweta-Mishra-ai/tokenmizer/pull/22)), `OutputTrimmer` level alignment ([#25](https://github.com/Shweta-Mishra-ai/tokenmizer/pull/25)), streaming cache-hit analytics ([#31](https://github.com/Shweta-Mishra-ai/tokenmizer/pull/31))\n- [**@pollychen-lab**](https://github.com/pollychen-lab) — graph node IDs derived from stored (truncated) labels ([#21](https://github.com/Shweta-Mishra-ai/tokenmizer/pull/21)), semantic-opposite decision detection ([#26](https://github.com/Shweta-Mishra-ai/tokenmizer/pull/26))\n- [**@floze-the-genius**](https://github.com/floze-the-genius) — dashboard stats authentication fix ([#35](https://github.com/Shweta-Mishra-ai/tokenmizer/pull/35))\n\n## Support\n\nIf TokenMizer is useful to you, please give it a\n[⭐ star](https://github.com/Shweta-Mishra-ai/tokenmizer). It takes a\nsecond and it genuinely helps.\n\n[Sponsorship](https://github.com/sponsors/Shweta-Mishra-ai) is open too,\nif you would like to support the work. Entirely optional.\n\n## License\n\nMIT © [Shweta Mishra](https://github.com/Shweta-Mishra-ai)\n", "bytes": 18114, "sha": "a8ba98f01561f10f81076d8052c2853e4650dad9471122d97cab7b7014117f74", "repo_slug": "shweta-mishra-ai/tokenmizer", "fonte": "repo", "truncated": false, "api": "https://agentalog.com/api/listings/mcp_io_github_shweta_mishra_ai_tokenmizer_79fde34f/readme" }