{
  "markdown": "# densely\n\n**Lossless context compression for LLMs.** Pack any text into 2x–8x fewer\ntokens with guaranteed byte-exact reconstruction — verified by sha256 on\nevery decompress.\n\n![demo](assets/demo.gif)\n\nMIT, free forever. (A hosted tier is coming for teams —\n[waitlist](https://densely.dev/#pro).)\n\nAn o200k token can carry up to ~17.6 bits of information, but typical code\noccupies tokens at only ~5–6 bits each. densely reclaims the difference:\n\n```\ntext -> lzma -> 16-bit chunks -> 65,536 single-token English words\n```\n\nEach carrier word (`\" the\"`, `\" of\"`, …) costs exactly **1 token** — the\no200k pre-tokenizer never merges across word boundaries — so every token\nin the payload carries 2 bytes of compressed data (16 of the ~17.6\ntheoretically available bits, 91% of channel capacity).\n\n## Benchmarks\n\nReproduce with `python3 bench.py` (fixed seeds, stdlib code sample):\n\n| Scenario                      | Backend | Tokens (o200k)    | Ratio | Saved |\n|-------------------------------|---------|-------------------|-------|-------|\n| Code (argparse.py + densely) | lzma    | 21,659 → 10,726   | 2.02x | 50.5% |\n| Code (same sample)            | neural  | 21,659 → 2,538    | 8.53x | **88.3%** |\n| Code never seen by the model  | neural  | 3,433 → 472       | 7.27x | **86.3%** |\n| JSON (code search, 100 hits)  | lzma    | 15,465 → 1,995    | 7.75x | 87.1% |\n| Logs (SRE incident, ~1600 ln) | lzma    | 117,766 → 16,962  | 6.94x | 85.6% |\n\nThe neural backend (`--backend neural`, `python3 bench.py --neural`) drives\nan integer arithmetic coder with next-token probabilities from\nQwen2.5-Coder-0.5B, NNCP-style batched across segments. The 88.3% figure\nbenefits from the model having seen Python's stdlib during training; the\n86.3% row is this repo's own sources — code that did not exist before\n2026-08-08 — and is the honest number for novel code (~0.56 bit/byte).\n\n### Measured on real-world data\n\nSame pipeline pointed at data we didn't curate: this repo's own sources, an\ninstalled pip module, a slice of a real macOS `/var/log/install.log`, a real\n`package-lock.json`, and the live npm registry response for `express`.\no200k counted locally with tiktoken; Claude counted via Anthropic's own\n`count_tokens` endpoint (Sonnet 5, request overhead calibrated out), using\nthe `claude1` alphabet. Every row round-tripped byte-exact.\n\n| Data                              | Size   | o200k raw → payload | Saved | Claude raw → payload | Saved |\n|-----------------------------------|--------|---------------------|-------|----------------------|-------|\n| This repo's own code              | 25 KB  | 7,008 → 4,254       | 39.3% | 10,724 → 6,811       | 36.5% |\n| Installed pip module (factory.py) | 32 KB  | 6,332 → 3,956       | 37.5% | 10,625 → 6,334       | 40.4% |\n| Real system log (install.log)     | 195 KB | 65,684 → 4,204      | **93.6%** | 104,328 → 6,732  | **93.5%** |\n| Real package-lock.json            | 27 KB  | 11,421 → 3,690      | 67.7% | 15,743 → 5,910       | 62.5% |\n| Live npm registry JSON (express)  | 195 KB | 76,167 → 12,293     | 83.9% | 113,444 → 19,672     | 82.7% |\n\nFor comparison, [Headroom](https://github.com/headroomlabs-ai/headroom)\nreports 15–20% savings for coding agents and 60–95% on JSON — achieved by\n*dropping* content from context, with originals kept in a local cache with\na TTL. densely keeps the full data in the context itself, restorable\nbyte-for-byte with no external storage and no expiry.\n\nLossless compression below the entropy of the data is mathematically\nimpossible (Shannon; see also [Fundamental Limits of Prompt\nCompression](https://arxiv.org/abs/2407.15504)) — within that bound,\ndensely sits near the practical ceiling for a deterministic, CPU-only\nmethod.\n\n## Usage\n\n```bash\npip install \"densely[mcp]\"\n# extras: [neural] for the neural backend (torch + transformers)\n\ndensely compress  big_context.txt -o payload.dense\ndensely compress  src.py -o payload.dense --backend neural\ndensely decompress payload.dense   -o restored.txt   # byte-identical\ndensely stats     file1.py file2.json                # token savings\n```\n\nLibrary:\n\n```python\nfrom densely import compress, decompress\n\npayload = compress(text)        # ~2x-8x fewer tokens\nassert decompress(payload) == text  # always true, sha256-checked\n```\n\n`compress` via the CLI self-verifies the round trip before writing output;\n`decompress` raises `ValueError` on any corruption or hash mismatch.\n\n## Use it in Claude Code / Cursor (MCP)\n\n```bash\n# Claude Code\nclaude mcp add --scope user densely -- \"$(which densely-mcp)\"\n\n# Cursor (~/.cursor/mcp.json) and other MCP clients\n{\"mcpServers\": {\"densely\": {\"command\": \"/absolute/path/to/densely-mcp\"}}}\n```\n\nThree tools:\n\n- **compress_file(path)** — agent calls this *instead of reading* a large\n  log/JSON/dump: gets a preview + dense payload at 2x-8x fewer tokens.\n- **compress_text(text)** — same for a big tool output already in hand.\n- **expand(payload | payload_file, start_line, end_line)** — exact\n  original back, sha256-verified; line ranges let the agent pay only for\n  the slice it needs.\n\nEvery compression also writes a `.dense` sidecar file next to the\noriginal — a plain text file you can commit, ship, or archive; no cache,\nno TTL, nothing to expire. Small payloads are additionally returned\ninline. This matters for context compaction: a compaction summary\nreplaces old conversation content, so an inline payload alone could be\nsummarized away — but the sidecar on disk (and its path, which carries\ninto summaries) cannot. If you use the Anthropic compaction API beta\ndirectly, add this to your `instructions` so payload references survive\nverbatim:\n\n> Preserve any densely payload references (paths ending in .dense, or\n> lines starting with DENSE1/DENSE2) verbatim in the summary. Do not\n> call any tools while writing this summary.\n\n## Automatic mode (hook)\n\nZero-effort savings: a PostToolUse hook compresses every large tool\noutput (Bash output, Read of non-code files, >= 5,000 tokens) on the\nfly — the agent sees a preview + stats, exact content stays available\nvia search/expand, and every replacement is recorded in a savings\nledger (`stats` tool, or the running total shown in each replacement).\n\nAdd to `~/.claude/settings.json` (absolute paths — hooks run outside\nyour shell PATH):\n\n```json\n{\n  \"hooks\": {\n    \"PostToolUse\": [{\n      \"matcher\": \"Read|Bash\",\n      \"hooks\": [{\n        \"type\": \"command\",\n        \"command\": \"/absolute/path/to/densely-hook\",\n        \"timeout\": 30\n      }]\n    }]\n  }\n}\n```\n\n(`which densely-hook` after install gives the absolute path — hooks run\noutside your shell PATH, so absolute paths only.)\n\nTune with `DENSELY_HOOK_MIN_TOKENS` (default 5000). Code files the agent\nis editing are never touched.\n\n## When it saves tokens (and when it doesn't)\n\nAgent sessions resend the whole history to the API on every turn, so\nanything sitting in context is paid for again and again. What densely\ndoes to each kind of content:\n\n| Content in context | Savings | Why |\n|---|---|---|\n| Code the agent is *actively editing* | **none — don't compress it** | the agent must read it; payloads are unreadable |\n| Reference code (read once, kept \"just in case\") | 20–40% | exact copy stays cheap; `expand` a line range when needed |\n| Tool outputs: logs, JSON, dumps | **78–87%** | the agent never needed all 1,500 lines — preview + targeted expand covers it |\n| Anything that must survive compaction | indirect | payloads pass through compaction verbatim; summaries don't |\n\nNote on prompt caching: cached history is cheaper, but the context\n*window* stays the same size — densely primarily buys you room, then\nmoney.\n\n## vs. the field\n\nThe main players make different trade-offs (numbers from each project's\nown published benchmarks):\n\n|  | densely | [Headroom](https://github.com/headroomlabs-ai/headroom) | [claw-compactor](https://github.com/open-compress/claw-compactor) |\n|---|---|---|---|\n| Approach | entropy coding -> single-token carrier | lossy selection, model-readable output | 14 readable transform stages (2 lossy) |\n| Exact recovery | **always: sha256-verified, in-conversation** | TTL cache, retrieval on demand | LRU \"RewindStore\", no verification documented |\n| Logs | **85.6%** | ~92% (lossy) | 24.1% |\n| JSON | 87.1% (lossless) | 60–95% (lossy) | 81.9% (via lossy sampling) |\n| Code (active use) | none — by design | 15–20% (AST skeletons) | 25% |\n| Model reads output | no (search/expand tools) | yes | yes |\n| Agent integrations | MCP + hook + skill (Claude Code, Cursor) | proxy + wrap + MCP | CLI only |\n\nDifferent philosophies: *maximum savings with silent degradation* vs.\n*exactness or nothing*. Readable-lossy tools win on content the model\nmust keep reading; densely wins when the data must never be wrong and\nmust survive compaction, export, and machine moves. Use both.\n\n## The honest caveats\n\n- **The payload is not readable** — by humans or by the model. It looks\n  like a stream of random English words. Use it as a dense carrier for\n  exact data (chat history, tool outputs, source files) alongside a\n  readable summary; expand it with a tool call when exact content is\n  needed.\n- Savings depend on redundancy: highly repetitive data (JSON, logs)\n  compresses 7x+, dense prose ~1.5–2x, already-compressed or random data\n  ~0% (payload is never larger than a few header tokens worse than raw\n  input — check `stats` before shipping).\n- Density is per-tokenizer; reconstruction is always byte-exact. Two\n  alphabets ship today: `o200k` (65,536 words, exactly 16 bits/token on\n  OpenAI o200k) and `claude1` (1,024 words, exactly 10 bits/token on\n  Claude Sonnet 5 / Opus 5 / Fable 5 — harvested empirically through the\n  count_tokens API, since Anthropic's vocabulary isn't public). The MCP\n  server and hook default to `claude1`; the CLI defaults to `o200k`\n  (`--alphabet` / `DENSELY_ALPHABET` to override). Measured on a 172KB\n  log with count_tokens on Sonnet 5: raw 93,117 tokens -> claude1\n  payload 25,904 (72.2% saved; the o200k alphabet managed only 47.8%\n  there). Verify any alphabet against any target with\n  `python3 tools/calibrate.py`; harvest new ones with\n  `tools/build_alphabet.py`.\n- **When you don't need this** (fair point from a redditor): if you\n  control your agent harness end-to-end, you can build the same pattern\n  yourself — zstd/lzma the big outputs into files, give the agent a\n  search tool, keep payloads out of the model's input. densely is that\n  architecture packaged (plus sha256 verification, line-level retrieval,\n  and per-tokenizer carrier alphabets for the subset that must live\n  *inside* the prompt — content that has to survive compaction verbatim\n  or travel with an exported session). It earns its keep inside closed\n  harnesses (Claude Code, Cursor), ephemeral sandboxes, and anywhere\n  \"reliable disk + grep\" doesn't reach.\n- **Neural backend caveats**: slow (~85 KB of code takes minutes on Apple\n  Silicon vs milliseconds for lzma) and requires torch + a ~1 GB model\n  download on first use. Reconstruction is bit-exact only when\n  decompression runs the same model/software stack as compression — which\n  is why `compress(backend=\"neural\")` verifies the full round trip before\n  returning and silently falls back to lzma on any mismatch. The 100%\n  guarantee never rests on the neural path.\n\n## Densely Pro (coming)\n\nThe library is MIT and stays free. We're building a managed tier for\nteams running agents in production:\n\n- **Cloud neural compression** — 86% on code without a local GPU\n- **Managed proxy** — savings with zero code changes\n- **Team dashboard** — token savings per agent, per day, in dollars\n- **Cross-machine payloads** — compress in CI, expand anywhere\n\n**[Join the waitlist →](https://densely.dev/#pro)** (early access order +\nfounding-user pricing, no rug pull — the library stays MIT)\n\n## Roadmap\n\n- Cross-machine determinism for the neural backend (integer/fixed-point\n  inference or a mismatch-tolerant coder), so payloads compressed on one\n  machine decompress on another.\n- Larger/faster models via llama.cpp for better ratios at higher speed.\n\n## Tests\n\n```bash\npython3 -m pytest tests/test_core.py tests/test_mcp.py   # fast suites\npython3 -m pytest tests/test_neural.py                   # slow (~1 min): real LLM round trips\n```\n\nFast suite: byte-exact round-trips (unicode, CJK, emoji, random bytes,\npayload-lookalike inputs), tamper detection, carrier-alphabet density.\nNeural suite: single- and multi-segment round trips, no-silent-fallback,\nbeats-lzma check.\n\n<!-- mcp-name: io.github.alibaizhanov/densely -->\n",
  "bytes": 12542,
  "sha": "ce69d1218a3fc15601acea63d7e186d77e5fa6f6ea61e1137a5cdee8ccb2d49b",
  "repo_slug": "alibaizhanov/densely",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_alibaizhanov_densely_c6984645/readme"
}