io.github.base76-research-lab/token-compressor
Compress prompts 40-60% using local LLM + embedding validation. Preserves all conditionals.
Open source Open in the app JSON README (API)
About
Compress prompts 40-60% using local LLM + embedding validation. Preserves all conditionals.
Details
- Kind
- MCP servers
- Topic
- AI, RAG & memory
- Publisher
- base76-research-lab
- Origin
- official
- Category
- ferramentas
- Transport
- local
- Version
- 0.1.1
- Stars
- 10
- Forks
- 2
- Open pull requests
- 1
- Last push
- 2026-03-08T16:29:24Z
- Repository state
- ativo
- Language
- Python
- License
- MIT
- Added
- 2026-08-29 03:02:28
- Updated
- 2026-08-29 03:02:28
- Origin id
io.github.base76-research-lab/token-compressor
README
# token-compressor
## Reduce LLM prompt tokens by 30–70% while preserving semantic meaning.
mcp-name: io.github.base76-research-lab/token-compressor
**Semantic prompt compression for LLM workflows. Reduce token usage by 40–60% without losing meaning.**
[](LICENSE)
[](https://ollama.com)
[](https://modelcontextprotocol.io)
Built by [Base76 Research Lab](https://base76.se) — research into epistemic AI architecture.
---
## Live demo
Intent Compiler MVP is now live and uses this project as part of the idea -> spec -> compressed output flow:
- Live: https://intent-compiler-mvp.pages.dev
- Product repo: https://github.com/base76-research-lab/token-compressor
---
## What it does
token-compressor is a two-stage pipeline that compresses prompts before they reach an LLM:
1. **LLM compression** — a local model (llama3.2:1b via Ollama) rewrites the prompt to its semantic minimum, preserving all conditionals and negations
2. **Embedding validation** — cosine similarity between original and compressed embeddings must exceed a threshold (default: 0.85) — if not, the original is sent unchanged
The result: shorter prompts, lower costs, same intent.
```
Input prompt (300 tokens)
↓
LLM compresses
↓
Embedding validates (cosine ≥ 0.85?)
↓
Pass → compressed (120 tokens) Fail → original (300 tokens)
```
**Key design principle:** conditionality is never sacrificed. If your prompt says "only do X if Y", that constraint survives compression.
---
## Requirements
- Python 3.10+
- [Ollama](https://ollama.com) running locally
- Two models pulled:
```bash
ollama pull llama3.2:1b
ollama pull nomic-embed-text
```
- Python dependencies:
```bash
pip install ollama numpy
```
---
## Quick start
```python
from compressor import LLMCompressEmbedValidate
pipeline = LLMCompressEmbedValidate()
result = pipeline.process("Your prompt text here...")
print(result.output_text) # compressed (or original if validation failed)
print(result.report()) # MODE / COVERAGE / TOKENS saved
```
**Result object:**
| Field | Description |
|-------|-------------|
| `output_text` | Text to send to your LLM |
| `mode` | `compressed` / `raw_fallback` / `skipped` |
| `coverage` | Cosine similarity (0.0–1.0) |
| `tokens_in` | Estimated input tokens |
| `tokens_out` | Estimated output tokens |
| `tokens_saved` | Difference |
---
## CLI usage
```bash
echo "Your long prompt here..." | python3 cli.py
```
Output: compressed text on stdout, stats on stderr.
---
## Claude Code hook (recommended setup)
Add to your `~/.claude/settings.json` under `hooks → UserPromptSubmit`:
```json
{
"type": "command",
"command": "echo \"${CLAUDE_USER_PROMPT:-}\" | python3 /path/to/token-compressor/cli.py > /tmp/compressed_prompt.txt 2>/tmp/compress.log || true"
}
```
This runs on every prompt submission and writes the compressed version to a temp file, which can be injected back into context via a second hook or MCP server.
---
## MCP server
The MCP server exposes compression as a tool callable from Claude Code and any MCP-compatible client.
**Install:**
```bash
pip install token-compressor-mcp
```
**Tool:** `compress_prompt`
- Input: `text` (string)
- Output: compressed text + stats footer
**Claude Code MCP config** (`~/.claude/settings.json`):
```json
{
"mcpServers": {
"token-compressor": {
"command": "uvx",
"args": ["token-compressor-mcp"]
}
}
}
```
Or from source:
```json
{
"mcpServers": {
"token-compressor": {
"command": "python3",
"args": ["-m", "token_compressor_mcp"],
"cwd": "/path/to/token-compressor"
}
}
}
```
---
## Configuration
```python
pipeline = LLMCompressEmbedValidate(
threshold=0.85, # cosine similarity floor (lower = more aggressive)
min_tokens=80, # skip pipeline below this (not worth compressing)
compress_model="llama3.2:1b",
embed_model="nomic-embed-text",
)
```
---
## How it works
**Stage 1 — LLM compression**
The compression prompt instructs the model to:
- Preserve all conditionals (`if`, `only if`, `unless`, `when`, `but only`)
- Preserve all negations
- Remove filler, hedging, redundancy
- Target 40–60% of original length
**Stage 2 — Embedding validation**
Computes cosine similarity between the original and compressed text using `nomic-embed-text`. If similarity falls below threshold, the original is returned unchanged. This prevents silent meaning loss.
---
## Results
Tested across Swedish and English prompts, technical and natural language:
| Input | Tokens in | Tokens out | Saved |
|-------|-----------|------------|-------|
| Research abstract (EN) | 89 | 38 | 57% |
| Session intent (SV) | 32 | 18 | 44% |
| Technical instruction | 47 | 22 | 53% |
| Short command (<80t) | — | — | skipped |
---
## Research background
This tool implements the architecture from:
> Wikström, B. (2026). *When Alignment Reduces Uncertainty: Epistemic Variance
> Collapse and Its Implications for Metacognitive AI.*
> DOI: [10.5281/zenodo.18731535](https://doi.org/10.5281/zenodo.18731535)
Part of the [Base76 Research Lab](https://base76.se) toolchain for epistemic AI infrastructure.
---
## License
MIT — Base76 Research Lab, Sweden