{ "markdown": "
\n

llm-transpile

\n\n

\n \"Stars\"\n \"Forks\"\n \"Issues\"\n \"Last\n

\n

\n \"Crates.io\"\n \"docs.rs\"\n \"License\"\n \"Rust\"\n \"Buy\n

\n\n**Token-optimized document transpiler for LLM pipelines**\n\n[한국어](docs/i18n/README.ko.md) · [日本語](docs/i18n/README.ja.md) · [中文](docs/i18n/README.zh.md) · [Español](docs/i18n/README.es.md) · [Français](docs/i18n/README.fr.md) · [Deutsch](docs/i18n/README.de.md) · [Português](docs/i18n/README.pt.md) · [Русский](docs/i18n/README.ru.md) · [العربية](docs/i18n/README.ar.md) · [हिन्दी](docs/i18n/README.hi.md)\n\n
\n\nRaw documents (Markdown, HTML, plain text) → structured bridge format `?` — with adaptive compression that keeps you under token budget.\n\n---\n\n
\nTable of Contents\n\n- [Why](#why)\n- [Installation](#installation)\n- [Updating](#updating)\n- [CLI Usage](#cli-usage)\n- [Usage Statistics](#usage-statistics)\n- [Benchmarking](#benchmarking)\n- [Library Usage](#library-usage)\n- [Output Format](#output-format)\n- [Fidelity Levels](#fidelity-levels)\n- [Adaptive Compression](#adaptive-compression)\n- [Input Formats](#input-formats)\n- [Error Handling](#error-handling)\n- [Performance](#performance)\n- [Contributing](#contributing)\n- [License](#license)\n\n
\n\n---\n\n## Why\n\nLLMs perform better when context is clean and dense. This library handles the mechanical work:\n\n| | Feature | Why it matters |\n|--|---------|----------------|\n| 🏗️ | **Structural parsing** | Markdown/HTML/plain text → typed IR nodes (headings, paragraphs, tables, lists, code blocks) |\n| 📉 | **Adaptive compression** | Automatically escalates through 4 stages as token budget fills up |\n| 🔣 | **Symbol substitution** | Repeated domain terms → Unicode PUA characters, decoded by `` dictionary header |\n| 📊 | **Table linearization** | Markdown tables → compact `Key:Val` (≤5 rows) or pipe-separated rows for larger tables |\n| 🌊 | **Streaming output** | Tokio stream delivers the first chunk immediately, minimizing TTFT |\n\n### Benchmarks\n\n48 documents, 3 formats, 15 languages — Apple M-series, `--release` build. Numbers below are measured with the **real `cl100k` BPE tokenizer** (not the self-referential heuristic — see the analysis). Full methodology and token-honesty breakdown: [`docs/EVALUATION.md`](docs/EVALUATION.md)\n\n| Format | Semantic reduction | Compressed reduction | Lossless word coverage | Throughput |\n|--------|-------------------:|--------------------:|----------------------:|-----------:|\n| Markdown | 27.4% | 69.4% | 99.0% | — |\n| HTML | 98.7% | 99.3% | 99.0% | — |\n| PlainText | −3.5% | 30.4% | 99.0% | — |\n| **Overall (BPE)** | **81.5%** | **91.8%** | **99.0%** | **~1,070 tok/ms** |\n\n> ⚠️ The overall figure is dominated by HTML markup stripping. **Markdown 27.4% is the genuine compression rate.** PlainText is net-negative in Semantic mode due to structural overhead. See [`docs/EVALUATION.md`](docs/EVALUATION.md) for the per-format reality.\n\n> HTML reduction reflects markup overhead removal (nav, scripts, styles), not prose compression alone.\n\n---\n\n## Installation\n\n### Claude Code\n\n```\n/plugin marketplace add epicsagas/plugins\n/plugin install transpile@epicsagas\n```\n\nAuto-installs the binary and seeds the PostToolUse hook on next session start — no additional setup required.\n\n### Codex CLI\n\n```bash\ncodex plugin marketplace add epicsagas/plugins\n```\n\nThe PostToolUse hook is registered automatically — no further steps needed.\n\n### macOS / Linux\n\n```bash\nbrew install epicsagas/tap/llm-transpile\n```\n\nNo Homebrew? Use the installer script:\n\n```bash\ncurl --proto '=https' --tlsv1.2 -LsSf \\\n https://github.com/epicsagas/llm-transpile/releases/latest/download/install.sh | sh\n```\n\n### Windows\n\n```powershell\nirm https://github.com/epicsagas/llm-transpile/releases/latest/download/install.ps1 | iex\n```\n\n### Via Rust toolchain\n\n```bash\ncargo binstall llm-transpile # pre-built binary (fast)\ncargo install llm-transpile # build from source\n```\n\n### After installing\n\nConfigure tool integrations:\n\n```bash\ntranspile install\n```\n\n`transpile install` launches an interactive wizard that detects and configures whichever tools are installed:\n\n| Tool | Integration method | What it does |\n|------|--------------------|--------------|\n| **Antigravity** | `SKILL.md` | LLM auto-invokes `transpile` on document file extensions |\n| **Cursor** | `.mdc` rule (`alwaysApply`) | Triggers `transpile` before reading document files |\n| **OpenCode** | `SKILL.md` | LLM auto-invokes `transpile` on document file extensions |\n| **Cline** | `SKILL.md` | LLM auto-invokes `transpile` on document file extensions |\n\nAll tools use a skill file that teaches the LLM to run `TRANSPILE_AGENT= transpile --input ` automatically — no size check needed, extension alone triggers it.\n\n**Selective install / uninstall**\n\n```bash\ntranspile install antigravity cursor # specific tools only\ntranspile install --all # everything at once\ntranspile install --dry-run # preview what would change\ntranspile install --list # show status of all integrations\n\ntranspile uninstall cursor # remove one\ntranspile uninstall --all # remove everything\ntranspile uninstall --dry-run # preview removals\n```\n\n### Library (Rust crate)\n\n```toml\n[dependencies]\nllm-transpile = \"0.1\"\n```\n\nRequires **Rust 1.92+**.\n\n**Antigravity (Gemini CLI)**\n\n```bash\nagy plugins install https://github.com/epicsagas/llm-transpile\n```\n\nAuto-installs the plugin (hooks) and registers it on next session start.\n\n---\n\n## Updating\n\n| Method | Command |\n|--------|---------|\n| Homebrew | `brew upgrade llm-transpile` |\n| curl / PowerShell installer | Re-run the install command above |\n| cargo binstall | `cargo binstall llm-transpile@latest` |\n| cargo install | `cargo install llm-transpile@latest` |\n\n```bash\ntranspile --version\n```\n\n---\n\n## CLI Usage\n\n```\ntranspile [OPTIONS]\n\nOptions:\n -i, --input Input file path (reads from stdin if omitted)\n -f, --format Input format: markdown | html | plaintext [default: markdown]\n Auto-detected from file extension when --input is used\n -l, --fidelity Compression level: lossless | semantic | compressed [default: semantic]\n -b, --budget Token budget upper limit (unlimited if omitted)\n -c, --count Print only the input token count, then exit\n -j, --json Output as JSON {input_tok, output_tok, reduction_pct, content}\n -q, --quiet Suppress the stats line on stderr\n --stats Print stats line to stdout after content (single-stream capture)\n -h, --help Print help\n -V, --version Print version\n```\n\n**Examples**\n\n```bash\n# Convert a Markdown file (format auto-detected from .md extension)\ntranspile --input doc.md\n\n# Read from stdin — clean stdout, stats on stderr\ncat doc.html | transpile --format html --fidelity compressed --budget 1024\n\n# Pipe cleanly — suppress stats entirely\ntranspile --input doc.md --quiet | send_to_llm_api\n\n# Check token count without converting\ntranspile --input doc.md --count\n\n# JSON output for scripts and pipelines\ntranspile --input doc.md --json | jq '.reduction_pct'\n\n# Capture content + stats in one stream (stdout)\ntranspile --input doc.md --stats > output_with_stats.txt\n\n# Lossless — no compression, full content preserved (legal/audit docs)\ntranspile --input contract.md --fidelity lossless\n\n# Aggressive compression into a 512-token budget\ntranspile --input article.md --fidelity compressed --budget 512\n```\n\n> Stats (`[273 → 150 tok 45.1% reduction]`) are written to **stderr** by default, so stdout stays clean for piping. Use `--quiet` to suppress, or `--stats` to redirect to stdout.\n\n---\n\n## Usage Statistics\n\nEvery `transpile` invocation automatically appends a record to `~/.agents/transpile/stats/YYYY-MM-DD.jsonl`.\n\n### ASCII table\n\n```bash\ntranspile stats show # today\ntranspile stats show --days 7 # last N days\ntranspile stats show --agent claude # filter by agent\n```\n\nExample output:\n\n```\ntranspile stats — last 7 days\n\n Date Agent Calls Input tok Output tok Saved Reduction\n ──────────────────────────────────────────────────────────────────────────\n 2026-05-18 238 4 999 355 4 248 769 750 586 15.0%\n 2026-05-19 390 1 577 739 1 463 504 114 235 7.2%\n 2026-05-20 288 2 148 207 1 836 916 311 291 14.5%\n 2026-05-21 99 635 313 544 709 90 604 14.3%\n 2026-05-22 299 8 328 530 7 732 860 595 670 7.2%\n 2026-05-23 418 15 939 148 13 501 134 2 438 014 15.3%\n 2026-05-24 186 3 313 950 2 782 467 531 483 16.0%\n ──────────────────────────────────────────────────────────────────────────\n Total 1919 36 942 242 32 110 359 4 831 883 13.1%\n```\n\n### HTML dashboard\n\n```bash\ntranspile stats report # opens in browser (default: last 7 days)\ntranspile stats report --days 30 # last 30 days\ntranspile stats report --no-open # generate without opening\ntranspile stats report --out /tmp/custom.html\n```\n\n> Reports are generated at `~/.agents/transpile/reports/` by default. Override with `--out`.\n\n**JSONL record fields**\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `ts` | ISO 8601 | Timestamp of the invocation |\n| `agent` | string | Tool that triggered the call (`claude`, `antigravity`, `codex`, `opencode`) |\n| `file` | string | Input file path (empty when reading from stdin) |\n| `format` | string | `markdown`, `html`, or `plaintext` |\n| `fidelity` | string | `lossless`, `semantic`, or `compressed` |\n| `input_tok` | integer | Token count before transpilation |\n| `output_tok` | integer | Token count after transpilation |\n| `reduction_pct` | float | Percentage of tokens saved |\n| `saved` | integer | Absolute tokens saved (`input_tok − output_tok`) |\n\n**`TRANSPILE_AGENT` environment variable**\n\nThe `agent` field is populated from the `TRANSPILE_AGENT` environment variable. Each integration sets this automatically (`claude`, `antigravity`, `codex`, `opencode`, `cursor`). You can also set it manually:\n\n```bash\nTRANSPILE_AGENT=claude transpile --input doc.md\n```\n\n### Benchmarking\n\n```bash\n# Run benchmarks against a directory of test files\ntranspile bench run --dataset ./eval # generates JSONL log\ntranspile bench run --dataset ./eval --report # run + open HTML report\ntranspile bench report # regenerate report from logs\n```\n\nThe HTML benchmark report includes:\n\n- **KPI cards** — semantic reduction, compressed reduction, throughput (tok/ms), word coverage, total input tokens, run count\n- **7 charts** — reduction trend over time, throughput per run, semantic vs throughput scatter, box plot per format, format distribution, token size histogram, word coverage donut\n- **Runs table** — per-run summary with aggregate metrics\n- **Records table** — per-file detail with filter by format, run, and filename\n- **Theme toggle** — dark / light mode with persistent preference\n- **Bilingual** — auto-detects Korean locale; manual 한/EN toggle\n\n---\n\n## Library Usage\n\n### Synchronous\n\n```rust\nuse llm_transpiler::{transpile, FidelityLevel, InputFormat};\n\nlet md = r#\"\n# Software License Agreement\n\nThis agreement is made between Licensor and Licensee.\n\n| Item | Cost |\n|----------|-------|\n| Base fee | $800 |\n| Support | $200 |\n\"#;\n\nlet output = transpile(md, InputFormat::Markdown, FidelityLevel::Semantic, Some(4096))?;\nprintln!(\"{}\", output);\n```\n\n### Streaming (Tokio)\n\n```rust\nuse llm_transpiler::{transpile_stream, FidelityLevel, InputFormat};\nuse futures::StreamExt;\n\nlet mut stream = transpile_stream(input, InputFormat::Markdown, FidelityLevel::Semantic, 4096).await;\n\nwhile let Some(chunk) = stream.next().await {\n let chunk = chunk?;\n print!(\"{}\", chunk.content);\n if chunk.is_final { break; }\n}\n```\n\n### Token count estimate\n\n```rust\nlet n = llm_transpiler::token_count(\"Hello, world!\");\n```\n\n---\n\n## Output Format\n\n```\n ← Symbol dictionary (omitted when no substitutions occur)\n{sym}=repeated-term\n\n ← YAML-like metadata header\nt: document title\ns: one-line summary\nk: [keyword1, keyword2]\n\n ← Document body (compressed + substituted)\n...content...\n\n```\n\nThe `` block uses Unicode Private Use Area characters (`U+E000–U+F8FF`) as compact symbol handles, avoiding collision with visible text patterns. The dictionary supports up to **6,400 unique terms** per document.\n\n---\n\n## Fidelity Levels\n\n| Level | Typical use case | Compression applied |\n|-------|-----------------|---------------------|\n| `Lossless` | Legal / audit documents | None — original content guaranteed |\n| `Semantic` | General RAG pipelines | Stopword removal + low-importance pruning |\n| `Compressed` | Summarization, tight budgets | Maximum compression, first-sentence extraction |\n\n---\n\n## Adaptive Compression\n\nThe compressor monitors budget usage in real time and escalates automatically:\n\n| Budget usage | Stage | What happens |\n|---|---|---|\n| 0–60% | `StopwordOnly` | English/Korean stopwords stripped |\n| 60–80% | `PruneLowImportance` | Bottom 20% of paragraphs by importance score removed |\n| 80–95% | `DeduplicateAndLinearize` | Duplicate sentences removed; tables linearized |\n| 95%+ | `MaxCompression` | Each paragraph truncated to first sentence |\n\n> `Lossless` mode bypasses all compression stages unconditionally.\n\nDuring streaming, when budget usage crosses 80%, remaining nodes are automatically switched to `Compressed` mode.\n\n---\n\n## Input Formats\n\n| `InputFormat` | Parser |\n|---|---|\n| `Markdown` | [pulldown-cmark](https://crates.io/crates/pulldown-cmark) — CommonMark + GFM tables |\n| `Html` | ammonia sanitization → tag stripping → plain text pipeline |\n| `PlainText` | Blank-line paragraph splitting |\n\n---\n\n## Error Handling\n\n```rust\nuse llm_transpiler::TranspileError;\n\nmatch transpile(input, format, fidelity, budget) {\n Ok(output) => { /* use output */ }\n Err(TranspileError::Parse(msg)) => eprintln!(\"parse failed: {msg}\"),\n Err(TranspileError::SymbolOverflow(e)) => eprintln!(\"too many unique terms: {e}\"),\n Err(TranspileError::LosslessModeViolation) => eprintln!(\"compression in lossless mode\"),\n Err(e) => eprintln!(\"error: {e}\"),\n}\n```\n\n---\n\n## Performance\n\nMeasured on release build (`cargo build --release`), Apple M-series, 48 documents across Markdown / HTML / PlainText. All reduction figures are measured with the **real `cl100k` BPE tokenizer** (not the self-referential heuristic). See [`docs/EVALUATION.md`](docs/EVALUATION.md) for the full methodology and per-format breakdown.\n\n| Metric | Measured | Notes |\n|--------|----------|-------|\n| Throughput (Markdown-only peak) | **10,975 tok/ms** | ≈75× faster than Python parsing baseline; single-format peak |\n| Throughput (dataset aggregate) | **~1,070 tok/ms** | Weighted across all 48 docs / 3 formats (BPE) — see Benchmarks table |\n| Semantic reduction | **27.4%** (Markdown) | Genuine compression rate; within the 15–30% target band |\n| Compressed reduction | **69.4%** (Markdown) | Budget-adaptive, guaranteed ≥ PruneLowImportance |\n| Lossless word coverage | **99.0% avg** | Across all formats and languages |\n| HTML reduction | **98.7%** | Reflects markup overhead removal (nav/scripts/styles) |\n| Multilingual support | 15 languages tested | AR/DE/ES/FR/HI/IT/JA/KO/NL/PL/PT/RU/SV/TR/ZH — 99.0% avg word coverage |\n\nRun the evaluation suite yourself:\n\n```bash\nmake eval # structured JSON (BPE + heuristic; consumed by `epic eval`)\nmake eval-report # human-readable per-file table + summary\n```\n\nFull per-file breakdown, methodology, and the token-honesty analysis: [`docs/EVALUATION.md`](docs/EVALUATION.md) (한국어: [`docs/i18n/EVALUATION.ko.md`](docs/i18n/EVALUATION.ko.md))\n\n---\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for full guidelines. PRs welcome — check open issues labeled `good first issue`.\n\n---\n\n## License\n\nApache-2.0 — see [LICENSE](LICENSE).\n", "bytes": 18137, "sha": "804efb8c790c0dd0d4229dc166f8c4bfcc5e1c337c96831950e73e5bf6d69853", "repo_slug": "epicsagas/llm-transpile", "fonte": "repo", "truncated": false, "api": "https://agentalog.com/api/listings/plg_epicsagas_llm_transpile_llm_transpile_16d83c2a/readme" }