{
  "markdown": "<p align=\"center\">\n  <img src=\"assets/datoon-banner.svg\" width=\"720\" />\n</p>\n\n<h1 align=\"center\">datoon</h1>\n\n<p align=\"center\">\n  <strong>smart structured-data→TOON gateway — converts only when it actually saves tokens</strong>\n</p>\n\n<p align=\"center\">\n  <a href=\"https://github.com/andrii-su/datoon/actions/workflows/tests.yml\"><img src=\"https://github.com/andrii-su/datoon/actions/workflows/tests.yml/badge.svg\" alt=\"Tests\"></a>\n  <a href=\"https://github.com/andrii-su/datoon/actions/workflows/pre-commit.yml\"><img src=\"https://github.com/andrii-su/datoon/actions/workflows/pre-commit.yml/badge.svg\" alt=\"Pre-commit\"></a>\n  <a href=\"https://github.com/andrii-su/datoon/actions/workflows/release.yml\"><img src=\"https://github.com/andrii-su/datoon/actions/workflows/release.yml/badge.svg\" alt=\"Release\"></a>\n  <a href=\"https://pypi.org/project/datoon/\"><img src=\"https://img.shields.io/pypi/v/datoon?color=0EA5E9\" alt=\"PyPI\"></a>\n  <a href=\"https://img.shields.io/badge/python-3.12%2B-3776AB\"><img src=\"https://img.shields.io/badge/python-3.12%2B-3776AB?logo=python&logoColor=white\" alt=\"Python\"></a>\n  <a href=\"./LICENSE\"><img src=\"https://img.shields.io/github/license/andrii-su/datoon\" alt=\"License: MIT\"></a>\n</p>\n\n<p align=\"center\">\n  <a href=\"#before--after\">Before/After</a> •\n  <a href=\"#install\">Install</a> •\n  <a href=\"#what-you-get\">What You Get</a> •\n  <a href=\"#how-it-works\">How It Works</a> •\n  <a href=\"#benchmarks\">Benchmarks</a> •\n  <a href=\"./INSTALL.md\">Full install guide</a>\n</p>\n\n______________________________________________________________________\n\nRaw structured data is often verbose in LLM prompts. [TOON](https://github.com/toon-format/toon) can save tokens — but blind conversion can also make payloads *worse*. `datoon` adds a decision layer: convert when structure and savings justify it, skip when they don't, and always explain why.\n\nSupports **JSON, CSV, JSONL, YAML, XML, Parquet, Avro, ORC, Excel, and Apple Numbers** — auto-detected from file extension.\n\n## Before / After\n\n<table>\n<tr>\n<td width=\"50%\">\n\n### JSON in the prompt (43 tokens)\n\n```json\n{\"users\":[\n  {\"id\":1,\"name\":\"Ada\",\"role\":\"admin\"},\n  {\"id\":2,\"name\":\"Lin\",\"role\":\"analyst\"},\n  {\"id\":3,\"name\":\"Grace\",\"role\":\"viewer\"}\n]}\n```\n\n</td>\n<td width=\"50%\">\n\n### datoon converts → TOON (24 tokens)\n\n```toon\nusers[3]{id,name,role}:\n  1,Ada,admin\n  2,Lin,analyst\n  3,Grace,viewer\n```\n\n```json\n{\"decision\":\"convert\",\"reason\":\"Estimated savings 44.19% (threshold 15.00%).\"}\n```\n\n</td>\n</tr>\n<tr>\n<td>\n\n### CSV from a data pipeline (111 tokens as JSON)\n\n```csv\nid,name,role\n1,Ada,admin\n2,Lin,analyst\n3,Grace,viewer\n```\n\n</td>\n<td>\n\n### datoon auto-converts → TOON (24 tokens)\n\n```bash\ndatoon data.csv --report-stdout\n```\n\nSame result. Zero JSON serialization in your code.\n\n</td>\n</tr>\n<tr>\n<td>\n\n### Non-uniform payload (26 tokens)\n\n```json\n{\"config\":{\"debug\":true},\"tags\":[\"a\",\"b\"]}\n```\n\n</td>\n<td>\n\n### datoon skips → keeps JSON\n\n```json\n{\"decision\":\"skip\",\"reason\":\"No uniform object arrays found with at least 3 rows.\"}\n```\n\nNo Node.js call. No silent corruption.\n\n</td>\n</tr>\n</table>\n\n**Same data. Right format. Always explained.**\n\n```\n┌──────────────────────────────────────────────────┐\n│  PAYLOAD SAVINGS (auto avg)    ████░░░░░░   28%  │\n│  PAYLOAD SAVINGS (agent skill) ████████░░   62%  │\n│  DECISION ACCURACY             ██████████  100%  │\n│  HARMFUL CONVERSIONS BLOCKED   ██████████  100%  │\n└──────────────────────────────────────────────────┘\n```\n\n> [!IMPORTANT]\n> datoon saves **payload tokens** — the structured data portion of your prompt. Token savings depend on payload shape: uniform tabular data converts well; deeply nested or non-uniform structures are skipped. Every decision includes a reason so pipelines can log, debug, and trust the outcome.\n\n## Install\n\n```bash\n# core (JSON, CSV, JSONL, XML — no extra deps)\nuv add datoon\npip install datoon\n\n# with YAML support\npip install \"datoon[yaml]\"\n\n# with Excel support\npip install \"datoon[excel]\"\n\n# with Parquet / ORC / Avro support\npip install \"datoon[columnar]\"\n\n# with Apple Numbers support\npip install \"datoon[numbers]\"\n\n# with tiktoken-based token counting\npip install \"datoon[tokens]\"\n\n# with MCP server\npip install \"datoon[mcp]\"\n\n# everything\npip install \"datoon[all]\"\n```\n\nRequires Python 3.12+. TOON conversion requires Node.js with `npx` in PATH — analysis and format reading work without it.\n\nFor Claude Code plugin, Codex, and MCP config → [**INSTALL.md**](./INSTALL.md).\n\n## What You Get\n\n| | What |\n|---|---|\n| `datoon` **CLI** | Auto-gate any supported format → TOON from terminal or scripts |\n| **Python API** | `convert_json_for_llm()` + `read_tabular()` for any LLM pipeline |\n| **MCP Server** | `convert_json`, `convert_text`, `analyze_json` tools for Claude Desktop, Cursor, Windsurf |\n| **Claude Code Plugin** | `/datoon` in-session trigger, installs from GitHub in one command |\n| **Codex Plugin** | Marketplace plugin — structured-data mode for Codex |\n\n### Supported input formats\n\n| Format | Extension | Extra needed |\n|---|---|---|\n| JSON | `.json` | — |\n| JSONL | `.jsonl`, `.ndjson` | — |\n| CSV | `.csv` | — |\n| XML | `.xml` | — |\n| YAML | `.yaml`, `.yml` | `datoon[yaml]` |\n| Excel | `.xlsx`, `.xls` | `datoon[excel]` |\n| Parquet | `.parquet` | `datoon[columnar]` |\n| Avro | `.avro` | `datoon[columnar]` |\n| ORC | `.orc` | `datoon[columnar]` |\n| Apple Numbers | `.numbers` | `datoon[numbers]` |\n\n## How It Works\n\n1. **Detect format** — from `--format` flag, file extension, or default to JSON for stdin\n1. **Read + normalize** — parse source into list of row dicts; serialize to compact JSON\n1. **Analyze structure** — uniform object arrays? acceptable depth? minimum rows?\n1. **Gate early** — non-candidates skip before any CLI call; no Node.js overhead\n1. **Convert + estimate** — TOON CLI runs, token savings calculated\n1. **Gate savings** — below threshold → return JSON; above → return TOON with report\n\nEvery path returns a `ConversionReport` with `decision`, `reason`, and token estimates. Pipelines never get silent surprises.\n\n______________________________________________________________________\n\n## Quick Start\n\n**JSON (stdin):**\n\n```bash\necho '{\"users\":[{\"id\":1,\"name\":\"Ada\"},{\"id\":2,\"name\":\"Lin\"},{\"id\":3,\"name\":\"Grace\"}]}' | datoon --report-stdout\n```\n\n**CSV (auto-detected from extension):**\n\n```bash\ndatoon data.csv --report-stdout\n```\n\n**JSONL:**\n\n```bash\ndatoon data.jsonl -o output.toon\n```\n\n**YAML (requires `datoon[yaml]`):**\n\n```bash\ndatoon data.yaml --report-stdout\n```\n\n**Parquet (requires `datoon[columnar]`):**\n\n```bash\ndatoon data.parquet --report ./report.json\n```\n\n**Explicit format override:**\n\n```bash\ndatoon --format csv < data.csv --report-stdout\n```\n\n**Force conversion (bypass gating — for experiments):**\n\n```bash\ndatoon data.json --force --report-stdout\n```\n\n______________________________________________________________________\n\n## Python API\n\n**JSON conversion:**\n\n```python\nfrom datoon import convert_json_for_llm, ConversionConfig, DatoonError\n\nconfig = ConversionConfig(min_savings_ratio=0.15, max_depth=6, min_uniform_rows=3)\n\ntry:\n    outcome = convert_json_for_llm(raw_json, config)\nexcept DatoonError as exc:\n    raise\n\n# outcome.payload_text  — TOON or original JSON\n# outcome.report.decision  — \"convert\" | \"skip\"\n# outcome.report.reason    — human-readable explanation\nsend_to_model(outcome.payload_text)\n```\n\n**Any format via `read_tabular`:**\n\n```python\nimport json\nfrom pathlib import Path\nfrom datoon import read_tabular, convert_json_for_llm, ConversionConfig\n\n# text formats: csv, jsonl, yaml, xml\nrows = read_tabular(\"csv\", text=csv_string)\n\n# binary formats: excel, parquet, orc, avro, numbers\nrows = read_tabular(\"parquet\", path=Path(\"data.parquet\"))\n\njson_text = json.dumps(rows, separators=(\",\", \":\"))\noutcome = convert_json_for_llm(json_text, ConversionConfig())\nsend_to_model(outcome.payload_text)\n```\n\n**Structure-only analysis (no Node.js required):**\n\n```python\nfrom datoon.analyzer import analyze_payload\nfrom datoon.models import ConversionConfig\n\nanalysis = analyze_payload(parsed_data, ConversionConfig())\nprint(analysis.is_candidate, analysis.reason)\n```\n\n______________________________________________________________________\n\n## MCP Server\n\n<!-- mcp-name: io.github.andrii-su/datoon -->\n\n`datoon` ships an [MCP](https://modelcontextprotocol.io) server with three tools:\n\n| Tool | Description |\n|---|---|\n| `convert_json` | Full JSON conversion with policy gating |\n| `convert_text` | Converts CSV, YAML, XML, or JSONL text with policy gating |\n| `analyze_json` | Structure analysis only — no Node.js needed |\n\n**Claude Desktop / Cursor / Windsurf config:**\n\n```json\n{\n  \"mcpServers\": {\n    \"datoon\": {\n      \"command\": \"uvx\",\n      \"args\": [\"--from\", \"datoon[mcp]\", \"datoon\", \"mcp\"]\n    }\n  }\n}\n```\n\n**Run locally:**\n\n```bash\ndatoon mcp     # or the standalone script: datoon-mcp\n```\n\nListed on the [MCP Registry](https://registry.modelcontextprotocol.io), [Smithery](https://smithery.ai), and [Glama](https://glama.ai). See [MARKETPLACES.md](./MARKETPLACES.md).\n\n______________________________________________________________________\n\n## Claude Code Plugin\n\nInstall directly from GitHub:\n\n```bash\nclaude plugin marketplace add andrii-su/datoon\nclaude plugin install datoon@datoon\n```\n\nTrigger in-session:\n\n```\n/datoon\nconvert this JSON to TOON if it saves tokens\nuse datoon mode for structured data\n```\n\n______________________________________________________________________\n\n## CLI Reference\n\n| Flag | Default | Description |\n|---|---|---|\n| `--format` | auto | Input format: `json`, `csv`, `jsonl`, `yaml`, `xml`, `excel`, `parquet`, `avro`, `orc`, `numbers` |\n| `--force` | `false` | Bypass gating and minimum savings threshold |\n| `--min-savings` | `0.15` | Minimum relative token savings required |\n| `--max-depth` | `6` | Maximum nesting depth for auto-conversion |\n| `--min-uniform-rows` | `3` | Minimum rows in uniform object arrays |\n| `--timeout` | `30` | Seconds before TOON CLI call is aborted |\n| `--report <path>` | — | Write JSON conversion report to file |\n| `--report-stdout` | — | Print JSON conversion report to stderr |\n| `-o <path>` | stdout | Output file path |\n| `--version` | — | Print version and exit |\n\nFormat is auto-detected from file extension. Use `--format` to override or when reading from stdin.\n\n______________________________________________________________________\n\n## Benchmarks\n\n```bash\nPYTHONPATH=src python benchmarks/run.py --dry-run\nPYTHONPATH=src python benchmarks/run.py\nPYTHONPATH=src python benchmarks/run.py --update-readme\n```\n\n### Why auto mode outperforms forced conversion\n\nAuto mode avoids low-benefit and high-risk payloads (`orders-nested`, `mixed-non-uniform`) while matching forced TOON's average token count on suitable ones. Every decision comes with a reasoned report.\n\n| Scenario | JSON Baseline | Forced TOON | `datoon` Auto |\n|---|---:|---:|---:|\n| Average tokens | 77 | 50 | 50 |\n| Avg token saved | 0.0% | 26.8% | **28.1%** |\n| Decision quality | n/a | Converts all | Converts `3/5`, skips harmful cases |\n\n<!-- BENCHMARK-TABLE-START -->\n\n| Dataset | JSON | TOON (forced) | Raw Saved | Auto | Auto Tokens | Auto Saved |\n|---|---:|---:|---:|---|---:|---:|\n| users-small | 54 | 40 | 25.9% | convert | 40 | 25.9% |\n| events-medium | 219 | 162 | 26.0% | convert | 162 | 26.0% |\n| orders-nested | 106 | 116 | -9.4% | skip | 106 | 0.0% |\n| mixed-non-uniform | 35 | 47 | -34.3% | skip | 35 | 0.0% |\n| metrics-wide | 142 | 103 | 27.5% | convert | 103 | 27.5% |\n| **Average** | **111** | **94** | **7.1%** | **3/5 convert** | **89** | **15.9%** |\n\n*Forced conversion succeeded for 5/5 payloads.*\n\n<!-- BENCHMARK-TABLE-END -->\n\n### Format conversion benchmark\n\nToken savings when converting from common structured formats (CSV, JSONL, XML, YAML).\nBaseline is the JSON representation of the same data — what an LLM would receive without datoon.\n\n<!-- FORMAT-BENCHMARK-TABLE-START -->\n\n| Dataset | Format | JSON Tokens | TOON (forced) | Auto | Auto Tokens | Auto Saved |\n|---|---|---:|---:|---|---:|---:|\n| users-csv | csv | 53 | 29 | convert | 29 | 45.3% |\n| events-jsonl | jsonl | 194 | 109 | convert | 109 | 43.8% |\n| catalog-xml | xml | 96 | 50 | convert | 50 | 47.9% |\n| metrics-yaml | yaml | 129 | 61 | convert | 61 | 52.7% |\n| **Average** | — | **118** | **62** | **4/4 convert** | **62** | **47.4%** |\n\n*Forced conversion succeeded for 4/4 payloads.*\n\n<!-- FORMAT-BENCHMARK-TABLE-END -->\n\n### Agent skill evaluation\n\nArtifact-based subagent comparison — identical analysis tasks, two modes:\n\n- `with_skill`: agent received the `datoon` skill and followed the conversion workflow.\n- `without_skill`: agent used JSON directly, no TOON or `datoon`.\n\n3 payload sizes × 3 iterations = 18 total agent runs. Both modes: 100% correct answers.\n\n| Scenario | Avg JSON Tokens | Avg TOON Tokens | Avg Payload Saved |\n|---|---:|---:|---:|\n| small | 225 | 118 | 47.6% |\n| medium | 2,972 | 1,138 | 61.7% |\n| large | 17,757 | 6,673 | 62.4% |\n\nFull report and raw outputs: [`benchmarks/agent_skill_eval/`](benchmarks/agent_skill_eval/). Savings are payload-token estimates, not full end-to-end model-token usage.\n\n______________________________________________________________________\n\n## Development\n\nContributor workflow: [CONTRIBUTING.md](./CONTRIBUTING.md). Maintainer/agent notes: [CLAUDE.md](./CLAUDE.md).\n\n**Setup:**\n\n```bash\nuv sync --extra dev\nuvx pre-commit install\n```\n\n**Tests:**\n\n```bash\npytest -m \"not integration\"   # unit only (102 tests)\npytest                        # with integration (requires Node.js + npx)\n```\n\n**Skill sync + plugin metadata:**\n\n```bash\npython scripts/validate_skill_sync.py\npython scripts/validate_plugin_metadata.py\n```\n\n______________________________________________________________________\n\n## Links\n\n- [INSTALL.md](./INSTALL.md) — full install matrix, all targets, per-agent detail\n- [CONTRIBUTING.md](./CONTRIBUTING.md) — contributor workflow\n- [CLAUDE.md](./CLAUDE.md) — maintainer guide for agents\n- [CHANGELOG.md](./CHANGELOG.md) — release history\n- [SECURITY.md](./SECURITY.md) — vulnerability reporting\n- [Live docs](https://andrii-su.github.io/datoon/) — `docs/`\n- [Issues](https://github.com/andrii-su/datoon/issues) — bugs, features, questions\n\n______________________________________________________________________\n\n## License\n\nMIT\n",
  "bytes": 14306,
  "sha": "efedc3354f46090fde5509e7ab581fecd1618ada5cff1392ed0297e445b1f891",
  "repo_slug": "andrii-su/datoon",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_andrii_su_datoon_05d45f06/readme"
}