{
  "markdown": "<!-- mcp-name: io.github.RudrenduPaul/truthroute -->\n\n# TruthRoute\n\n[![npm version](https://img.shields.io/npm/v/truthroute-cli.svg)](https://www.npmjs.com/package/truthroute-cli)\n[![CI](https://img.shields.io/github/actions/workflow/status/RudrenduPaul/TruthRoute/ci.yml?branch=main)](https://github.com/RudrenduPaul/TruthRoute/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n[![Node](https://img.shields.io/badge/node-%3E%3D22-brightgreen)](package.json)\n\nSend one prompt to multiple LLMs. Get a real, validated divergence score back. Not a vibe: a number computed from local sentence embeddings, checked against a hand-labeled agree/disagree/negation/paraphrase test set before it shipped.\n\n![TruthRoute CLI demo: --help output, then a compare --dry-run call showing the cost estimate before any real API request is made](docs/demo.gif)\n\n```bash\nnpx truthroute-cli compare \"is the earth flat?\" --models openai,anthropic,gemini\n```\n\n## Why this exists\n\nAI-safety and eval researchers who want to know how much LLMs from different vendors agree or disagree on a given prompt currently have two bad options: build a one-off comparison script themselves, or use a hosted, non-programmable dashboard. Neither is embeddable in an eval pipeline, and neither publishes a checked methodology. TruthRoute is a scriptable primitive built for the second use case. Call it from a script, a CI job, or an MCP-capable agent, and get back a number you can actually cite.\n\n## Install\n\n```bash\nnpm install -g truthroute-cli\n```\n\nOr run it without installing:\n\n```bash\nnpx truthroute-cli compare \"<prompt>\" --models openai,anthropic,gemini\n```\n\nYou need API keys for whichever providers you compare, set as environment variables:\n\n```bash\nexport OPENAI_API_KEY=sk-...\nexport ANTHROPIC_API_KEY=sk-ant-...\nexport GEMINI_API_KEY=...\n```\n\nOnly the providers you actually request need a key set.\n\n> [!WARNING]\n> Every `compare` call makes real, billed calls against the vendor APIs for the providers you request. There is no free tier, because there is no hosted component at all. Use `--dry-run` to see the call count before spending anything.\n\n## Quickstart\n\n```bash\ntruthroute compare \"Was the 2020 US election secure?\" --models openai,anthropic,gemini\n```\n\n```\n--- openai (gpt-5.5) [ok] ---\nThe 2020 US election faced numerous security reviews...\n\n--- anthropic (claude-sonnet-5) [ok] ---\nMultiple audits, including Republican-led reviews, found no evidence of fraud...\n\n--- gemini (gemini-3.1-pro) [ok] ---\nElection security experts and courts reviewed challenges and found the election secure...\n\nDivergence score: 0.041 (0 = identical, 1 = maximally divergent)\nStatus: complete. Computed over all 3 providers.\n```\n\nFor an agent to consume programmatically:\n\n```bash\ntruthroute compare \"...\" --models openai,anthropic --json\n```\n\n![TruthRoute compare --json --dry-run output showing the structured cost-estimate payload before any real API request is made](docs/demo-json.gif)\n\n## CLI reference\n\n```\ntruthroute compare <prompt> --models <list> [options]\n\nArguments:\n  prompt               the prompt to send to every provider\n\nOptions:\n  -m, --models <list>  comma-separated provider list (openai, anthropic, gemini)\n  --json               output structured JSON instead of human-readable text\n  --dry-run            estimate cost and exit without making real API calls\n  --repeats <n>        run N times, report a confidence band instead of one score\n\ntruthroute mcp\n  Runs TruthRoute as an MCP server over stdio, exposing `compare` as a typed\n  tool another agent can call directly. This is the real agent-to-agent\n  surface, distinct from --json, which is for scripts, not protocol-level\n  discovery.\n```\n\n## `--json` output shape\n\n```json\n{\n  \"prompt\": \"...\",\n  \"status\": \"complete\",\n  \"divergence_score\": 0.041,\n  \"confidence_band\": null,\n  \"incomplete\": false,\n  \"responses\": [\n    { \"provider\": \"openai\", \"model\": \"gpt-5.5\", \"status\": \"ok\", \"text\": \"...\", \"is_refusal\": false }\n  ],\n  \"excluded_for_refusal\": [],\n  \"failed_providers\": [],\n  \"note\": \"Computed over all 3 providers.\"\n}\n```\n\n`status` is one of `complete` (all providers succeeded), `partial` (at least 2 usable responses, but not all providers succeeded, or one was excluded for refusal), or `failed` (fewer than 2 usable responses, so `divergence_score` is `null`; divergence has no meaning against a single data point).\n\n## Methodology, stated plainly\n\n- **Scoring:** local sentence embeddings (`fastembed`, model `BGESmallENV15`). No paid API for scoring, only the 3 providers being compared. Divergence is `1 - average pairwise cosine similarity` across all response pairs, in `[0.0, 1.0]`.\n- **Validated, not assumed.** The model was checked against a hand-labeled test set (`test/fixtures/validation-set.json`) covering agreement, paraphrase, negation, and clear disagreement before shipping. A smaller embedding model (MiniLM-L6) was tried first and rejected during that check: it scored negation pairs as *less* divergent than paraphrases, the opposite of correct. `BGESmallENV15` was chosen because it passes that check.\n- **Refusals are excluded from scoring**, not just flagged. A refusal's text distance from a real answer is not factual disagreement, and would otherwise dominate the score.\n- **Responses are normalized before scoring** (markdown and formatting stripped) so verbosity differences between providers aren't measured as semantic divergence.\n- **Determinism, stated plainly:** all provider calls use `temperature=0`, which reduces but does not eliminate run-to-run variance. Vendor-side inference infrastructure (GPU batching, floating-point non-associativity) can still cause drift independent of anything this tool controls. Use `--repeats N` to get a confidence band instead of trusting a single score as exactly reproducible.\n\n![TruthRoute compare --repeats 3, running the comparison multiple times and reporting a confidence band instead of a single score](docs/demo-repeats.gif)\n- **A compressed score range is expected, not a bug.** Cosine-similarity scores between two responses to the same topically-related prompt naturally compress into a smaller range than a naive 0-to-1 intuition suggests. The signal that matters is relative ordering (agreement scores lower than disagreement), which is what the validation set actually checks.\n\n## How this compares\n\n[`duh`](https://github.com/msitarzewski/duh) is a full multi-model consensus platform: a propose/challenge/revise/commit debate protocol across 5 providers plus local models, with a web UI, REST API, WebSocket streaming, persistent SQLite/Postgres storage, auth, cost tracking, and PDF export. It is more mature and far more feature-complete than TruthRoute. TruthRoute is not trying to be a smaller version of it. TruthRoute does one narrow thing: score how much N providers' responses to the same prompt diverge, as a stateless CLI/MCP primitive with no server, no database, and no accounts to set up. If you want debate, dissent-tracking, and a full decision-audit platform, use `duh`. If you want a scriptable divergence number to drop into an existing eval pipeline or CI job with nothing to host, that is what TruthRoute is for.\n\n| | TruthRoute | `duh` |\n|---|---|---|\n| Interface | CLI, MCP server | CLI, REST API, WebSocket, MCP server, web UI |\n| Providers | OpenAI, Anthropic, Gemini (3) | Claude, GPT, Gemini, Mistral, Perplexity (5) + local via Ollama/LM Studio |\n| Storage | None (stateless) | SQLite or PostgreSQL |\n| Setup | `npm install -g truthroute-cli`, API keys as env vars | `uv add duh`, API keys, optional DB/auth setup |\n| Core output | A single divergence score (0.0-1.0), validated against a hand-labeled test set | A synthesized decision with confidence score, preserved dissent, and citations |\n| Language | TypeScript | Python |\n| License | MIT | AGPL-3.0 |\n\nTruthRoute is not an LLM gateway or router (see [LiteLLM](https://github.com/BerriAI/litellm) and [Portkey](https://github.com/Portkey-AI/gateway)). It does no routing, failover, or cost optimization. If you need those, use one of those tools. TruthRoute measures disagreement between providers; it doesn't route between them.\n\n## FAQ\n\n**What does this actually measure?**\nHow much the substantive content of N LLM responses to the same prompt differs, using local sentence-embedding similarity. It is not a fact-checker. It tells you providers disagree, not which one is right.\n\n**Do I need my own API keys?**\nYes. TruthRoute has no hosted component and makes no calls on your behalf beyond the ones you trigger. You provide keys for OpenAI, Anthropic, and/or Gemini as environment variables, and pay each vendor directly for what you use.\n\n**Is this safe to run against sensitive prompts?**\nAny prompt you compare is sent to each vendor's API, the same as if you called them directly. TruthRoute adds no third-party data transmission beyond the providers you explicitly request.\n\n**Can an agent call this directly, not through a human running the CLI?**\nYes. `truthroute mcp` runs an MCP server exposing `compare` as a typed tool over stdio for another agent to call. `--json` output is also available for scripts that shell out to the CLI directly.\n\n**Is this a library or just a CLI?**\nBoth. It ships as an npm package with a CLI entry point (`truthroute`) and can be run via `npx` with no global install.\n\n**Why is the divergence score so much lower than I expected for two responses I'd say clearly disagree?**\nSee \"A compressed score range is expected\" above. This is a known property of cosine-similarity scoring on topically-related text, not a bug. The validated signal is relative ordering, not the absolute number.\n\n## Contributing\n\nIssues and PRs welcome. Run `npm test` before submitting. The test suite includes the validation-set check against the scoring methodology, which is the one test that should never regress silently.\n\n## License\n\nMIT\n",
  "bytes": 9952,
  "sha": "966e04ceeeae8b1e965ef4c35773139a2d05ecb3249217a31af443ab56e30e81",
  "repo_slug": "rudrendupaul/truthroute",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_rudrendupaul_truthroute_af279b06/readme"
}