{
  "markdown": "# retriEVAL\n\n**LLM evaluation as an MCP server.** Score your AI's outputs for faithfulness,\nrelevancy, and hallucination from inside any MCP client — no pipeline, no test\nharness. Every result comes back with a link to a dashboard that keeps the history.\n\n**[Try it live](https://retrieval-mcp.com)** (no signup) · **[Watch the 2-minute demo](https://youtu.be/zpXiv9isDmg)** · **[Dashboard](https://retrieval-mcp.com/dashboard)**\n\n---\n\n## Why\n\nFive customer-support answers, scored on two metrics:\n\n| metric | score | passing |\n|---|---|---|\n| answer_relevancy | 0.98 | 5/5 |\n| faithfulness | 0.70 | 3/5 |\n\nEvery answer was on-topic and well-written. Two of them contradicted the policy\nthey were supposedly grounded in — one promised free return shipping the policy\ndoesn't offer, another invented a free overnight replacement. Reviewing by eye,\nyou'd sign off on all five.\n\nThat gap is the point. Relevancy asks *did it answer the question*. Faithfulness\nasks *is it actually in the source*. You need both, and the second one catches\nthe expensive failures.\n\n## Connect\n\nSelf-hosted. Clone it, point it at a judge, run it — your data never leaves your\nmachine and there's no service to sign up for.\n\n```bash\ngit clone https://github.com/hcarrillo001/retrieval-mcp\ncd retrieval-mcp\npip install -r requirements.txt\n\nexport ANTHROPIC_API_KEY=sk-ant-...        # or a local judge, below\npython server.py                            # stdio, for Claude Desktop / Cursor\n```\n\nThen just ask:\n\n> Score these cases with faithfulness: [{\"input\": \"...\", \"actual_output\": \"...\", \"retrieval_context\": [\"...\"]}]\n\nPass your cases inline and nothing is stored — one call, no setup step. See\n[Run locally (stdio)](#run-locally-stdio--claude-desktop) for client config, and\n[Deploy as HTTP](#deploy-as-http-reach-it-from-anywhere) if you want your own\nalways-on instance with a dashboard.\n\n**Want to try it before installing anything?** There's a live sandbox at\n[retrieval-mcp.com](https://retrieval-mcp.com) — no signup, runs on a free judge,\nnothing saved.\n\n**Keeping everything local:** set `RETRIEVAL_JUDGE_BACKEND=ollama` and the judge\nruns on your machine too, so no data leaves your network at any point. Useful if\nyou're evaluating anything you can't send to a third party.\n\n## What you get\n\n- **9 built-in metrics** plus custom metrics you author in plain English\n- **Swappable judges** — Anthropic, Groq, Gemini, OpenRouter, or a local Ollama\n  model, so nothing has to leave your network\n- **Golden sets** from files, URLs, inline JSON, JSONL, CSV, or TSV\n- **Run history** in Supabase with shareable permalinks and run comparison\n- **A spend cap**, because a judge-based tool can otherwise run up a bill\n\n## Honest limitations\n\n- Judge agreement hasn't been validated against human labels yet, so treat\n  scores as a signal rather than ground truth.\n- Golden sets currently hold their own outputs, so re-running one against new\n  model outputs means loading a second set. Splitting them is the next change.\n- Golden sets and authored metrics live in the server process and are lost on\n  restart. Runs persist; those don't.\n\n---\n\n## Metrics (DeepEval-aligned)\n\n`faithfulness` · `answer_relevancy` · `contextual_precision` ·\n`contextual_recall` · `contextual_relevancy` · `hallucination` · `bias` ·\n`toxicity` · `summarization` — plus **authored G-Eval** metrics you define in\nplain language. All are normalized so **higher = better** (bias/toxicity report\nthe clean fraction), and each reasons before scoring.\n\n## Versatile golden sets\n\n`load_golden_set` accepts a **file path** (including uploaded files), an\n**http(s) URL**, an **inline JSON array**, or **JSONL text**, in\nJSON / JSONL / CSV / TSV. Field names are auto-normalized (`question`→input,\n`answer`→actual_output, `ground_truth`→expected_output, `contexts`→context,\n`passages`→retrieval_context, …), so most public benchmarks load as-is.\n\n## Short replies by default\n\n`run_eval` scores every case but returns only the **3 lowest-scoring** by\ndefault (tune with `limit`), with `total_cases`/`shown` and a pointer to\n`show_run_cases(run_id, offset, limit, metric)` to page through the rest.\n\n## Spend cap (so it can't run up a bill)\n\nJudge spend is metered from real token usage and persisted. Set a hard cap:\n\n```bash\nexport RETRIEVAL_BUDGET_USD=20     # 0/unset = unlimited\n```\n\nOnce cumulative spend hits the cap, further Anthropic calls stop and tools\nreturn a clear `budget_exceeded` message. Check/clear with `get_budget` /\n`reset_budget`. (Prices are approximate — override `RETRIEVAL_PRICE_IN/OUT`\n$/1M tokens to match current pricing for your model.)\n\n## Hybrid setup (local + always-on dashboard)\n\nRun history uses a pluggable store, chosen by env:\n- **FileStore** (default) — JSONL in `~/.retrieval`. Zero setup, local only.\n- **SupabaseStore** — when `SUPABASE_URL` + `SUPABASE_SERVICE_KEY` are set. Run\n  history lives in Postgres, shared by the local CLI, the deployed MCP, and the\n  website dashboard.\n\nRecommended hybrid flow:\n1. Run `supabase_schema.sql` in Supabase (creates the `runs` table).\n2. Set `SUPABASE_URL` + `SUPABASE_SERVICE_KEY` on the MCP (local and/or Railway)\n   so every run is written centrally. Each run records its `generator_model` and\n   `judge_model` for cross-model comparison.\n3. Deploy `web/` to Vercel (set the same Supabase env vars) and map it to\n   `retrieval-mcp.com`. The dashboard reads history via `/api/runs` (service key\n   stays server-side) and renders trend-by-model, a model leaderboard, and run\n   history. It shows sample data until Supabase is wired.\n\nLocal stays your free sandbox (Ollama judge, file history); the website is the\nalways-on window into the shared history.\n\n## Run locally (stdio) — Claude Desktop\n\n```bash\npip install -r requirements.txt\nexport ANTHROPIC_API_KEY=sk-ant-...\n```\n\n```json\n{\n  \"mcpServers\": {\n    \"retrieval\": {\n      \"command\": \"python\",\n      \"args\": [\"/ABSOLUTE/PATH/server.py\"],\n      \"env\": { \"ANTHROPIC_API_KEY\": \"sk-ant-...\", \"RETRIEVAL_BUDGET_USD\": \"10\" }\n    }\n  }\n}\n```\n\nThen: *\"Load examples/rag_golden.jsonl as 'space', run faithfulness, label it v1.\"*\n\n## Deploy as HTTP (reach it from anywhere)\n\n```bash\nexport RETRIEVAL_TOKEN=$(openssl rand -hex 24)   # required for a public endpoint\nexport ANTHROPIC_API_KEY=sk-ant-...\nexport RETRIEVAL_BUDGET_USD=20\npython app.py        # serves $PORT (default 8000); MCP at /mcp, health at /healthz\n```\n\nDeploy to **Railway** (or any host): the included `Dockerfile` / `Procfile`\nwork as-is. Set `ANTHROPIC_API_KEY`, `RETRIEVAL_TOKEN`, `RETRIEVAL_BUDGET_USD`\nin the host env. Clients connect to `https://<host>/mcp` with header\n`Authorization: Bearer <token>` — add it as a custom connector in\nclaude.ai / Claude Desktop, or point Agent Builder / CI at it. State (golden\nsets, run history, spend) lives server-side, so it persists across machines.\n\n## See it on a real RAG pipeline (demo)\n\n`demo/rag_demo.py` builds a tiny end-to-end RAG over a small *labeled* dataset\n(`demo/labeled.json` + `demo/corpus.json`): it retrieves with BM25, computes\n`recall@k` against the gold passages (deterministic — the retriever's score),\ngenerates an answer, then scores faithfulness (the generator's score). One answer\nis deliberately hallucinated so you watch the two failure modes separate.\n\n```bash\npython demo/rag_demo.py            # offline, no key needed\npython demo/rag_demo.py --real     # real generation + RetriEval judge (needs ANTHROPIC_API_KEY)\n```\n\nIt also writes `demo/generated_goldenset.jsonl` — load *that* into the MCP\n(`load_golden_set` → `run_eval`) for the judge-scored version. This is the bridge:\nyour pipeline emits predictions, the dataset supplies the labels, and RetriEval\nscores retriever and generator independently.\n\n## Connecting to a RAG pipeline\n\n- **Offline (default):** export your pipeline's retrieved context + answer into a\n  golden set and score it — RetriEval never touches your pipeline.\n- **Live:** add a `query_rag(question)` tool that calls your RAG endpoint or your\n  vector store (Chroma / Supabase pgvector), captures context + answer, and scores\n  in one shot.\n\n## Judge backend\n\n```bash\nexport RETRIEVAL_JUDGE_BACKEND=anthropic          # default\nexport RETRIEVAL_JUDGE_MODEL=claude-sonnet-4-6\n# or local, free:\nexport RETRIEVAL_JUDGE_BACKEND=ollama\nexport RETRIEVAL_JUDGE_MODEL=deepseek-r1:70b\n```\n\n## Tools\n\n| Tool | Purpose |\n|------|---------|\n| `list_metrics` | built-in + authored metrics |\n| `load_golden_set(name, source, fmt)` | name a set for reuse (self-host only — shared and lost on restart) |\n| `list_golden_sets` | what's loaded |\n| `author_metric(name, criteria, examples)` | plain language → a scorer |\n| `run_eval(metrics, cases, golden_set, threshold, outputs, label, limit)` | score a set; pass `cases` inline (JSON/JSONL/CSV/TSV/path/URL) — nothing stored |\n| `show_run_cases(run_id, offset, limit, metric)` | page the rest |\n| `evaluate_case(...)` | one-off score |\n| `ground_against_url(url, output, question)` | check an output's *consistency* with a web page (no labels — consistency, not correctness) |\n| `list_runs(golden_set, last_n)` | saved runs |\n| `plot_metric_trend / plot_run / compare_runs` | inline charts |\n| `get_budget / reset_budget` | spend cap status / reset |\n\n---\n\n## License\n\nApache License 2.0. Built by Hanns Carrillo.\n",
  "bytes": 9302,
  "sha": "d9e05cb1534c56d1708e8da95adc753776afaabce36ddf1bb391becae251f67b",
  "repo_slug": "hcarrillo001/retrieval-mcp",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_hcarrillo001_retrieval_mcp_432d43b7/readme"
}