{ "markdown": "

StudyDiff

\n\n

\n Understand why two scientific studies reach different conclusions —
with every claim verified against the source.\n

\n\n

\n \"CI\"\n \"License:\n \"Node\n \"Live\n \"Built\n \"MCP\n \"npm\"\n \"npm\n \"Built\n \"PRs\n

\n\n

\n Live demo ·\n Quick start ·\n How it works ·\n Examples ·\n MCP server ·\n Roadmap ·\n Contributing\n

\n\n

\n \"StudyDiff\n

\n\n---\n\nTwo well-run papers often reach opposite conclusions. Usually the reason isn't that one is wrong — it's a methodological difference (a cell type, a dose, a follow-up window, an analysis choice) that a reader has to dig out of the methods sections by hand. **StudyDiff does that digging.** Give it two studies and it extracts each one's design, surfaces the differences that could explain the disagreement, and — critically — grounds every statement in the source text, so it never invents a finding.\n\nIt is built for a bench scientist deciding which of two conflicting papers to trust before planning an experiment.\n\n## Why it's different\n\nMost \"AI literature\" tools generate a fluent answer and ask you to trust it. StudyDiff inverts that:\n\n- **It shows the evidence, then gets out of the way.** The design differences up top; every value with the verbatim sentence that supports it underneath.\n- **It refuses to guess.** Any field the source doesn't state is shown as *not reported*, never inferred.\n- **It verifies itself.** A deterministic grounding check (no second LLM acting as judge) confirms every extracted value and every explanation is backed by a verbatim quote and traceable numbers. Anything that fails is downgraded *before* it can be used as a reason.\n- **It knows what it can't do**, because it was measured — see below.\n\n## Does it work? A measured answer\n\nStudyDiff used to rank the divergent design dimensions and present the top one as the\nlikely driver of a disagreement. **We built a benchmark to test that, and it doesn't work.**\n\n15 documented contradictions where the literature has since established *why* the papers\ndisagreed — each label carrying its own citation, the set built blind before any accuracy\nnumber existed. Scored against StudyDiff's top-ranked driver:\n\n```\nTop-1 accuracy (strict) 13.3% (95% CI 3.7-37.9%) [2/15]\nBaseline \"always say assay\" 13.3% (95% CI 3.7-37.9%) [2/15]\n → discordant on 0 of 15 cases\nOracle ceiling (reachable) 66.7% (95% CI 41.7-84.8%) [10/15]\nNon-assay-labelled cases 0.0% (95% CI 0.0-22.8%) [0/13]\n```\n\nThe ranking was a fixed prior (`DRIVER_RANK` in `src/compare.mjs`) in which `assay`\noutranks everything. Two papers almost always use somewhat different methods, so `assay`\nalmost always diverges, so it was picked 13 times out of 15 — and the two hits are exactly\nthe two assay-labelled cases. **It is not merely as good as guessing a constant; it is\nbehaviourally identical to it on every case in the set.**\n\nFixing grounding first (Phase 2) removed that excuse. Recovering 20 of 26 false-positive\nrejections **doubled the oracle ceiling from 33% to 67%** — the established cause is now an\navailable candidate in 10 of 15 cases instead of 5 — and top-1 accuracy did not move at\nall. The ranker was handed the right answer five more times and took none of them.\n\n**What changed as a result:** the app no longer nominates a primary driver. It presents the\ndivergent dimensions as an unranked list, because that list is informative (it contains the\nestablished cause 10 times in 15) while the ordering is not. Choosing among them needs\ndomain knowledge the tool doesn't have.\n\n**What still holds:** which dimensions differ, which are identical (*ruled out*), and the\nverbatim sentence behind every value. None of that depends on the ranking.\n\n### Confirmed blind, on a second set\n\nThose 15 cases had by then been read across two phases — failures analysed, spans re-audited —\nso every post-fix figure from them is *development-set* accuracy, not a blind measurement. So we\nbuilt a second set and measured it once.\n\n`eval/cases-heldout.json` is 15 further documented contradictions, curated to a protocol written\nand committed *before any case was selected* ([`eval/HELDOUT-PROTOCOL.md`](eval/HELDOUT-PROTOCOL.md)),\nby a curator kept blind to the dev set's per-case failures, in deliberately different fields:\nmicrobiome, marine ecology, toxicology, psychology, critical care, oncology, infectious disease.\nNo paper and no contradiction is shared with the dev set — `selftest` enforces that mechanically.\n\n```\nTop-1 accuracy (strict) 13.3% (95% CI 3.7-37.9%) [2/15]\nBaseline \"always say assay\" 20.0% (95% CI 7.0-45.2%) [3/15]\n → discordant on 1 of 15 cases\nOracle ceiling (reachable) 73.3% (95% CI 48.0-89.1%) [11/15]\nNon-assay-labelled cases 0.0% (95% CI 0.0-24.3%) [0/12]\n```\n\nOn unseen data the prior scores **below** the constant guess — by exactly one case. The intervals\noverlap almost entirely and the two strategies disagree on 1 of 15, so the honest statement is\nthat it remains indistinguishable from guessing `assay` every time, not that it is worse.\n\nThe line that does not move is the last one. **Across both sets, 25 cases where the established\ncause was something other than `assay`, the prior identified none of them.** And the ceiling here\nis *higher* than on the dev set — 73.3% against 66.7% — so extraction put the right answer in\nfront of the ranker more often, and it was taken no more often. That is the Phase 1–2 conclusion\nreproduced on data the development loop never saw, which is the only way it could have been\nstrengthened.\n\nThe number is reported as-is, and is never pooled with the dev-set number: summing them into an\n\"n=30\" figure would relaunder read data as blind data. Nothing in `src/` was changed on the basis\nof it. The set was fetched once on a single pre-registered arm, but the scoring was not one clean\npass — the first run reported n=14 after one paper failed to fetch, and the full set was scored\nafter retrieving it. Both figures, the sixteen defects an adversarial verification pass found and\ncorrected *before* any scoring, and the reasoning behind each label are recorded in the file's own\nprovenance block rather than summarised away.\n\n```bash\nnpm run eval:heldout # the blind number, offline and free\nnpm run eval:selftest:heldout # set integrity, incl. zero overlap with the dev set\n```\n\nFull method, the pre-registered decisions, and every prediction that turned out wrong:\n[`eval/README.md`](eval/README.md) and [`eval/PHASE2.md`](eval/PHASE2.md). The benchmark\nsets are [`eval/cases.json`](eval/cases.json) (development) and\n[`eval/cases-heldout.json`](eval/cases-heldout.json) (held-out).\n\n```bash\nnpm run eval # offline, free, no API key — regenerates the numbers above\nnpm run eval:selftest # validates the harness maths and set integrity\n```\n\n`eval/cache/` and `eval/cache-heldout/` are committed on purpose. They aren't build output,\nthey're evidence: the published numbers are reproducible from artefacts in the repo rather than\ntaken on faith.\n\n## Quick start\n\nUnder 60 seconds, no API key, no network:\n\n```bash\ngit clone https://github.com/nickjlamb/studydiff && cd studydiff\nnpm install\nnpm run demo # explains a real, famous contradiction\nnpm run demo -- resveratrol-sirt1 # a second worked example\nnpm run demo -- treg-stability # a third: Treg lineage stability\n```\n\nRun the web app:\n\n```bash\ncp .env.example .env # add ANTHROPIC_API_KEY for live comparisons\nnpm run serve # http://localhost:4173\n```\n\nThe built-in examples run on cached data and need no key. To compare live, add your key and use the **PMID / DOI**, **Upload PDF**, or **Paste** inputs.\n\n## How it works\n\n```mermaid\nflowchart LR\n IN[\"Two papers
PMID · DOI · PDF · text\"] --> R[\"Retrieve
PubMed / PMC / PDF\"]\n R --> E[\"Extract
Claude → structured study cards\"]\n E --> V[\"Verify
deterministic grounding\"]\n V -- \"ungrounded → not reported\" --> E\n V --> C[\"Compare
divergent vs. shared design\"]\n C --> X[\"Present
divergent dimensions + ruled out\"]\n```\n\n1. **Retrieve** — a PubMed/PMC client with a full-text-to-abstract fallback that tags how deep it read (`fulltext` / `abstract` / `pasted`); uploaded PDFs are text-extracted server-side.\n2. **Extract** — Claude turns each paper into a fixed **study card** (species, model, assay, dose, timing, endpoint, sample size, statistic, finding, limitations). Every field carries a verbatim supporting quote; absent fields default to *not reported*.\n3. **Verify** — grounding runs **first**: any value whose quote isn't in the source, or whose numbers don't trace, is downgraded to *not reported*. StudyDiff can't cite a fact it hasn't verified.\n4. **Compare** — deterministic: which design dimensions are reported by both papers, which of those diverge, and which are identical. Divergence is a token-overlap test on design dimensions and strict inequality on the conclusion itself, because a conclusion's *polarity* matters where a method list's *content* does.\n5. **Present** — the divergent dimensions **unranked**, each with both papers' values and the verbatim sentence behind them, and the identical ones explicitly *ruled out*. StudyDiff does not nominate one as the cause: [it was measured and it doesn't work](#does-it-work-a-measured-answer).\n\nTwo things hold the guarantee up: the API key never leaves the server, and verification is deterministic — grounding and comparison involve no model judgment at all, so the same verified evidence always yields the same divergent dimensions.\n\n## Why Claude?\n\nStudyDiff's reliability comes from *how* it uses Claude, not just that it does:\n\n- **Structured extraction via tool-use.** Each paper becomes a study card through a forced Claude (Sonnet) tool schema, so every field returns validated and carries a *verbatim supporting quote* — no free-text parsing, no \"mostly-JSON\" failures.\n- **Reasoning that maps claims to evidence.** Claude reads the methods/abstract prose and identifies both the design value and the exact sentence that supports it — the hard part of turning unstructured papers into comparable, auditable cards.\n- **Claude proposes, grounding disposes.** StudyDiff never uses an LLM as the judge. A deterministic check verifies Claude's output against the source and downgrades anything unsupported *before* it's shown. Pairing Claude's tool-use extraction with non-LLM verification is what lets the tool trust its own output and sidesteps the usual LLM-as-evaluator pitfalls.\n- **Built with Claude Code.** The whole app — pipeline, UI, hardening, deploy — was built iteratively with Claude Code in a single agentic loop.\n\n## Examples\n\n| Question | Papers | What StudyDiff finds |\n|---|---|---|\n| Do mouse models mimic human inflammation? | [Seok 2013](https://pubmed.ncbi.nlm.nih.gov/23401516/) vs [Takao & Miyakawa 2015](https://pubmed.ncbi.nlm.nih.gov/25092317/) | Same datasets, opposite conclusions — driven by the **gene-selection strategy**. |\n| Does resveratrol activate SIRT1? | [Howitz 2003](https://pubmed.ncbi.nlm.nih.gov/12939617/) vs [Beher 2009](https://pubmed.ncbi.nlm.nih.gov/19843076/) | An **assay artifact** — the Fluor de Lys peptide substrate vs. native substrates. |\n| Is the Treg lineage stable in vivo? | [Zhou 2009](https://pubmed.ncbi.nlm.nih.gov/19633673/) vs [Rubtsov 2010](https://pubmed.ncbi.nlm.nih.gov/20929851/) | A landmark T-cell controversy — stable vs. unstable, driven by the **fate-mapping method**. |\n\nAll three ship as offline demos (`npm run demo` / `npm run demo -- resveratrol-sirt1` / `npm run demo -- treg-stability`).\n\n## Using it\n\n- **Web app** (`npm run serve`) — examples, PMID/DOI lookup, PDF upload, or paste; streams each pipeline step live, and exports a reproducible **Markdown report** with every value's verbatim supporting sentence.\n- **MCP server** (`npm run mcp`) — lets Claude, or any agent, call the contradiction engine directly. See [MCP server](#mcp-server).\n- **CLI** — `node src/cli.mjs --q \"Does resveratrol activate SIRT1?\" 12939617 19843076`\n- **Deploy** — see [DEPLOY.md](DEPLOY.md) (Railway + custom domain).\n\n## MCP server\n\nStudyDiff's engine is also a [Model Context Protocol](https://modelcontextprotocol.io) server, so an agent can ask *why two papers disagree* as a tool call — the same grounded pipeline, no browser.\n\nPublished to the [official MCP Registry](https://registry.modelcontextprotocol.io) as **`io.github.nickjlamb/studydiff`**, and to npm as [`studydiff-mcp`](https://www.npmjs.com/package/studydiff-mcp).\n\n| Tool | What it does |\n|---|---|\n| `compare_studies(paperA, paperB, question?)` | The full pipeline on two papers. Each paper is `{id}` (PMID or DOI) or `{citation, text}`. Needs `ANTHROPIC_API_KEY`. |\n| `compare_example(example)` | Runs a cached worked example — **no API key, no network**. The quickest way to see the grounded output. |\n| `list_examples()` | Lists the built-in worked examples. |\n\nIt returns the same auditable report the web app exports: the verdict, the divergent design dimensions (unranked) and the identical ones (**ruled out**), every value with the verbatim sentence that supports it, the verification counts, and what evidence would resolve the disagreement. Ungrounded fields come back as *not reported* — never guessed — and the tool never picks a winner, never nominates a primary driver, and never reports a confidence it didn't compute.\n\nAdd it to **Claude Code**:\n\n```bash\nclaude mcp add studydiff -- npx -y studydiff-mcp\n```\n\n…or to **Claude Desktop** (`claude_desktop_config.json`):\n\n```json\n{\n \"mcpServers\": {\n \"studydiff\": {\n \"command\": \"npx\",\n \"args\": [\"-y\", \"studydiff-mcp\"],\n \"env\": { \"ANTHROPIC_API_KEY\": \"sk-ant-...\" }\n }\n }\n}\n```\n\n`ANTHROPIC_API_KEY` is only needed for `compare_studies`. Leave it out and `list_examples` / `compare_example` still work — the cached examples run with no key and no network. (To run from a local clone instead, swap the command for `node /absolute/path/to/studydiff/src/mcp.mjs`.)\n\nThen just ask:\n\n> Use studydiff to compare PMID 19633673 and 20929851 — why do they disagree?\n\nNo key handy? `list_examples`, then `compare_example(\"treg-stability\")` runs entirely offline.\n\n## Project layout\n\n```\nsrc/ncbi.mjs PubMed / PMC retrieval (+ DOI resolution, source-depth tagging)\nsrc/pdf.mjs PDF text extraction (pure JS)\nsrc/extract.mjs Claude tool-use → structured study cards\nsrc/grounding.mjs deterministic verification (OpenGATE)\nsrc/compare.mjs divergence detection (divergent vs. shared design dimensions)\neval/ driver-ranking benchmark: 15 cited contradictions + scorer\nsrc/gaps.mjs bounded \"observed across these papers\"\nsrc/pipeline.mjs orchestration: retrieve → extract → verify → compare\nsrc/report.mjs shared Markdown report (answer, drivers, quotes, verification)\nsrc/server.mjs web server + streaming API (+ rate limiting, caching)\nsrc/mcp.mjs MCP server: compare_studies / compare_example / list_examples\npublic/index.html single-file dashboard UI\nfixtures/ cached real papers for the offline demos\n```\n\n## Roadmap\n\nKeyword search with a results picker, a source viewer that highlights each grounded quote in the original text, batch comparison, and an exportable report. Full list in [ROADMAP.md](ROADMAP.md).\n\n## Contributing\n\nContributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for the setup and the invariants that keep the trust guarantee intact.\n\n## Provenance\n\nStudyDiff was built for Anthropic's [Built with Claude: Life Sciences](https://cerebralvalley.ai/e/built-with-claude-life-sciences) hackathon (Builder track). All application code in this repository was written from scratch during the event. Grounding uses [OpenGATE](https://github.com/nickjlamb/opengate) and PDF extraction uses `unpdf`, both as published dependencies. Retrieval uses public NCBI E-utilities; extraction uses the Claude API.\n\n## License\n\n[MIT](LICENSE) © Nick Lamb\n", "bytes": 17645, "sha": "222f52529d3e0b121805295bf40a518da99cd0674b62d5630b80481c137721e8", "repo_slug": "nickjlamb/studydiff", "fonte": "repo", "truncated": false, "api": "https://agentalog.com/api/listings/mcp_io_github_nickjlamb_studydiff_e422c130/readme" }