{
  "markdown": "# Does Google's Open Knowledge Format improve RAG?\n\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.21778673.svg)](https://doi.org/10.5281/zenodo.21778673)\n\nA controlled study on one large regulatory document. Code, data, raw model\noutputs, and the paper.\n\n**Short answer: no.** Writing the corpus in OKF and following its links did not\nimprove retrieval or answer quality. The improvement OKF appears to give turns\nout to depend entirely on what you compare it against.\n\nPaper: **[PDF](okf_trial_data/paper/PAPER.pdf)** · **[Markdown](okf_trial_data/paper/PAPER.md)**\n\n## The three findings\n\n**1. The apparent OKF win is an artifact.** A lexical OKF retriever scored 91.1%\npage-hit against 65.8% for the vector-database baseline. But the baseline's\nencoder (`all-MiniLM-L6-v2`, 256-token limit) could only read a median of 64% of\neach passage — 80.9% of passages are longer than its input window. Plain BM25\nover the same text, using **no OKF at all**, scored 97.5%.\n\n| Retrieval arm | Uses OKF | Page hit |\n|---|---|---:|\n| Dense, 256-token limit (frozen baseline) | no | 65.8% |\n| Dense, 8192-token limit | no | 86.1% |\n| **Plain BM25 over raw chunks** | **no** | **97.5%** |\n| BM25 over OKF concepts + links | yes | 91.1% |\n\n**2. OKF built the way it's meant to be used is worse, not better.** We rebuilt\nthe corpus as 1,011 topic concepts — one per topic, nested 6 levels, with parent,\nchild and sibling links — from the PDF's own 1,006-entry outline, keeping 99.9% of\nthe document's words verbatim. At a matched 2,200-token context budget it scored\n75.9% against 88.6% for plain chunk retrieval.\n\nWhy, measured rather than guessed:\n- **Links carried no new evidence.** 104 of 588 context units did arrive by\n  traversal, but they supplied **zero** answer pages the direct matches hadn't\n  already found.\n- **Coarse topics become unreachable.** 12 topics exceed the entire context\n  budget, so their text is in the bundle but can never be retrieved. The Table of\n  Contents is a single 27,768-token concept, which alone loses the four questions\n  asking what page a section begins on.\n\n**3. The A/B depends on the baseline.** Adding OKF to a *weak* vector-only\npipeline improves page recall by +0.226 (95% CI +0.133 to +0.323, Holm p<0.001).\nAdding it to a *strong* pipeline — BM25 + dense + reranking — gives nothing, and\nanswer quality does not improve on any measure:\n\n| Arm | Correctness | Citation quality | Page hit |\n|---|---:|---:|---:|\n| **A: hybrid RAG** | **4.620** | **4.680** | **86.1%** |\n| B: + OKF topics | 4.532 | 4.429 | 84.8% |\n| B: + OKF chain | 4.506 | 4.583 | 82.3% |\n\nSeven of eight paired dimension estimates are negative across two independent OKF\nvariants. Correctness is a null; citation quality (topics) and completeness\n(chain) are measurably worse, intervals excluding zero.\n\n## What OKF *is* good for\n\nThe study verifies real properties that are not retrieval properties: the bundle\nis portable, reviewable in version control, addressed by stable identifiers,\ncarries provenance and page metadata that survive being handed to another\nconsumer, rebuilds to a matching content digest, and its passages are\nbyte-identical to the rows in the vector database. This matches OKF's own\nmaterials, which describe \"OKF **plus** RAG\" as complementary layers — not OKF\ninstead of RAG.\n\n## Layout\n\n```\nokf_trial_data/\n  paper/PAPER.pdf                   typeset paper, 9 pages with figures\n  paper/PAPER.md                    same content as Markdown\n  paper/render_paper.py             regenerates both; no number is hand-typed\n  paper/make_figures.py             regenerates the three figures\n  scripts/                          build, run, judge, analyse\n  src/okf_trial_data/               OKF producer, consumers, evaluator\n  data/benchmark_questions.json     93 questions with page-level answer keys\n  data/okf_bundles/wmp_all_v0_2/    chunk-preserving bundle (1,837 concepts)\n  data/okf_bundles/pge_topics_v0_2/ topic-structured bundle (1,011 concepts)\n  protocol/                         pre-results protocol + full deviations log\n  results/                          every raw model output and analysis summary\n  tests/                            52 offline tests\neval_harness/                       minimal retrieval/generation dependency\n```\n\n## Reproducing\n\n```bash\ncd okf_trial_data\npython3.11 -m venv .venv\n.venv/bin/pip install -r ../eval_harness/requirements.txt\n.venv/bin/pip install -e '.[dev,analysis,paper]'\n../eval_harness/scripts/start_pgvector.sh    # if present; else see eval_config.yaml\n\n# free, no model calls\n./scripts/with_experiment_env.sh .venv/bin/python scripts/build_topic_okf_bundle.py\n./scripts/with_experiment_env.sh .venv/bin/python scripts/measure_embedding_truncation.py\n./scripts/with_experiment_env.sh .venv/bin/python scripts/run_retrieval_diagnostics.py\n./scripts/with_experiment_env.sh .venv/bin/python scripts/run_topic_retrieval_comparison.py\n\n# billable, about $8 on Amazon Bedrock\n./scripts/with_experiment_env.sh .venv/bin/python scripts/run_hybrid_ab_experiment.py --stage all\n\n.venv/bin/python paper/make_figures.py\n.venv/bin/python paper/render_paper.py\n```\n\n## Caveats worth reading before citing\n\n- **One document, one utility, 93 questions.** Nothing here generalises by itself.\n- **Two producers, both verbatim-text.** A producer that re-authored passages, or\n  added semantic rather than structural links, might behave differently.\n- **The frozen dense baseline was weak** (truncating encoder). Reported as run\n  rather than quietly swapped; an untruncated arm is added as a diagnostic.\n- **Exploratory status.** The diagnostic arms, topic producer and A/B were all\n  specified after retrieval results were seen. Each carries its own multiplicity\n  family. These are attribution and estimation, not preregistered tests.\n- **The A/B design is ours, not Google's.** Querying a vector store and an OKF\n  bundle in parallel and merging is our construction; the OKF materials describe\n  OKF as an authored source ingested *into* the retrieval stack.\n- **A halted run.** An earlier five-pipeline matrix was stopped at 1,254 of 1,395\n  cells when the study was redirected; its answer-quality endpoints are\n  unreported. Raw records are published and labelled. See the deviations log.\n- **Blinded human validation of the answer keys is outstanding.**\n\n## Licence and citation\n\nCode: MIT ([okf_trial_data/LICENSE](okf_trial_data/LICENSE)). Data and\nannotations: CC BY 4.0, with source-document terms explained in\n[okf_trial_data/DATA_LICENSE.md](okf_trial_data/DATA_LICENSE.md). Cite via\n[okf_trial_data/CITATION.cff](okf_trial_data/CITATION.cff).\n\nKumar Abhinav, AiDash — ORCID [0009-0009-1839-841X](https://orcid.org/0009-0009-1839-841X)\n\nArchived on Zenodo. Cite the concept DOI, which always resolves to the newest\nversion; cite the version DOI to pin an exact snapshot.\n\n- Concept DOI (all versions): [10.5281/zenodo.21778673](https://doi.org/10.5281/zenodo.21778673)\n- Version DOI (v1.0.0): [10.5281/zenodo.21778674](https://doi.org/10.5281/zenodo.21778674)\n\n```bibtex\n@software{abhinav_okf_rag_2026,\n  author    = {Abhinav, Kumar},\n  title     = {Does Google's Open Knowledge Format Improve Retrieval-Augmented\n               Generation? A Controlled Study on One Regulatory Document},\n  year      = {2026},\n  publisher = {Zenodo},\n  doi       = {10.5281/zenodo.21778673},\n  url       = {https://github.com/Abhinav0905/okf-vs-rag-evaluation}\n}\n```\n",
  "bytes": 7430,
  "sha": "b7b47d7b1e12bb86f0d18cc31448eb2afa43c8a36ceda800f7dcb441917ca565",
  "repo_slug": "abhinav0905/okf-vs-rag-evaluation",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_abhinav0905_okf_vs_rag_evaluation_okf_tr_3fcd5d7a/readme"
}