Does retrieval augmentation reduce hallucination?
Bundle OKF 0.2 · 5 conceitos · lavallee/flip-examples
Open source Repository Open in the app JSON README (API)
About
# Does retrieval augmentation reduce hallucination?
* [References](references/) - 3 captured sources with custody and grading
* [Claims](claims/) - 5 claims with status and citations
* [Questions](questions/) - 1 question, 1 open
* [Sessions](sessions/) - 1 work session
* [Update Log](log.md) - 4 logged events, newest first
Details
- Kind
- OKF bundles
- Topic
- No topic detected
- Publisher
- lavallee
- Origin
- okf_github
- Category
- dados
- Version
- 0.2
- Last push
- 2026-08-20T20:24:45Z
- Repository state
- ativo
- Language
- HTML
- License
- MIT
- Added
- 2026-09-08 09:05:43
- Updated
- 2026-09-08 09:05:43
- Origin id
lavallee/flip-examples:rag-hallucination-lit/index.md
README
# flip-examples Real, browsable [flip](https://github.com/lavallee/flip) notebooks — research corpora with custody, grading, and corroboration discipline. Each is a conformant OKF v0.2 knowledge bundle: plain markdown + YAML on disk, readable here on GitHub, in any editor, or as an Obsidian vault. **No human typed the flip commands in these notebooks.** That's the point. Each was built by an AI agent inside an ordinary working conversation: the human asked a question and directed follow-ups the way they normally would; flip is the system that guided the agent to capture sources before citing them, record judgments, gate claims behind a corroboration bar, and log its own sessions. What you're browsing is the durable record that conversation left behind — reusable, remixable, and auditable in a way a chat transcript or a one-shot research PDF is not. There was no conversation at all behind the last entry, [`autonomous-research-loop`](#the-big-one--a-case-study-not-a-notebook) — a case study of what flip's record looks like after 43 unattended sessions over three days. Start with the small notebooks; finish there. **Browse them rendered:** [lavallee.github.io/flip-examples](https://lavallee.github.io/flip-examples/) — every page generated from the notebook itself. ## nj-schools — "did NJ school enrollment actually dip?" The whole notebook came from an exchange like this: > **Human:** People keep saying NJ school enrollment dipped in the > pandemic. Did it? Did it come back? > > **Agent:** *captures four NJ DOE fall-enrollment files (hashed at > capture), grades them, computes statewide totals two independent ways, > and answers:* The dip was real but modest — down 0.98% in fall 2020 — > and it fully recovered by 2023-24. The story nobody quotes: enrollment > has fallen 1.63% since 2023-24, a bigger decline than the pandemic dip. > Three claims verified by recomputation; one question answered; a new one > opened — what's driving the recent decline? The findings now live at [njschooldata.fyi/reports/statewide-enrollment/](https://njschooldata.fyi/reports/statewide-enrollment/) — a public page that cites the upstream NJ DOE files and links back to this notebook as its provenance trail. The notebook is canonical; the page is a render. What that left on disk, in [`nj-schools/`](nj-schools/): - [`notebook.md`](nj-schools/notebook.md) — the tip, hypotheses with named falsifiers (one survived weakened, one falsified, one emerged), gaps. - [`references/`](nj-schools/references/) — five graded sources; the four NJ DOE workbooks are grade-A originals with their raw bytes under [`sources/raw/`](nj-schools/sources/raw/) and SHA-256 fixity in the capture ledger. - [`claims/`](nj-schools/claims/) — three load-bearing claims, each footnote-cited to sources and carrying a recomputation verification event. flip refuses `verified` status until the bar is met. - [`questions/`](nj-schools/questions/), [`decisions/`](nj-schools/decisions/), [`sessions/`](nj-schools/sessions/), [`log.md`](nj-schools/log.md) — the answered and open questions, the era-selection decision, the agent's session record, and the work log (including two file oddities the state won't tell you about: a stale intro sheet, and silent revision of a published ZIP). ## rag-hallucination-lit — "review the literature on retrieval and hallucination" The first notebook built with an **outcome kind**: `flip new rag-hallucination-lit --kind "literature review"` — the plain-language phrase resolves to the `lit-review` kind, whose collection contract the doctor enforces from day one. What that discipline produced, in [`rag-hallucination-lit/`](rag-hallucination-lit/): - [`criteria.md`](rag-hallucination-lit/criteria.md) — inclusion/exclusion **frozen before the first search** (the contract's unrecoverable-by-construction entry, done prospectively). - [`search-log.md`](rag-hallucination-lit/search-log.md) and [`flow.md`](rag-hallucination-lit/flow.md) — the denominator: 2,600+ identified, 31 examined, 7 advanced, 4 excluded with typed reasons, 3 included. The canonical Shuster 2021 paper is excluded **on license alone** (this notebook redistributes its captured PDFs, so only CC-licensed papers can be included) — recorded as negative evidence, not erased. - [`review.md`](rag-hallucination-lit/review.md) — edition e1: the effect is consistent (retrieval improved every reported metric, including a 2.7B model beating vanilla GPT-3 175B on long-tail QA), the construct is not (three papers, three different operationalizations of "hallucination"), and retrieval hurt on ~10% of questions. - Every included source: hashed CC-BY PDF, support tuple (independent · measured · base-defined), screening decision and extraction fields in frontmatter. The synthesis claim is verified through the corroboration bar at 3; the claim ledger also shows ten burned claim ids from the agent's own YAML mistakes — ids are never reused, and the record keeps the stumbles. ## nj-enrollment-forward — "what should we watch?" The companion to nj-schools, built with the `forward-set` kind (flip 0.13's Forecast class): the backward notebook proved what enrollment *did*; this one commits to what the same watched surface will show next. Three dated forecasts with probabilities, confidence, annulment conditions, and resolver ladders — including one that resolves by re-fetching the state's ZIP and comparing bytes against nj-schools' captured hash. The [naive baseline](nj-enrollment-forward/baseline.md) is declared before anything resolves; the record starts honestly at zero. Claims carry grades, never probabilities; forecasts carry probabilities, never grades — and `flip doctor` enforces it. Also featured: [flip's own website notebook](https://github.com/lavallee/flip/tree/main/website/notebook) — the notebook backing every claim on the flip site, including a superseded claim kept on the record when OKF moved from v0.1 to v0.2. ## okf-origins — "where did OKF come from, and is there a roadmap?" > **Human:** let's use flip to track the origins of OKF and what precedents it > builds on, what critiques / gaps currently exist, and whether there's any > evidence of a roadmap or other planning. > > **Agent:** *traces the public import history, captures the declared > predecessor and current spec, reads six independently authored gap > proposals, and checks bounded roadmap surfaces:* OKF explicitly formalizes > Karpathy's LLM-wiki pattern, but its public Google repository begins at a > snapshot that did not preserve the upstream history. Current pressure > clusters around identity, routing, relationship semantics, retrieval, and > composition. No formal roadmap appeared in the checked files or milestones; > spec deferrals and active proposals still show planning without a schedule. What to look at in [`okf-origins/`](okf-origins/): - [`drafts/v1/draft.md`](okf-origins/drafts/v1/draft.md) — the finished reader-facing report: clear lineage, incomplete origin custody; real planning, no visible schedule. - [`TRANSCRIPT.md`](okf-origins/TRANSCRIPT.md) — the human supplies one reporting assignment; the agent's research operations stay underneath the conversation instead of becoming a CLI tutorial. - [`analysis/findings.md`](okf-origins/analysis/findings.md) — direct design precedent, implementation substrate, and second-order Memex lineage kept separate. - [`claims/`](okf-origins/claims/) — eleven load-bearing verified claims: document-subject claims gated by severe attribution tests, a six-source gap synthesis, and a bounded roadmap null with a severe scope test. - [`references/`](okf-origins/references/) and [`sources/`](okf-origins/sources/) — seventeen research sources plus the captured transcript. The unavailable pre-import repository is recorded as `PRIMARY-LOST`, not silently dropped or treated as evidence. - [`questions/`](okf-origins/questions/) — all four assignment questions answered, with reopen conditions for a recovered upstream history, resolved proposals, or a newly visible roadmap surface. ## ten-thousand-steps — "where does 10,000 steps actually come from?" > **Human:** Is 10,000 steps a day actually an evidence-based target? > > **Agent:** *captures the origin trail and the cohort evidence, and takes > the claim apart:* The number is the trade name of a 1965 Japanese > pedometer — manpo-kei, "10,000-steps meter" — not a clinical threshold. > In device-measured cohorts, the mortality benefit plateaus below it: > ~7,500 steps in older women (Lee 2019), 6,000–8,000 for adults 60+ > (Paluch 2022). The folk claim stays on the record — held by the popular > press, rejected by this notebook, superseded by what the evidence > supports. What to look at: a belief kept as data with its holder (`stance holding --holder popular-press` beside the notebook's `rejecting`), a claim *about* a document carrying a severe attribution test instead of a corroboration count, a derivation chain (`--derives-from`), and a rival declared before the supersession — flip refuses a hollow supersede. ## retraction-watch — "have any of these five classics been retracted?" > **Human:** Pick five famous, heavily-cited papers. Any of them > retracted or corrected? > > **Agent:** *captures Crossref's update metadata and the publisher pages > for each:* Watson & Crick 1953, Shannon 1948, Lowry 1951, and LeCun > 2015 — clean, and each "clean" is an absence claim naming exactly the > surfaces checked, armed to reopen if a Crossref update relation ever > appears. Séralini 2012 is the contrast: retracted in 2013, republished > in 2014, and the republication's metadata carries no relation back — a > one-directional trail, which is itself a finding. What to look at: honest nulls as first-class claims (`--absent-from named_surfaces --surface …` — the null's weight IS its coverage), reopen triggers armed on settled questions, zero-yield probes recorded with their cause, and anti-bot challenge pages kept in custody as evidence of what refused to be fetched. ## ev-charger-watch — "is public EV charging getting more reliable?" > **Human:** Did US public EV-charger reliability actually improve? > > **Agent:** *works the question rather than answering it once:* J.D. > Power's failed-visit series says yes — 20% → 19% → 14% across > 2023–2025 — but that answers a *narrower* question than asked > (satisfaction visits, not measured uptime). Port counts answer an > *adjacent* one. The measured national trend is unresolved on purpose: > federal uptime reporting flows into a database the public can't read. > The answer is banded, three reopen triggers are armed, and the refresh > is written as a commission contract before anyone runs it. What to look at: the question journey end-to-end — evidence notes with scope verdicts (`--answers narrower|adjacent|as-worded`), a zero-yield probe with its cause tag, a re-pose `--sharpened scope`, a dormant sub-question with a review date, and a `pursuit`-kind question plan whose dated revisions show the route changing as evidence arrived. --- # The big one — a case study, not a notebook Everything above is a notebook you can clone and run `flip doctor` against. This last entry is different, and it is the reason the set is worth reading twice. ## [autonomous-research-loop](autonomous-research-loop/) — "what does this look like after 43 unattended sessions?" Every notebook above was built in a conversation, in one sitting or a few, and closed at 11–45 files. **[`autonomous-research-loop/`](autonomous-research-loop/)** is a case study of two **private** notebooks that were not: they were built by an unattended research loop that ran for just under 67 hours across 43 sessions, with no person present, sessions starting two minutes after the last one ended at four in the morning. | | practitioner notebook | `retraction-watch` (largest above) | | --- | ---: | ---: | | Captured sources | 380 | 19 | | Claims | 82 | 8 | | Questions | 47 (0 open) | 6 (1 open) | | Work sessions | 43 | 1 | | Markdown files | 664 | 45 | | On disk | 390 MB | 2.6 MB | Those notebooks cannot ship — their own manifests say `visibility: private` — so the case study works the way a case study should: [twelve verbatim excerpts](autonomous-research-loop/excerpts/) from the real pages, every redaction marked where it occurs and [explained in full](autonomous-research-loop/SANITIZATION.md), and every count checkable against the file listing it came from. What only this scale can show: - **A [session cadence](autonomous-research-loop/cadence.md) no human produces.** All 43 sessions with real start/end timestamps. Median gap between sessions: 16 minutes. On the last day, 22 sessions from 00:01 to 20:18, with 2–5 minutes between them for the whole afternoon. - **A question that [reopened](autonomous-research-loop/excerpts/question-Q11-reopened.md).** Answered, reopened 50 minutes later because an audit of a paper the notebook already held failed recomputation, re-answered 40 seconds after that. The first answer stays on the page, with the reason for the reopen between them. - **Claims [deliberately left unconfirmed](autonomous-research-loop/excerpts/claim-C18-unconfirmed.md).** Two of 82, both `load_bearing: true`, both named by id in the handoff so nobody mistakes them for unfinished work. One is the best anecdote in the corpus and stays at `unconfirmed` because its source records `base_defined: false`. - **An [honest null](autonomous-research-loop/excerpts/claim-C15-bounded-null.md).** The counterpart notebook has 8 of 15 questions still open; what it verified instead is that across nine held evidence fragments, *zero* supply what the question needs — `absence: scope: corpus`, with a probe aimed at the temptation to universalize it. - **What [`verified` actually costs](autonomous-research-loop/excerpts/claim-C22-kubestellar.md):** a hundred lines of adversarial test frontmatter guarding one sentence that is half concession. It also documents an [attribution correction](autonomous-research-loop/SANITIZATION.md#r1--attribution) — nearly every event in these notebooks is attributed to a human who was asleep — which turns out to be a finding about flip's `by:` field rather than a redaction note. ## Rights The seven browsable notebooks ship their captured bytes, so every capture in them is redistributable: NJ DOE enrollment files are New Jersey public records published for reuse; the njschooldata.fyi explainer is the maintainer's own work. The newer notebooks capture public/government pages, open-access papers (PMC/CC), public API responses, and dated archive snapshots of publisher pages; anti-bot shells kept as custody are the refusals themselves. `autonomous-research-loop/` is the exception and says so: it is a case study of private notebooks, it **cites captures it does not redistribute**, and its sources are arXiv and conference preprints, peer-reviewed proceedings, GitHub API records for public repositories, public practitioner surveys, and public forum testimony cited by thread rather than by poster. Named third-party projects in it are real public open-source projects, evidenced from their own public artifacts. Notebook prose and structure: MIT, like flip itself. ## Exploring ```bash uv tool install flip-notebook cd nj-schools flip show # open questions, claims needing work, recent activity flip doctor # audit custody, grading, and the verification bar ``` Or skip the CLI entirely — every entity is one markdown page with its metadata in frontmatter. Start at [`nj-schools/index.md`](nj-schools/index.md).