{
  "markdown": "<div align=\"right\">\n\n**English** · [繁體中文](README.zh-TW.md)\n\n</div>\n\n# OpenWiki Source Anchoring\n\nAn evidence-first experiment on making AI-generated code documentation easier to verify.\n\nThe repository asks a narrow question:\n\n> When generated documentation is forced to cite source files with verbatim quotes, which observed changes come from the verification work, and which come from the visible citation markers?\n\nThis is a **research artifact with a working harness**, not a finished product or a universal benchmark for hiring agents.\n\n## Ten-second result\n\n- Retrofitting source verification improved QA PASS from **7/30 to 12/30** in one public run.\n- Removing the visible markers from that same generated content left aggregate PASS at **12/30** in that run, but it also removed quote content and changed other outcomes.\n- This equality is **not evidence of equivalence**. It is one stochastic observation with `n = 30` and no repeats.\n- The mechanical gate verifies **lexical validity**: the path exists and the quote occurs in the file. It does not prove that the quote supports the surrounding claim.\n\n## Public results\n\n| Arm | Procedure | Anchors | Anchored / claim blocks | Anchor rate | QA PASS | PASS + PARTIAL |\n|---|---|---:|---:|---:|---:|---:|\n| A | Official baseline pipeline | 0 | 0/156 | 0% | 7/30 | 17/30 |\n| B | Retrofit verification with gate | 312 | 152/152 | 100% | 12/30 | 20/30 |\n| Bs | Arm B with citation spans removed | 0 | 0/154 | 0% | 12/30 | 17/30 |\n| C | Fresh authoring, no gate — one generation | 590 | 64/155 | 41.3% | 14/30 | 26/30 |\n| D | Fresh authoring with gate — a different generation | 1,053 | 119/119 | 100% | Not measured | Not measured |\n\nRates exclude `nonofficial/`, which is delivered with every arm but was not generated by the pipeline under test. The full target-dependent audit is recomputed against [`repo-snapshot/`](repo-snapshot/) by [`harness/src/audit_arms.ts`](harness/src/audit_arms.ts).\n\n### Measurement correction: the denominator moved\n\nC and D are not one variable apart. They are independent cold-start generations, share only **10 page paths**, created substantially different page sets, and used target copies that differed in two source files. The observed `41.3% → 100%` jump also moves both sides of the fraction:\n\n```text\nC = 64 / 155\nD = 119 / 119\n```\n\nHolding C's denominator fixed, D is `119/155 = 76.8%`. About **23.2 percentage points**, roughly **40% of the headline gap**, come from fewer blocks being counted as claims, not from additional anchored blocks. D did not become narrower: its measured pages contain more words, name more source files, and retain the same entrypoint coverage. What fell is claim density, from **5.93 to 4.35 claim blocks per 1,000 words**.\n\nThe detailed remeasurement, marker-stripping caveat, cost boundary, and alternative second metric are in [`docs/DENOMINATOR_AND_CONFOUNDING_REVIEW.md`](docs/DENOMINATOR_AND_CONFOUNDING_REVIEW.md).\n\n## What the evidence supports\n\n**Supported observations**\n\n- The A-to-B result is directionally favorable and worth replication.\n- B and Bs produced the same aggregate PASS count in one run, while individual verdicts and other aggregate outcomes differed.\n- B and D reached 100% anchor rate on the claim denominators those arms produced.\n- Anchor rate behaved as a process metric, not a reader-quality metric, in these runs.\n- Attempting to anchor statements exposed contradictions that the original pipeline checks did not test.\n- The vendored target and pinned arm baseline now let CI re-audit the published arms rather than merely check that their directories exist.\n\n**Not established**\n\n- That citation markers have zero causal effect.\n- That the C-to-D difference is caused by the gate; page selection, authoring session, and the metric denominator also changed.\n- That `100%` anchor rate means semantic or evidentiary correctness.\n- That the published token differences identify the cost of verification or the gate.\n- That anchor rate is positively or negatively correlated with reader quality from four single-run arms.\n- That the results generalize to organic repositories, other models, or repeated generations.\n\nThe next causal design is a factorial comparison of retrofit versus fresh authoring and gate versus no gate. See [`experiments/factorial-v1/PROTOCOL.md`](experiments/factorial-v1/PROTOCOL.md) and [`docs/NEXT_EXPERIMENT.md`](docs/NEXT_EXPERIMENT.md).\n\n## Reusable OpenWiki evaluation contract\n\nThe reusable evaluation layer compares multiple OpenWiki document trees against exact frozen source snapshots and measures repository understanding rather than prose polish or anchor count alone.\n\n- [OpenWiki Evaluation Contract User Guide](docs/OPENWIKI_EVALUATION_CONTRACT_USER_GUIDE.md)\n- [Traditional Chinese guide / 繁體中文使用說明](docs/OPENWIKI_EVALUATION_CONTRACT_USER_GUIDE.zh-TW.md)\n- [`evaluation/README.md`](evaluation/README.md): methodology and scorecard\n- [`evaluation/manifest.example.json`](evaluation/manifest.example.json): machine-readable contract template\n- [`evaluation/prompts/OPENWIKI_EVALUATION_PROMPTS.md`](evaluation/prompts/OPENWIKI_EVALUATION_PROMPTS.md): isolated task-author, answerer, executor, and judge prompts\n\nThe guide includes the recommended workspace tree, Mermaid directory/data-flow diagrams, manifest setup, validation commands, role isolation, result storage, CI integration, reuse patterns, and release-ready checks.\n\nQuick contract verification:\n\n```sh\nsh evaluation/selftest.sh\ncp evaluation/manifest.example.json evaluation/manifest.local.json\nbun run evaluation/src/validate_manifest.mjs \\\n  evaluation/manifest.local.json \\\n  --root . \\\n  --check-paths\n```\n\nThe deterministic validator checks the declared contract and filesystem boundaries. Model-backed QA, navigation, change-impact, and engineering-task runs still require separate processes or sandboxes; prompt instructions alone are not an access boundary.\n\n## Run the verifier\n\nRequirements:\n\n- Bun **1.3.13**, pinned in [`.bun-version`](.bun-version)\n- POSIX shell\n\n```sh\nsh harness/selftest.sh\nbun run harness/src/audit_arms.ts\nbun run harness/src/check_published_arms.ts\nsh reproduction/recompute.sh\n```\n\nExpected results include:\n\n```text\nselftest: PASS(...)\nPASS: all published arms reproduce their pinned target-dependent measurements.\nPASS: target-independent arm figures and denominators match the published data.\nreproduction: PASS (protocol, fixtures, and receipts match)\n```\n\nAudit one wiki directly:\n\n```sh\nbun run harness/src/audit_wiki.ts \\\n  wiki/arm-d-gate-driven \\\n  repo-snapshot \\\n  --exclude nonofficial\n```\n\nRun the alternative denominator-resistant second reading:\n\n```sh\nbun run harness/src/audit_anchor_invariant.ts \\\n  wiki/arm-c-generated repo-snapshot --exclude nonofficial\n```\n\nCI runs the public self-test, full arm audit, denominator/density check, invariant-metric parity checks, and adversarial negative controls on a clean Ubuntu runner with pinned action revisions and Bun version.\n\n## What the gate proves\n\nAn anchor has this form:\n\n```text\n(src: scripts/git_gate.py `lineage manifest must be staged`)\n```\n\nThe harness checks:\n\n1. the anchor syntax is well formed;\n2. the path stays inside the target repository;\n3. the referenced file exists;\n4. the quoted substring occurs verbatim in that file;\n5. the measured claim, coverage, and verifiable-share thresholds pass.\n\nThe harness does **not** perform claim segmentation, claim-to-anchor entailment, or semantic adjudication. Use the following terms consistently:\n\n| Term | Meaning |\n|---|---|\n| Path validity | The referenced file resolves inside the target repository |\n| Lexical validity | The quote exists verbatim in the referenced file |\n| Semantic support | The evidence supports the surrounding claim |\n| Human adjudication | Independent reviewers agree on the claim and evidence |\n\nSee [`METRICS.md`](METRICS.md) and [`EXPERIMENT_REVIEW.md`](EXPERIMENT_REVIEW.md).\n\n## Reproducibility status\n\n| Classification | Status |\n|---|---|\n| Available | Yes |\n| Licensed | Yes, MIT |\n| Harness functional | Tested in CI by `harness/selftest.sh` |\n| Published arm audit | Recomputed in CI against the vendored target by `audit_arms.ts` |\n| Deterministic bundle | Recomputed by `reproduction/recompute.sh` |\n| Reusable | Partially; the verifier, target snapshot, fixtures, and execution infrastructure are public |\n| Generation reproducible | No; the original authoring pipeline and exact model execution are not fully pinned |\n| Adjudication reproducible | Partially; verdict files are public, but the full model execution environment is not pinned |\n| Independently reproduced | Not yet |\n\nThe exact limits are documented in [`METHOD.md`](METHOD.md), [`THRESHOLDS.md`](THRESHOLDS.md), and [`REPRODUCE.md`](REPRODUCE.md).\n\n## Portfolio evidence\n\nThis repository separates claims about the project from the evidence used to evaluate them:\n\n- [`PROJECT_EVIDENCE.yaml`](PROJECT_EVIDENCE.yaml): machine-readable claim-to-evidence manifest\n- [`AUTHORSHIP.md`](AUTHORSHIP.md): human decisions and agent-assisted work\n- [`REVIEWER_GUIDE.md`](REVIEWER_GUIDE.md): ten-minute independent review path\n- [`docs/AGENT_REVIEW_PROMPT.md`](docs/AGENT_REVIEW_PROMPT.md): reusable multi-role review protocol\n\nThe intended evaluation chain is:\n\n```text\nskill claim → repository evidence → verification command → observed result → limitation\n```\n\nStars, commit volume, and generated prose are not treated as primary evidence.\n\n## Repository map\n\n| Path | Purpose |\n|---|---|\n| [`METHOD.md`](METHOD.md) | Experimental procedure and provenance limits |\n| [`FINDINGS.md`](FINDINGS.md) | Contradiction inventory and adjudication notes |\n| [`docs/DENOMINATOR_AND_CONFOUNDING_REVIEW.md`](docs/DENOMINATOR_AND_CONFOUNDING_REVIEW.md) | Remeasurement that supersedes gate-only attribution |\n| [`docs/OPENWIKI_EVALUATION_CONTRACT_USER_GUIDE.md`](docs/OPENWIKI_EVALUATION_CONTRACT_USER_GUIDE.md) | Executable contract setup, directory structure, data-flow diagrams, isolation, and reuse guide |\n| [`evaluation/`](evaluation/) | Multi-OpenWiki manifest, prompts, result schema, analysis plan, and adversarial self-test |\n| [`STAGES.md`](STAGES.md) | Measurement failures and corrections made during the work |\n| [`THRESHOLDS.md`](THRESHOLDS.md) | Mechanical thresholds and their provenance |\n| [`harness/`](harness/) | Auditor, retry loop, fixtures, and self-test |\n| [`repo-snapshot/`](repo-snapshot/) | Desensitized snapshot used to audit the published arms |\n| [`qa/`](qa/) | Question bank, splits, and per-question verdicts |\n| [`data/`](data/) | Audit receipts and derived comparisons |\n| [`wiki/`](wiki/) | Published outputs for arms A, B, Bs, C, and D |\n\n## Contributing and security\n\nBefore opening a change, read:\n\n- [`CONTRIBUTING.md`](CONTRIBUTING.md)\n- [`SECURITY.md`](SECURITY.md)\n- [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)\n- [`GOVERNANCE.md`](GOVERNANCE.md)\n\nA contribution should include a falsifiable claim, a verification command, an expected result, and a negative control when applicable.\n\n## Primary limitations\n\n- `n = 30`, one generation and one judge run per public comparison.\n- No repeated production generations, equivalence margin result, or inter-rater agreement estimate.\n- The target repository is synthetic.\n- The original authoring model and complete execution environment were not exactly pinned.\n- Only 22 of 53 author-reported corrections received blind adjudication; the inventory is a floor rather than a guaranteed exhaustive count.\n- The historical threshold-freezing commit is outside this public repository.\n- The legacy claim denominator is heuristic and can move with parser rules and authoring structure.\n- The alternative invariant metric is a second reading with its own open gaming channels, not a replacement truth metric.\n- Lexical validity is not semantic correctness.\n\nTreat the reported effects as **directional evidence and an invitation to reproduce**, not a settled causal result.\n",
  "bytes": 11962,
  "sha": "0de49b3f9ec2676d9db9c20a2ea5dfb8dcbd1c62b403a3244d8b4b2eb9d536e8",
  "repo_slug": "ed3c/openwiki-source-anchoring",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_ed3c_openwiki_source_anchoring_wiki_arm__6c22ca3e/readme"
}