Files
Bundle OKF 0.1 · 10 conceitos · ed3c/openwiki-source-anchoring
Open source Repository Open in the app JSON README (API)
About
# Files
- [Quickstart](quickstart.md) - Agent entry point for agent-skills-repo — a map of the five planes, a change-area routing table from intent to owning files, tests and minimal validation, and the tracked backlog.
# Directories
- [architecture](architecture/)
- [governance](governance/)
- [lifecycle](lifecycle/)
- [nonofficial](nonofficial/)
- [operations](operations/)
- [skill-assets](skill-assets/)
- [terminal-operator](terminal-operator/)
- [testing](testing/)
- [validation](validation/)
Details
- Kind
- OKF bundles
- Topic
- Developer tools
- Publisher
- ed3c
- Origin
- okf_github
- Category
- dados
- Version
- 0.1
- Open pull requests
- 3
- Last push
- 2026-08-18T10:21:34Z
- Repository state
- ativo
- Language
- TypeScript
- License
- Apache-2.0
- Added
- 2026-09-08 09:05:43
- Updated
- 2026-09-08 09:05:43
- Origin id
ed3c/openwiki-source-anchoring:wiki/arm-b-stripped/index.md
README
<div align="right"> **English** · [繁體中文](README.zh-TW.md) </div> # OpenWiki Source Anchoring An evidence-first experiment on making AI-generated code documentation easier to verify. The repository asks a narrow question: > When generated documentation is forced to cite source files with verbatim quotes, which observed changes come from the verification work, and which come from the visible citation markers? This is a **research artifact with a working harness**, not a finished product or a universal benchmark for hiring agents. ## Ten-second result - Retrofitting source verification improved QA PASS from **7/30 to 12/30** in one public run. - Removing the visible markers from that same generated content left aggregate PASS at **12/30** in that run, but it also removed quote content and changed other outcomes. - This equality is **not evidence of equivalence**. It is one stochastic observation with `n = 30` and no repeats. - The mechanical gate verifies **lexical validity**: the path exists and the quote occurs in the file. It does not prove that the quote supports the surrounding claim. ## Public results | Arm | Procedure | Anchors | Anchored / claim blocks | Anchor rate | QA PASS | PASS + PARTIAL | |---|---|---:|---:|---:|---:|---:| | A | Official baseline pipeline | 0 | 0/156 | 0% | 7/30 | 17/30 | | B | Retrofit verification with gate | 312 | 152/152 | 100% | 12/30 | 20/30 | | Bs | Arm B with citation spans removed | 0 | 0/154 | 0% | 12/30 | 17/30 | | C | Fresh authoring, no gate — one generation | 590 | 64/155 | 41.3% | 14/30 | 26/30 | | D | Fresh authoring with gate — a different generation | 1,053 | 119/119 | 100% | Not measured | Not measured | Rates exclude `nonofficial/`, which is delivered with every arm but was not generated by the pipeline under test. The full target-dependent audit is recomputed against [`repo-snapshot/`](repo-snapshot/) by [`harness/src/audit_arms.ts`](harness/src/audit_arms.ts). ### Measurement correction: the denominator moved C and D are not one variable apart. They are independent cold-start generations, share only **10 page paths**, created substantially different page sets, and used target copies that differed in two source files. The observed `41.3% → 100%` jump also moves both sides of the fraction: ```text C = 64 / 155 D = 119 / 119 ``` Holding C's denominator fixed, D is `119/155 = 76.8%`. About **23.2 percentage points**, roughly **40% of the headline gap**, come from fewer blocks being counted as claims, not from additional anchored blocks. D did not become narrower: its measured pages contain more words, name more source files, and retain the same entrypoint coverage. What fell is claim density, from **5.93 to 4.35 claim blocks per 1,000 words**. The detailed remeasurement, marker-stripping caveat, cost boundary, and alternative second metric are in [`docs/DENOMINATOR_AND_CONFOUNDING_REVIEW.md`](docs/DENOMINATOR_AND_CONFOUNDING_REVIEW.md). ## What the evidence supports **Supported observations** - The A-to-B result is directionally favorable and worth replication. - B and Bs produced the same aggregate PASS count in one run, while individual verdicts and other aggregate outcomes differed. - B and D reached 100% anchor rate on the claim denominators those arms produced. - Anchor rate behaved as a process metric, not a reader-quality metric, in these runs. - Attempting to anchor statements exposed contradictions that the original pipeline checks did not test. - The vendored target and pinned arm baseline now let CI re-audit the published arms rather than merely check that their directories exist. **Not established** - That citation markers have zero causal effect. - That the C-to-D difference is caused by the gate; page selection, authoring session, and the metric denominator also changed. - That `100%` anchor rate means semantic or evidentiary correctness. - That the published token differences identify the cost of verification or the gate. - That anchor rate is positively or negatively correlated with reader quality from four single-run arms. - That the results generalize to organic repositories, other models, or repeated generations. The next causal design is a factorial comparison of retrofit versus fresh authoring and gate versus no gate. See [`experiments/factorial-v1/PROTOCOL.md`](experiments/factorial-v1/PROTOCOL.md) and [`docs/NEXT_EXPERIMENT.md`](docs/NEXT_EXPERIMENT.md). ## Reusable OpenWiki evaluation contract The reusable evaluation layer compares multiple OpenWiki document trees against exact frozen source snapshots and measures repository understanding rather than prose polish or anchor count alone. - [OpenWiki Evaluation Contract User Guide](docs/OPENWIKI_EVALUATION_CONTRACT_USER_GUIDE.md) - [Traditional Chinese guide / 繁體中文使用說明](docs/OPENWIKI_EVALUATION_CONTRACT_USER_GUIDE.zh-TW.md) - [`evaluation/README.md`](evaluation/README.md): methodology and scorecard - [`evaluation/manifest.example.json`](evaluation/manifest.example.json): machine-readable contract template - [`evaluation/prompts/OPENWIKI_EVALUATION_PROMPTS.md`](evaluation/prompts/OPENWIKI_EVALUATION_PROMPTS.md): isolated task-author, answerer, executor, and judge prompts The guide includes the recommended workspace tree, Mermaid directory/data-flow diagrams, manifest setup, validation commands, role isolation, result storage, CI integration, reuse patterns, and release-ready checks. Quick contract verification: ```sh sh evaluation/selftest.sh cp evaluation/manifest.example.json evaluation/manifest.local.json bun run evaluation/src/validate_manifest.mjs \ evaluation/manifest.local.json \ --root . \ --check-paths ``` The deterministic validator checks the declared contract and filesystem boundaries. Model-backed QA, navigation, change-impact, and engineering-task runs still require separate processes or sandboxes; prompt instructions alone are not an access boundary. ## Run the verifier Requirements: - Bun **1.3.13**, pinned in [`.bun-version`](.bun-version) - POSIX shell ```sh sh harness/selftest.sh bun run harness/src/audit_arms.ts bun run harness/src/check_published_arms.ts sh reproduction/recompute.sh ``` Expected results include: ```text selftest: PASS(...) PASS: all published arms reproduce their pinned target-dependent measurements. PASS: target-independent arm figures and denominators match the published data. reproduction: PASS (protocol, fixtures, and receipts match) ``` Audit one wiki directly: ```sh bun run harness/src/audit_wiki.ts \ wiki/arm-d-gate-driven \ repo-snapshot \ --exclude nonofficial ``` Run the alternative denominator-resistant second reading: ```sh bun run harness/src/audit_anchor_invariant.ts \ wiki/arm-c-generated repo-snapshot --exclude nonofficial ``` CI runs the public self-test, full arm audit, denominator/density check, invariant-metric parity checks, and adversarial negative controls on a clean Ubuntu runner with pinned action revisions and Bun version. ## What the gate proves An anchor has this form: ```text (src: scripts/git_gate.py `lineage manifest must be staged`) ``` The harness checks: 1. the anchor syntax is well formed; 2. the path stays inside the target repository; 3. the referenced file exists; 4. the quoted substring occurs verbatim in that file; 5. the measured claim, coverage, and verifiable-share thresholds pass. The harness does **not** perform claim segmentation, claim-to-anchor entailment, or semantic adjudication. Use the following terms consistently: | Term | Meaning | |---|---| | Path validity | The referenced file resolves inside the target repository | | Lexical validity | The quote exists verbatim in the referenced file | | Semantic support | The evidence supports the surrounding claim | | Human adjudication | Independent reviewers agree on the claim and evidence | See [`METRICS.md`](METRICS.md) and [`EXPERIMENT_REVIEW.md`](EXPERIMENT_REVIEW.md). ## Reproducibility status | Classification | Status | |---|---| | Available | Yes | | Licensed | Yes, MIT | | Harness functional | Tested in CI by `harness/selftest.sh` | | Published arm audit | Recomputed in CI against the vendored target by `audit_arms.ts` | | Deterministic bundle | Recomputed by `reproduction/recompute.sh` | | Reusable | Partially; the verifier, target snapshot, fixtures, and execution infrastructure are public | | Generation reproducible | No; the original authoring pipeline and exact model execution are not fully pinned | | Adjudication reproducible | Partially; verdict files are public, but the full model execution environment is not pinned | | Independently reproduced | Not yet | The exact limits are documented in [`METHOD.md`](METHOD.md), [`THRESHOLDS.md`](THRESHOLDS.md), and [`REPRODUCE.md`](REPRODUCE.md). ## Portfolio evidence This repository separates claims about the project from the evidence used to evaluate them: - [`PROJECT_EVIDENCE.yaml`](PROJECT_EVIDENCE.yaml): machine-readable claim-to-evidence manifest - [`AUTHORSHIP.md`](AUTHORSHIP.md): human decisions and agent-assisted work - [`REVIEWER_GUIDE.md`](REVIEWER_GUIDE.md): ten-minute independent review path - [`docs/AGENT_REVIEW_PROMPT.md`](docs/AGENT_REVIEW_PROMPT.md): reusable multi-role review protocol The intended evaluation chain is: ```text skill claim → repository evidence → verification command → observed result → limitation ``` Stars, commit volume, and generated prose are not treated as primary evidence. ## Repository map | Path | Purpose | |---|---| | [`METHOD.md`](METHOD.md) | Experimental procedure and provenance limits | | [`FINDINGS.md`](FINDINGS.md) | Contradiction inventory and adjudication notes | | [`docs/DENOMINATOR_AND_CONFOUNDING_REVIEW.md`](docs/DENOMINATOR_AND_CONFOUNDING_REVIEW.md) | Remeasurement that supersedes gate-only attribution | | [`docs/OPENWIKI_EVALUATION_CONTRACT_USER_GUIDE.md`](docs/OPENWIKI_EVALUATION_CONTRACT_USER_GUIDE.md) | Executable contract setup, directory structure, data-flow diagrams, isolation, and reuse guide | | [`evaluation/`](evaluation/) | Multi-OpenWiki manifest, prompts, result schema, analysis plan, and adversarial self-test | | [`STAGES.md`](STAGES.md) | Measurement failures and corrections made during the work | | [`THRESHOLDS.md`](THRESHOLDS.md) | Mechanical thresholds and their provenance | | [`harness/`](harness/) | Auditor, retry loop, fixtures, and self-test | | [`repo-snapshot/`](repo-snapshot/) | Desensitized snapshot used to audit the published arms | | [`qa/`](qa/) | Question bank, splits, and per-question verdicts | | [`data/`](data/) | Audit receipts and derived comparisons | | [`wiki/`](wiki/) | Published outputs for arms A, B, Bs, C, and D | ## Contributing and security Before opening a change, read: - [`CONTRIBUTING.md`](CONTRIBUTING.md) - [`SECURITY.md`](SECURITY.md) - [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md) - [`GOVERNANCE.md`](GOVERNANCE.md) A contribution should include a falsifiable claim, a verification command, an expected result, and a negative control when applicable. ## Primary limitations - `n = 30`, one generation and one judge run per public comparison. - No repeated production generations, equivalence margin result, or inter-rater agreement estimate. - The target repository is synthetic. - The original authoring model and complete execution environment were not exactly pinned. - Only 22 of 53 author-reported corrections received blind adjudication; the inventory is a floor rather than a guaranteed exhaustive count. - The historical threshold-freezing commit is outside this public repository. - The legacy claim denominator is heuristic and can move with parser rules and authoring structure. - The alternative invariant metric is a second reading with its own open gaming channels, not a replacement truth metric. - Lexical validity is not semantic correctness. Treat the reported effects as **directional evidence and an invitation to reproduce**, not a settled causal result.