{
  "markdown": "# OpenWiki Bench\n\nAn official evaluation harness for comparing the **agent systems** that generate OpenWiki documentation: model, runtime, tool loop, and recovery behavior. It preserves run artifacts, verifies their source grounding against a pinned repository revision, measures output structure, and produces machine-readable results plus deliberately scoped recommendations.\n\nThe historical pilot dataset is five OpenWiki 0.4.3 runs over [`manikanda-kumar/background-agents`](https://github.com/manikanda-kumar/background-agents/tree/main/openwiki-bench) at `32470cc2`. The planned official benchmark is 3 OpenCode Go models × 3 independent trials × 5 pinned repositories = 45 runs.\n\n## What it measures\n\n- **Navigation and code understanding:** source/package coverage, seed-path coverage, and exact cited evidence.\n- **Reasoning and writing:** six anchored, blind per-page rubric scores from two judges plus disagreement-only tiebreaks.\n- **Correctness:** statement-versus-source probes labelled supported, unsupported, or contradicted, with a Wilson interval.\n- **Agent loops:** completion, deaths/resumes, tool failures, redundant reads, latency, and tokens when telemetry exists.\n- **Economics:** cost per finished page and verified claim; missing historical usage remains `null`.\n- **Reliability:** mean and sample standard deviation across trials and subjects.\n\nThe checked-in historical dataset has one seed and one subject per system. The framework therefore marks it **not publishable** rather than manufacturing statistical confidence.\n\n## Run it\n\nRequires Node 22+ and Git.\n\n```bash\nnpm ci\nnpm run score   # deterministic metrics; clones the pinned subject automatically\nnpm run bench -- prepare --repo /path/to/background-agents\nnpm run report\nnpm test\nnpm run typecheck\n```\n\nTo reuse an existing subject checkout, it must be at the exact recorded SHA:\n\n```bash\nnpm run score -- --repo /path/to/background-agents\n```\n\nOutputs are stable: no timestamps or local paths are written into scores.\n\nAmp orbs run `.agents/setup` to install pinned OpenCode and OpenWiki versions, project\ndependencies, and the user-level OpenWiki MCP integration. Before starting paid trials, run:\n\n```bash\nnpm run preflight:opencode\n```\n\nThe preflight verifies credentials, exact CLI versions, all three OpenCode Go model IDs, and the\nOpenWiki MCP registration without sending a model request. Project secrets may use\n`OPENCODE_GO_API_KEY`; orb login shells safely expose it to OpenCode as `OPENCODE_API_KEY`.\n\n## Planned official matrix\n\nEvery run will use the same OpenCode agent and OpenWiki MCP generation path. Only the OpenCode Go model changes:\n\n- `opencode-go/deepseek-v4-flash`\n- `opencode-go/glm-5.3-flash`\n- `opencode-go/qwen3.8-flash`\n\nThe five subjects are `cloudflare-os`, `smallstep-cli`, `extractthinker`, `celld`, and `pi-desktop`. See [PLAN.md](PLAN.md#31-official-45-run-matrix) for pins and selection rationale.\n\nThe non-interactive OpenCode invocation and telemetry capture are frozen in `systems/*.json`.\nThe runner checks out each pinned subject in isolation, installs the shared brief and ignore policy,\nrecords prompt/ignore/system hashes, validates the exported OpenCode session against the exact\nprovider/model, captures raw and normalized telemetry, and refuses to overwrite a trial.\n\n## Semantic evaluation\n\nThe checked-in pilot judgments used OpenRouter and are preserved as dated provenance. Official evaluation runs blind tasks in fresh, repository-less Amp threads using GPT-5.5 through `deep-classic` as primary, Opus 5 as secondary, and Fable 5 for tiebreaks and correctness probes. Exact resolved model IDs must be recorded; Amp routing may not silently substitute another model.\n\n```bash\n# Prepare opaque, deterministic page and claim tasks\nnpm run bench -- prepare --repo /path/to/background-agents\n\n# After importing Amp results, select only primary/secondary disagreements\nnpm run bench -- judge disagreements \\\n  --primary gpt-5.5 --secondary opus-5 \\\n  --primary-judgments results/evaluation/judgments-gpt-5.5.json \\\n  --secondary-judgments results/evaluation/judgments-opus-5.json\nnpm run bench -- judge aggregate \\\n  --judgments results/evaluation/judgments-gpt-5.5.json,results/evaluation/judgments-opus-5.json,results/evaluation/judgments-fable-5.json \\\n  --judges gpt-5.5,opus-5,fable-5\n\nnpm run bench -- probe aggregate \\\n  --labels results/evaluation/probe-labels-probe-fable-5.json\n```\n\nAggregation rejects missing/duplicate probe labels and duplicate judgments. Judge identities are checked against contestant models. `leaderboard` refuses publication below linear-weighted Cohen's κ = 0.6, with unresolved judgments, without probes, or below three trials × five subjects per system. Exact agreement and unweighted κ remain visible diagnostics.\n\nTelemetry accepts normalized proxy JSON/JSONL and LangSmith exports:\n\n```bash\nnpm run bench -- cost --input traces.jsonl\nnpm run bench -- leaderboard \\\n  --scores results/scores.json \\\n  --judged results/evaluation/judged-scores.json \\\n  --probes results/evaluation/probe-scores.json\nnpm run report\n```\n\n## Layout\n\n```text\nbin/bench.ts                         CLI\nsrc/grounding.ts                     claim evidence verification\nsrc/structure.ts                     deterministic wiki metrics\nsrc/report.ts                        scoped recommendations\nsrc/run.ts                           isolated run + artifact capture\nsrc/judge.ts                         blind rubric judging and agreement\nsrc/probe.ts                         source-only correctness probes\nsrc/cost.ts                          normalized telemetry economics\nsubjects/background-agents.json      pinned subject definition\nsubjects/{cloudflare-os,...}.json    five official pinned subjects\nsystems/README.md                    planned OpenCode Go systems\njudges/README.md                     official Amp judge protocol\nruns/background-agents/*/seed-0/     immutable run artifacts + manifests\nruns-invalid/                         preserved infrastructure-invalid attempts excluded from scoring\nresults/scores.json                   generated machine results\nresults/leaderboard.json              publication-gated cross-run results\nresults/REPORT.md                     generated human report\nresults/COMPARISON-2026-08-29.md      original qualitative study\n```\n\nSee [METHODOLOGY.md](METHODOLOGY.md) for metric definitions and validity limits, and [PLAN.md](PLAN.md) for the remaining phases.\n",
  "bytes": 6453,
  "sha": "d7e985b0ce7b338dcef67fb39904d72f3cd6b799a7c7f5781452fd6193bdc798",
  "repo_slug": "manikanda-kumar/openwiki-bench",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_manikanda_kumar_openwiki_bench_runs_extr_6f047579/readme"
}