{
  "markdown": "# AI task benchmark pipeline\n\nTakes a Python repository, pins its dependencies, containerizes it, runs and\nextends its test suite, builds a machine-readable knowledge layer, and mines\n10 validated benchmark tasks for AI coding agents.\n\n```\n./run.sh https://github.com/mahmoud/glom\n```\n\nThree stages run in sequence: **hygiene** (pin, containerize, baseline, test-gen,\nlint), **knowledge** (repo graph, coverage indexes, OKF bundle), and **tasks**\n(excision + history funnels, build, validate, instruct, select). Each step is\nresumable. All target-code execution happens inside a throwaway Docker container\nwith no network access. LLM decisions are cached by content hash, so reruns on\nan unchanged repo cost zero tokens.\n\n\n## Results on glom\n\n| Metric | Value |\n|---|---|\n| Wall clock (full `--fresh` run) | ~13 min |\n| LLM tokens | 779,614 total (big 762,657 / small 16,957) |\n| Baseline tests | 202 passing before test generation, 0 quarantined |\n| Generated tests kept | 4 functions across 2 modules (14/16 mutants killed) |\n| Suite after test-gen | 239 tests passing (verify-twice identical) |\n| Graph | 378 nodes, 4,612 edges; verification precision 1.0 all edge types |\n| OKF | 164 pages total; of the 150 function pages, 106 stable / 44 draft; conformant |\n| Tasks built | 14 (5 excision, 9 history); 13 VALID |\n| Selected 10 | 4 excision + 6 history, 4 distinct modules |\n| Difficulty spread | easy 5, medium 4, hard 1 |\n\n202 is the suite as glom shipped it; `baseline.json` is re-recorded after test\ngeneration, so the committed baseline holds the 239. Of the 164 OKF pages, 150\nare function pages (106 stable / 44 draft); the 11 module pages and the three\nreserved pages (index, repo, log) are outside those counts and stay draft.\n\n\n## Quick start\n\nPrerequisites: Docker (running), Python 3.12, [uv](https://docs.astral.sh/uv/),\nan OpenAI-compatible LLM endpoint.\n\n```bash\n# clone and set up\ngit clone https://github.com/Adithyan777/repo-bench-pipeline.git && cd repo-bench-pipeline\nuv venv --python 3.12 .venv\nuv pip install --python .venv/bin/python -r requirements-dev.txt\ncp .env.example .env          # fill in LLM_BASE_URL + LLM_API_KEY\n\n# full pipeline\n./run.sh https://github.com/mahmoud/glom\n\n# or one stage at a time (resumable)\n./run.sh <repo> --stage hygiene\n./run.sh <repo> --stage knowledge\n./run.sh <repo> --stage tasks\n```\n\nModels default to open-source Kimi-K2.6 (BIG tier) and DeepSeek-V4-Flash\n(SMALL tier) via Baseten. Override with `LLM_MODEL_BIG` / `LLM_MODEL_SMALL`\nin `.env`.\n\n\n## What happens per stage\n\n**Hygiene** (detect, pin, dockerfile, compose, build, baseline, testgen, lint):\ndetects packaging style and Python version, pins all dependencies with hashes\nvia `uv pip compile`, writes a digest-pinned Dockerfile, builds a `bench-<repo>`\nimage, runs the baseline suite in-container, generates tests for under-covered\nfunctions (mutation-gated), and runs ruff lint/format (reverted if any test\nregresses). Outputs: `output/<repo>/repo/` (transformed clone with labeled\ncommits), `output/<repo>/hygiene/*.json`.\n\n**Knowledge** (symbol_index, indexes, graph, verify, okf): builds a\ndeterministic repo graph from AST analysis (nodes for modules/classes/functions,\nedges for imports/calls/contains/inherits/tested_by, each with file+line\nevidence), coverage indexes and test map, history index, hotspots, and an\nOKF v0.2 knowledge bundle with LLM-authored function contracts. Outputs:\n`output/<repo>/knowledge/repo_graph.json`, `.okf/`, verification files.\n\n**Tasks** (excision_funnel, build_excision, history_funnel, build_history,\nvalidate, instruct, manifest, select): funnels candidate functions and commits,\nbuilds task folders (`input/`, `solution/`, `verifier/`, evidence), validates\neach (fail-before with right-reason check, pass-after, determinism, collateral),\nwrites LLM-authored instructions (leak-gated, reviewer-checked), labels\ndifficulty, and selects exactly 10 under hard quotas. Outputs:\n`tasks/<repo>/<task_id>/`, `tasks/<repo>/tasks.json`, root `tasks.json`,\n`output/<repo>/REPORT.md`.\n\n\n## CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--stage hygiene\\|knowledge\\|tasks\\|all` | Run one stage (default: all) |\n| `--fresh` | Ignore all cached state, rerun everything |\n| `--force STEP` | Rerun a specific step (repeatable) |\n| `--set section.key=value` | Override a config value (repeatable) |\n| `--no-testgen` | Skip test generation |\n| `--no-lint` | Skip lint/format |\n| `--no-report-draft` | Skip the LLM narrative draft in REPORT.md |\n| `--verify-twice` | Run the test suite a second time after hygiene |\n| `--excision-hard` | Strip docstrings from excised functions |\n| `--verifier-visibility visible\\|hidden` | Solver sees verifier tests or not |\n| `--min-failing-tests N` | Minimum failing tests for a valid fail-before |\n| `--llm-cache` | Enable prompt-to-response disk cache |\n| `--prune-images` | Remove dangling images with this pipeline's label |\n| `--quiet` | Stage-level progress only |\n\n\n## Operational knobs (via `--set`)\n\nThese are the most commonly tuned values. The full reference with defaults,\nrationale, and glom observations is in [docs/configuration.md](docs/configuration.md).\n\n| Key | Default | What it controls |\n|---|---|---|\n| `testgen.agent_max_turns` | 12 | Turns per test-gen agent run |\n| `testgen.max_agent_runs_per_repo` | 10 | Total agent runs (write + retry) across all modules |\n| `testgen.top_k_modules` | 5 | Modules ranked for test generation |\n| `history.build_target` | 10 | History tasks to build (headroom for selection) |\n| `history.shortlist_size` | 20 | History candidates shortlisted after classify |\n| `history.max_agent_runs_per_repo` | 6 | Verifier/rewrite agent runs per repo |\n| `history.agent_max_turns` | 12 | Turns per history agent run |\n| `harness.min_failing_tests` | 1 | Minimum failing tests in fail-before |\n| `harness.determinism_runs` | 3 | Repeat count for determinism check |\n| `selection.total_tasks` | 10 | Tasks to select |\n| `selection.min_history` | 4 | Minimum history tasks in the final 10 |\n| `selection.max_excision` | 4 | Maximum excision tasks |\n| `selection.min_distinct_modules` | 4 | Minimum distinct modules across the 10 |\n| `llm.max_tokens_per_repo` | 5,000,000 | Per-repo token budget (abort on exceed) |\n| `okf.max_function_pages` | 150 | Cap on individual function pages in the OKF bundle |\n| `lint.format` | true | Whether ruff format runs alongside ruff check |\n\nExample: `./run.sh <repo> --set testgen.top_k_modules=3 --set history.build_target=15`\n\n\n## Validate a task standalone\n\nEach task folder is self-contained. To re-validate on a fresh machine:\n\n```bash\n# build the image from the task's own Dockerfile\ndocker build -t <image_tag> tasks/<repo>/<task_id>/input\n\n# run the harness (add --json to print the full verdicts, --set to override config)\npython -m pipeline.validate tasks/<repo>/<task_id>\npython -m pipeline.validate tasks/<repo>/<task_id> --json\n```\n\nThe harness runs fail-before (with right-reason classification), pass-after,\ndeterminism (3x by default), and collateral checks. It re-copies the canonical\n`verifier/` into the workdir before each run. Results go to\n`<task>/evidence/verdict.json`; `--json` prints the same verdicts to stdout.\n\n\n## Rebuild the report\n\n`REPORT.md` and `report_summary.json` for a run dir are rebuilt from the\ncommitted artifacts alone -- no container, no LLM call when `--no-draft` is used:\n\n```bash\npython -m pipeline.report glom --no-draft            # tables only, 0 tokens\npython -m pipeline.report glom                       # + one BIG narrative draft\npython -m pipeline.report glom --output-root output  # non-default run root\n```\n\n`<repo>` is a repo URL/path or the `output/<name>` run-dir name. Output goes to\n`output/<repo>/report_summary.json` and `output/<repo>/REPORT.md`; the repo-root\n`REPORT.md` is hand-maintained and never overwritten.\n\n\n## Pipeline tests\n\n```bash\n.venv/bin/python -m pytest              # default: skips slow\n.venv/bin/python -m pytest -m slow      # multi-build container tests\n.venv/bin/ruff check .\n```\n\nTests use real Docker, real uv, real git, and real fixture repos built with\nreproducible history. LLM calls are replayed from committed cassettes (no\nnetwork, no tokens). Last full run: 199 passed, 1 skipped, 3 deselected;\nslow: 3 passed; ruff clean.\n\n\n## Deliverables map\n\n`output/glom/REPORT.md` is the raw generated report; the finished report is\n`REPORT.md` at the repo root.\n\n| Path | Committed | Description |\n|---|---|---|\n| `pipeline/` | yes | Pipeline source |\n| `tests/` | yes | Integration tests, fixtures, cassettes |\n| `docs/` | yes | Architecture, pipeline docs, configuration, decisions, gaps |\n| `transcripts/dev/` | yes | Development methodology, prompts, review rounds |\n| `transcripts/glom-console.log` | yes | Console log of the glom run (wall-clock evidence) |\n| `heldout/` | yes | Held-out runs on toolz and minidump: logs, reports, task folders (see docs/gaps.md section 14) |\n| `transcripts/pipeline/`, `transcripts/agent/` | yes | Per-call LLM transcripts (236) and agent trajectories (12) |\n| `tasks.json` (root) | yes | The final 10 selected tasks |\n| `tasks/glom/<id>/` | yes | The 10 selected task folders |\n| `tasks/glom/tasks.json` | yes | Full manifest of all built tasks |\n| `output/glom/repo/` | yes | Transformed clone, 79 files (working tree only, no nested `.git`) |\n| `output/glom/knowledge/repo_graph.json` | yes | Static knowledge graph |\n| `output/glom/knowledge/.okf/` | yes | OKF v0.2 knowledge bundle |\n| `output/glom/report_summary.json` | yes | Aggregated run data |\n| `output/glom/REPORT.md` | yes | Generated report (tables auto-filled, `AUTHOR` markers) |\n| `REPORT.md` | yes | Assignment report (six sections), hand-finished |\n| `output/glom/hygiene/`, `output/glom/tasks/`, `output/glom/audit/` | no | Step records, caches (regenerable) |\n\n\n## Documentation index\n\n| Document | Contents |\n|---|---|\n| [docs/architecture.md](docs/architecture.md) | Stages, resumability, Docker model, agent loop, LLM client |\n| [docs/pipeline-1-hygiene.md](docs/pipeline-1-hygiene.md) | Hygiene stage: each step, edge cases, glom outcomes |\n| [docs/pipeline-2-knowledge.md](docs/pipeline-2-knowledge.md) | Knowledge stage: graph, indexes, OKF, verification |\n| [docs/pipeline-3-tasks.md](docs/pipeline-3-tasks.md) | Tasks stage: funnels, harness rules, instructions, selection |\n| [docs/configuration.md](docs/configuration.md) | Every config key with default, meaning, and rationale |\n| [docs/decisions.md](docs/decisions.md) | Design decisions with rejected alternatives |\n| [docs/gaps.md](docs/gaps.md) | Known gaps with evidence and next steps |\n| [REPORT.md](REPORT.md) | Assignment report |\n\n\n## Folder READMEs\n\nEach subfolder has its own README describing its files and purpose:\n`pipeline/`, `pipeline/*/`, `tests/`, `tests/fixtures/`, `tests/cassettes/`,\n`scripts/`, `transcripts/`, `tasks/`, `output/`.\n",
  "bytes": 10879,
  "sha": "f561ad33ea4b7dc9ba6c0367fd5bc1a9da822a7737f0ca526fd0fdf2d0329615",
  "repo_slug": "adithyan777/repo-bench-pipeline",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_adithyan777_repo_bench_pipeline_output_g_88b7b5c2/readme"
}