{
"markdown": "\n\n# ForgeJudge\n\n**An open, always-on leaderboard and CI gate for autonomous coding agents — every patch runs in a sandbox, every run has a public trace, every regression fails the build.**\n\n[](https://github.com/ahmedEid1/forgejudge/actions/workflows/ci.yml)\n[](https://github.com/ahmedEid1/forgejudge/actions/workflows/gate.yml)\n[](./LICENSE)\n[](https://www.python.org)\n\n**▶ Live leaderboard: [forgejudge.ahmedhobeishy.tech](https://forgejudge.ahmedhobeishy.tech)** · [playground](https://forgejudge.ahmedhobeishy.tech/playground) · [methodology](https://forgejudge.ahmedhobeishy.tech/methodology) · [model swap](https://forgejudge.ahmedhobeishy.tech/model-swap) · [MCP registry](https://registry.modelcontextprotocol.io/v0/servers?search=forgejudge)\n\n
\n\n\n\n\n\n> **Current numbers** (hidden-test = the agent never sees the failing test; $0 free tier; same harness, swap the model; 18 tasks × 3 seeds = 54 runs/model, 162 total):\n>\n> | Model | pass@1 | pass@3 |\n> |---|---|---|\n> | `gpt-oss-120b` | 90.7% | 100% |\n> | `llama-3.3-70b` | 88.9% | 94.4% |\n> | `llama-3.1-8b` | 48.1% | 66.7% |\n>\n> The score rises with the better model while the harness stays fixed (model-swap proof), and `pass@3 > pass@1` shows real run-to-run variance — which is exactly why the CI gate is multi-seed. Every run [deep-links its Langfuse trace](https://forgejudge.ahmedhobeishy.tech).\n\nForgeJudge is the only open-source autonomous software-engineering agent that **proves its quality in public on every commit**: a hand-rolled single-agent solver, a deterministic execution-as-judge harness, an always-on leaderboard with per-run traces, and a CI gate that blocks regressions — all on a **`$0` / self-hostable** stack against a **contamination-resistant, intrinsically-verifiable** golden set.\n\n> **The engineered harness, observability, and gate are the deliverable — not a high resolution rate.** A `$0` free-model agent will score modestly *by design*. We prove value with a **model-swap comparison**: the score rises with a better model while the harness stays fixed.\n\n## How it works\n\n```mermaid\nflowchart TD\n G[\"Golden set · Git-canonical
18 intrinsically-verifiable, mutation-hardened
make-CI-green tasks\"]\n\n subgraph SOLVER[\"Single-agent solver\"]\n direction LR\n L[\"localize
(BM25)\"] --> R[\"repair
(LLM router · critic · syntax edit-gate)\"] --> V[\"validate
(run tests)\"]\n end\n\n G --> SOLVER\n SOLVER --> PATCH[\"unified diff\"]\n SOLVER -. \"every step traced\" .-> TRACE[\"OTel → Langfuse
per-run public trace\"]\n\n PATCH --> H[\"Deterministic harness, in a sandbox
apply test_patch + candidate patch · run F2P / P2P
RESOLVED iff every FAIL_TO_PASS passes AND every PASS_TO_PASS stays green
swebench-equivalent · stricter on skips · cheat-resistant\"]\n\n H --> STORE[\"Run store
Neon + pgvector\"]\n STORE --> LB[\"Leaderboard
pass@1 / pass@3 · cost · tokens · trace link\"]\n H --> GATE[\"Multi-seed CI gate
a PR that lowers the resolution rate fails the build\"]\n\n style G stroke:#3fb950,stroke-width:2px\n style H stroke:#4cc2ff,stroke-width:2px\n style GATE stroke:#f0883e,stroke-width:2px\n```\n\n- **Solver** — a single, phase-structured loop (`localize → repair → validate`), *not* a multi-agent swarm: cheapest, most deterministic, most debuggable. BM25 localization, an LLM router over free tiers, a syntax edit-gate, a cheap critic pre-filter, and a cost/step budget with autosubmit.\n- **Harness** — encodes the SWE-bench `RESOLVED_FULL` rule and is **verified equivalent to `swebench.harness.grading`** on real PASS/FAIL/ERROR/XFAIL outcomes in CI — and *deliberately stricter* on a **skipped** `FAIL_TO_PASS`: swebench 4.1.0 rates a skipped oracle test `RESOLVED_FULL` (a skip is neither success nor failure), so a patch that makes the oracle *skip* rather than run grades as resolved. ForgeJudge counts a skip as not-passed, closing that cheat vector. Patches are also **cheat-resistant**: the canonical test files are restored before grading, so a patch can't neuter the oracle.\n- **Golden set** — 15 purpose-built post-cutoff fixtures + 3 tasks mined from the author's own repos (real commit SHAs, MIT/own license — zero leak/copyleft risk). Each is **mutation-hardened**: a wrong fix to the patched region is caught (16 mutation-hardened at mean score 0.94; 2 inconclusive for regex/string code; **0 weak**).\n- **Sandbox / CI / cron** — GitHub Actions on a public repo does triple duty (ephemeral isolated VM sandbox + regression gate + scheduled sweep) at `$0`.\n- **Observability** — OpenTelemetry GenAI spans (`invoke_agent → retrieval / chat / execute_tool`, `gen_ai.usage.*`, a `gen_ai.evaluation.result` pass/fail verdict) exported to Langfuse Cloud; every run is a clickable trace.\n\n### Two gates, two jobs\n\nThe deterministic **gold-integrity gate** (does the harness itself still work?) is kept separate from the stochastic **regression gate** (did a change make the *agent* meaningfully worse?) — because gold grading is deterministic and must never be averaged with noisy per-seed runs.\n\n```mermaid\nflowchart TD\n PR[\"Pull request / commit\"] --> GG[\"Gold-integrity gate
deterministic · $0 · re-grade all gold patches\"]\n GG -->|\"any gold task unresolved\"| F1[\"fail — the harness broke\"]\n GG -->|\"all gold tasks resolved\"| OK1[\"harness intact\"]\n\n CRON[\"Scheduled multi-seed sweep\"] --> SEEDS[\"run the agent × N seeds
→ one resolution rate per seed\"]\n SEEDS --> RG[\"Regression gate
small-sample CI (Student-t / Wilson)\"]\n BASE[\"baseline_scores.json
per-seed reference\"] --> RG\n RG -->|\"candidate CI upper bound < baseline CI lower bound\"| F2[\"❌ fail — real regression\"]\n RG -->|\"overlapping · equal · improved\"| OK2[\"✓ no regression\"]\n\n style GG stroke:#4cc2ff,stroke-width:2px\n style RG stroke:#f0883e,stroke-width:2px\n```\n\n## Quickstart\n\nPrereq: [`uv`](https://docs.astral.sh/uv/) (Python 3.12 is provisioned for you) — `curl -LsSf https://astral.sh/uv/install.sh | sh`.\n\n```bash\ngit clone https://github.com/ahmedEid1/forgejudge && cd forgejudge\nuv sync # Python 3.12, deps via uv\n\n# Run the deterministic harness self-test (no API key, no network):\nuv run python -m forgejudge.harness.runner_actions --patch-source gold # 18/18 resolved\n\n# Solve a task with a free model and grade it.\n# Needs a (free) Groq key. Either export it, or put it in .env and pass --env-file:\n# export GROQ_API_KEY=... # or\n# cp .env.example .env && edit GROQ_API_KEY # then: uv run --env-file .env python - <<'PY'\nuv run python - <<'PY'\nfrom forgejudge.golden.loader import load_tasks\nfrom forgejudge.agent.solver import solve\nfrom forgejudge.harness.grade import grade\ntask = {t.instance_id: t for t in load_tasks(\"golden/dataset.jsonl\")}[\"fixture-semver-001\"]\nres = solve(task, run_id=\"demo\", budget_usd=0.10, seed=0)\nprint(res.status, \"→ resolved:\", grade(task, res.patch).resolved)\nPY\n```\n\nFast tests: `uv run pytest -m \"not slow\"`. Full golden validation + mutation hardening: `uv run pytest -m slow`. Sweep the leaderboard: `uv run python -m forgejudge.eval.sweep --model groq/llama-3.3-70b-versatile --seeds 0,1,2`. See [`CONTRIBUTING.md`](./CONTRIBUTING.md) for the full pytest marker map and dev workflow.\n\n## Install\n\nWorking on the agent/harness itself? Clone and `uv sync` (above). To consume ForgeJudge as a package:\n\n```bash\n# Library + the `forgejudge` CLI (selftest / mcp / info):\npip install forgejudge\nforgejudge selftest # deterministic harness check — 18/18 resolved, no key\nforgejudge mcp # MCP server over stdio (needs the [mcp] extra)\n\n# Zero-install MCP server (no venv to manage) — for an MCP client config:\nuvx --from \"forgejudge[mcp]\" forgejudge mcp\n```\n\nOptional extras (installed only when you need them):\n\n| Extra | Pulls in | For |\n|---|---|---|\n| `forgejudge[harness]` | `swebench` | the swebench-equivalence grading check |\n| `forgejudge[mcp]` | `fastmcp` | the MCP server (`forgejudge mcp`) |\n| `forgejudge[playground]` | `fastapi`, `uvicorn`, `httpx` | the guarded live playground API |\n\n```bash\npip install \"forgejudge[mcp]\" # one extra\npip install \"forgejudge[harness,mcp]\" # several\n```\n\n`forgejudge selftest` and `forgejudge info` work with the base install — no extras, no API key, no network.\n\n## Six objections, pre-empted\n\n1. **\"Your benchmark is contaminated / cherry-picked.\"** The golden set is freshly authored / post-cutoff, sourced only from the author's own repos + fixtures (no third-party leak surface), and **mutation-hardened** so a wrong patch can't pass. SWE-bench Verified is now widely held contaminated — OpenAI [stopped reporting it](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/) (2026-02); >32% of \"passed\" cases [leaked the solution](https://arxiv.org/abs/2410.06992) and ~31% passed on weak tests. Decontamination here is a documented, tested property — not a footnote.\n2. **\"Thin wrapper around an LLM / a framework.\"** The orchestrator is hand-rolled (no LangChain): the control loop, the sandbox-and-score harness, the cheat-resistant grader, the mutation hardener, the OTel instrumentation, and the multi-seed CI gate are the work.\n3. **\"Your resolution rate is low vs SOTA.\"** SOTA is ~88–94% with premium models and budgets; a `$0` free-model number is modest *on purpose*. The deliverable is the engineered system; the **model-swap comparison** (score rises with a better model, harness fixed) is the proof.\n4. **\"Is it actually autonomous or staged?\"** Every run has a public OpenTelemetry/Langfuse trace and a deterministic, reproducible score. The replay-first playground demos a real solve without exposing cost/abuse surface.\n5. **\"Three agent projects — one-trick pony?\"** One eval methodology — golden set + judge + traces + CI gate — across three domains at rising autonomy (Lumen → Thoth → ForgeJudge).\n6. **Determinism.** temperature=0 does [not guarantee determinism](https://arxiv.org/pdf/2602.07150) (pass@1 varies 2–6pp). The scorer is fully deterministic; the **gate is multi-seed** (fail only when the candidate's CI upper bound is below the baseline's CI lower bound), so flaky single runs don't break the build.\n\n## Repository layout\n\n| Path | What |\n|---|---|\n| `forgejudge/golden/` | golden-set loader, fixture contract, dataset builder, mutation hardener |\n| `forgejudge/harness/` | deterministic `grade()`, cheat-resistant runner, swebench-equivalence check, sandbox executor |\n| `forgejudge/agent/` | `localize → repair → validate` solve loop, critic |\n| `forgejudge/llm/` | role-based LiteLLM router with fallback + cost accounting |\n| `forgejudge/obs/` | OpenTelemetry GenAI tracing → Langfuse / Phoenix |\n| `forgejudge/eval/` | scheduled sweep, multi-seed regression gate, LLM-as-judge + Cohen's κ |\n| `forgejudge/store/` | Neon (Postgres + pgvector) run store + leaderboard query |\n| `golden/dataset.jsonl` | canonical golden set (one `Task` per line) |\n| `.github/workflows/` | `ci`, `eval` (sandbox), `sweep` (cron), `gate` (regression) |\n\n## License\n\n[MIT](./LICENSE) © 2026 Ahmed Hobeishy. Imports and attributes the MIT-licensed [`swebench`](https://github.com/SWE-bench/SWE-bench) grading harness.\n",
"bytes": 11715,
"sha": "5af463d2d3909a2785d8f64c197c913530a06c17b944a4b1a616ae4950840196",
"repo_slug": "ahmedeid1/forgejudge",
"fonte": "repo",
"truncated": false,
"api": "https://agentalog.com/api/listings/mcp_io_github_ahmedeid1_forgejudge_dfdfe3c5/readme"
}