{
  "markdown": "<p align=\"center\"><img src=\"logo.png\" width=\"120\" alt=\"logo\" /></p>\n\n# skill-ab-eval\n\n**Prove what actually works — for any task, in any domain.**\n\nTwo questions, one evaluation harness, real evidence:\n\n1. **Skill axis** — does loading a `SKILL.md` change the agent's behavior, or is it\n   dead weight in the context window? (`with_skill` vs `without_skill`)\n2. **Harness axis** — for *this* task, which CLI agent is best? (`claude` vs\n   `codex` vs `gemini` vs `agy` vs an OpenAI-compatible API)\n\nRun them separately or crossed, on a skill's evals or on **an ad-hoc task you\ntype**. A judge grades every output against your assertions, repeated over trials.\nYou get a **skill-lift table** and a **harness leaderboard** — not vibes.\n\n```\n        one task (a skill's eval, or one you type)\n                        │\n     matrix: harness × {with_skill, without_skill}     ← each cell = a FRESH context\n   claude·codex·gemini·agy·openai   ×   with / without\n                        │\n                  judge (any harness), × N trials\n                        ▼\n     skill-lift per harness   +   harness leaderboard\n```\n\nIt's an agent-native take on\n[`agent-skills-eval`](https://github.com/darkrishabh/agent-skills-eval) (same\n`evals.json` and with/without model), wired to the multi-CLI delegation pattern of\n[`cc-agent-call`](https://github.com/cskwork/cc-agent-call): where cc-agent-call\n**routes** work to the best CLI, this **measures** which CLI is best.\n\n## Why\n\nYou ship a `SKILL.md` and assume the agent is better. You pick a CLI and assume\nit's the right one. The hard part is *proving* either. This is the proof — and\n**fresh, isolated context per cell** is the trick: the only variables are the\nharness and whether the skill is loaded. Reuse a context and the skill leaks into\nthe baseline, silently invalidating the result.\n\n## Install\n\n### Claude Code (and `~/.claude/skills` agents)\n\n```bash\ngit clone https://github.com/cskwork/skill-ab-eval ~/.claude/skills/skill-ab-eval\n```\n\nThen `/skill-ab-eval`, or ask *\"does my X skill actually do anything?\"* /\n*\"which CLI is best at Y?\"* and it triggers.\n\n### As a standalone CLI\n\n```bash\ngit clone https://github.com/cskwork/skill-ab-eval && cd skill-ab-eval\nexport PATH=\"$PWD/bin:$PATH\"     # optional\nskill-ab-eval runners            # list usable harnesses\n```\n\nNeeds `bash` + `python3` and at least one of `claude`, `codex`, `gemini`, `agy`,\nor `OPENAI_API_KEY`. No pip install — stdlib only.\n\n## Quickstart\n\n```bash\n# compare CLIs on a task you give (no skill, any domain)\nskill-ab-eval task \"Explain async/await to a junior in 5 bullets.\" \\\n  --runners claude,codex,gemini --judge claude --trials 2\n\n# does a skill help — and on which harness?\nskill-ab-eval task \"Write a git commit message for the staged diff.\" \\\n  --skill examples/conventional-commit \\\n  --assert \"Subject line is 50 characters or fewer.\" \\\n  --assert \"Ends with a 'Refs:' footer.\" \\\n  --runners claude,codex --judge claude\n\n# a whole evals suite across harnesses\nskill-ab-eval run examples/conventional-commit --runners claude,gemini --judge claude\n```\n\nInside Claude Code you can also run the **agent-native** mode (fresh subagents, no\nAPI keys) — see `SKILL.md`.\n\n## What you get\n\n`skill-ab-eval-workspace/<name>/iteration-1/`:\n\n- `report.md` — skill-lift table + harness leaderboard\n- `results.json` — every cell, metric, judge, trials\n- `<eval-id>/<runner>/<side>/trial-N/{answer.md,judge.json}` — full receipts\n\n```\n## Skill lift by harness\n| harness | with | without | lift | verdict        |\n|---------|------|---------|------|----------------|\n| claude  | 1.00 | 0.50    | +0.50| clear positive |\n| codex   | 1.00 | 0.62    | +0.38| clear positive |\n\n## Harness leaderboard (with skill)\n| rank | harness | score |\n|------|---------|-------|\n| 1    | claude  | 1.00  |\n| 2    | codex   | 1.00  |\n```\n\n## Reading the verdict\n\n| lift (pass-rate / score delta) | verdict |\n|--------------------------------|---------|\n| ≥ +0.20                        | **clear positive** — the skill helps |\n| +0.05 … +0.20                  | marginal — directional |\n| −0.05 … +0.05                  | **no measurable effect** — dead weight |\n| ≤ −0.05                        | **negative** — the skill hurts |\n\nA few trials is *directional*, not statistically significant — raise `--trials`.\n\"No effect\" is a real result: the skill is already-default, never triggers, or too\nvague to act on. Per-harness differences are real too — a skill can lift Claude and\ndo nothing for Codex.\n\n## Runners (the harness axis)\n\n| runner | CLI / API | auth |\n|--------|-----------|------|\n| `claude` | Claude Code (`claude -p`) | claude login / `ANTHROPIC_API_KEY` |\n| `codex` | OpenAI Codex (`codex exec`) | codex login / `OPENAI_API_KEY` |\n| `gemini` | Gemini CLI (`gemini -p`) | gemini login |\n| `agy` | Antigravity (`agy -p`) | `agy install` + Google sign-in |\n| `openai` | OpenAI HTTP API | `OPENAI_API_KEY` |\n\nAdd your own harness by dropping a `runners/<name>.sh` (prompt on stdin → text on\nstdout). See [`runners/README.md`](runners/README.md) and\n[`reference/harnesses.md`](reference/harnesses.md).\n\n## Eval format\n\n`evals/evals.json`, compatible with the\n[agentskills.io spec](https://agentskills.io/specification). Assertions are atomic\nbinary claims — they *are* the score. See\n[`reference/eval-schema.md`](reference/eval-schema.md).\n\n## Worked example (real, committed results)\n\n[`examples/conventional-commit`](examples/conventional-commit) enforces non-default\ncommit rules (≤50-char subject, mandatory `Refs:` footer). Two real runs are\ncommitted under\n[`examples/conventional-commit/result/`](examples/conventional-commit/result) — and\nthey disagree, on purpose:\n\n- **agent-native** (fresh subagents, no global rules): skill shows **+0.29 lift,\n  clear positive** — and the harness honestly notes `with_skill` still slipped once.\n- **cli-orchestrator** (`claude -p` + `gemini -p`): on claude the skill shows\n  **0.00 lift** because the user's global `CLAUDE.md` already mandates the same\n  conventions — the \"dead weight\" signal. gemini returned a non-answer in the\n  headless nested setup, which the leaderboard surfaces.\n\nSame skill, two harnesses, opposite verdicts — both correct. \"Does my skill work?\"\nhas no answer without naming the harness. See\n[`result/README.md`](examples/conventional-commit/result/README.md).\n\n## Relationship to cc-agent-call\n\n[cc-agent-call](https://github.com/cskwork/cc-agent-call) delegates work between\nCLIs in production; skill-ab-eval *measures* which CLI to prefer. Same shell-out\npattern (`runners/` mirrors cc-agent-call's adapters), opposite direction. Install\nboth: evaluate here, route there.\n\n## Credits\n\nInspired by [`darkrishabh/agent-skills-eval`](https://github.com/darkrishabh/agent-skills-eval)\nand the [agentskills.io](https://agentskills.io) standard; harness axis built on\n[`cskwork/cc-agent-call`](https://github.com/cskwork/cc-agent-call). MIT licensed.\n",
  "bytes": 6932,
  "sha": "5c57970fafd8b57eade3408c2f5c17dfa73c6fff0a09eb2bfe280b5b2c71999c",
  "repo_slug": "cskwork/skill-ab-eval",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/plg_cskwork_skill_ab_eval_e7be00e9/readme"
}