{ "markdown": "\"Agent-Eval:\n\n# Agent Evaluation\n\n\n\n\n\n[![PyPI](https://img.shields.io/pypi/v/agent-regress-cli)](https://pypi.org/project/agent-regress-cli/)\n[![npm](https://img.shields.io/npm/v/agent-regress-cli)](https://www.npmjs.com/package/agent-regress-cli)\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)\n[![CI](https://github.com/RudrenduPaul/agent-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/RudrenduPaul/agent-eval/actions/workflows/ci.yml)\n[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/RudrenduPaul/agent-eval/badge)](https://api.securityscorecards.dev/projects/github.com/RudrenduPaul/agent-eval)\n\n**Statistical regression testing for LLM agents: run version A and version B 50x each and get a p-value, an effect size, and a 95% confidence interval on whether behavior actually shifted.**\n\n![agent-eval running the basic-comparison example and reporting a REGRESSED verdict with p-value, Cohen's d, and a 95% confidence interval](https://raw.githubusercontent.com/RudrenduPaul/agent-eval/main/docs/assets/demo-1-comparison.gif)\n\n```bash\npip install agent-regress-cli\n```\n\n(`uv add` and the npm/npx wrapper are covered in [Install](#install) below.)\n\n## The test that every eval framework skips\n\nYou changed a prompt. Your evals still pass.\nBut your agent's tool accuracy dropped from 84% to 70%.\n\nIs that a real regression? Or is it LLM run-to-run noise?\n\nThreshold testing cannot answer that question.\nagent-eval can.\n\nRun your agent 50x on version A, 50x on version B.\nGet a p-value, an effect size, and a 95% confidence interval\non whether behavior actually shifted.\n\n```\np=0.003, Cohen's d=-0.61 -> REGRESSED (deploy blocked)\np=0.410, Cohen's d=0.021 -> STABLE (safe to ship)\n```\n\n> [!NOTE]\n> Promptfoo, one of the most widely used open-source LLM eval frameworks, was [acquired by OpenAI in March 2026](https://techcrunch.com/2026/03/09/openai-acquires-promptfoo-to-secure-its-ai-agents/), staying open source but folding its team into OpenAI's Frontier platform. agent-eval is Apache 2.0-licensed, self-hostable, and has no commercial dependency. The statistical core (Mann-Whitney U, bootstrap CI, Cohen's d) will never be paywalled.\n\n---\n\n## Install\n\n```bash\npip install agent-regress-cli\n# or\nuv add agent-regress-cli\n# or, from Node/npx (thin wrapper around the same Python CLI)\nnpx agent-regress-cli\n```\n\n---\n\n## Quickstart\n\n### In 30 seconds (CLI)\n\nAlready have per-run scores from your own harness? Point the CLI at two JSON arrays of scores, one per version:\n\n```bash\npip install agent-regress-cli\n\nagent-regress compare \\\n --version-a-results v1_scores.json \\\n --version-b-results v2_scores.json \\\n --metric tool_accuracy\n\n# ============================================================\n# agent-regress Report -- tool_accuracy\n# ============================================================\n# Verdict: REGRESSED\n# p-value: 0.0000\n# Cohen's d: -2.193\n# 95% CI: [-0.213, -0.148]\n#\n# Version A: 0.8470 +/- 0.0525 (n=50)\n# Version B: 0.6685 +/- 0.1025 (n=50)\n# Delta: -0.1786\n# ============================================================\n```\n\nAdd `--json --fail-on-regression` to get clean, parseable output and a non-zero exit code on `REGRESSED`, for wiring straight into CI.\n\n![agent-regress compare run from the CLI end to end, from install through a REGRESSED verdict](https://raw.githubusercontent.com/RudrenduPaul/agent-eval/main/docs/assets/demo-3-cli.gif)\n\n**All `agent-regress compare` flags:**\n\n| Flag | Default | Description |\n|---|---|---|\n| `--version-a-results PATH` | *(required)* | Path to a JSON array of per-run scores for version A (baseline). |\n| `--version-b-results PATH` | *(required)* | Path to a JSON array of per-run scores for version B (candidate). |\n| `--metric NAME` | `accuracy` | Name of the metric being compared, shown in the report. |\n| `--p-threshold P` | `0.05` | Significance threshold for the Mann-Whitney U p-value. |\n| `--min-effect D` | `0.2` | Minimum \\|Cohen's d\\| to call a statistically significant difference REGRESSED/IMPROVED rather than STABLE. |\n| `--n-resamples N` | `1000` | Number of bootstrap resamples used for the confidence interval (minimum: 100). |\n| `--json` | off | Print the report as a single JSON object to stdout instead of the human-readable format. Warnings still go to stderr, so stdout stays clean, parseable JSON. |\n| `--fail-on-regression` | off | Exit with status 1 if the verdict is REGRESSED (useful for CI). Without this flag, the command exits 0 regardless of verdict. |\n\nThe top-level `agent-regress --version` flag prints the installed version and exits.\n\n**Exit codes:**\n\n| Code | Meaning |\n|---|---|\n| `0` | Ran successfully. Verdict may be REGRESSED, STABLE, IMPROVED, or INSUFFICIENT_DATA — without `--fail-on-regression`, the exit code doesn't reflect the verdict. |\n| `1` | `--fail-on-regression` was passed and the verdict is REGRESSED. |\n| `2` | Usage or data error: invalid/missing arguments, no subcommand given, a `--version-*-results` file that doesn't exist or isn't valid JSON, an empty/non-numeric scores array, or an out-of-range value for `--p-threshold` (must be in `(0, 1)`), `--min-effect` (must be `>= 0`), or `--n-resamples` (must be `>= 100`). |\n\n### In your code (Python API)\n\nDriving the agent yourself instead of pre-computing scores? Use the Python API:\n\n```python\nfrom agent_regress import compare\n\n# Any callable that takes a test case dict and returns a score 0.0-1.0\ndef agent_v1(test_case: dict) -> float:\n ... # your existing agent\n\ndef agent_v2(test_case: dict) -> float:\n ... # your updated agent\n\ntest_suite = [\n {\"query\": \"find SKU for order 8823\", \"expected\": \"SKU-4492\"},\n # ... more test cases\n]\n\nreport = compare(\n version_a=agent_v1,\n version_b=agent_v2,\n test_suite=test_suite,\n n_runs=50,\n metric=\"tool_accuracy\", # use any name except \"accuracy\" when agents return floats\n)\n\nprint(report) # structured output with p-value, CI, effect size\nreport.assert_stable() # raises AssertionError if behavior regressed\n```\n\nAgent returns text? Pass a scorer or use the built-ins:\n\n```python\nfrom agent_regress import compare, exact_match_scorer, f1_scorer\n\n# exact_match_scorer: 1.0 if str(output).strip() == str(expected).strip()\n# f1_scorer: token-level F1 (multiset — handles repeated tokens correctly)\nreport = compare(\n version_a=agent_v1,\n version_b=agent_v2,\n test_suite=test_suite,\n n_runs=50,\n scorer=exact_match_scorer, # test_case must have an \"expected\" key\n)\n```\n\nOr write your own:\n\n```python\ndef my_scorer(output: str, test_case: dict) -> float:\n return 1.0 if output.strip() == test_case[\"expected\"] else 0.0\n\nreport = compare(..., scorer=my_scorer)\n```\n\n---\n\n## MCP Server\n\nagent-eval ships a [Model Context Protocol](https://modelcontextprotocol.io) server so an AI agent\n(Claude, Cursor, or any MCP-compatible client) can run statistical regression tests directly,\nwithout a human invoking the CLI by hand.\n\nInstall the extra:\n\n```bash\npip install \"agent-regress-cli[mcp]\"\n```\n\nAdd it to your MCP client's config (for Claude Desktop, `claude_desktop_config.json`):\n\n```json\n{\n \"mcpServers\": {\n \"agent-eval\": {\n \"command\": \"uvx\",\n \"args\": [\"--from\", \"agent-regress-cli\", \"agent-regress-mcp\"]\n }\n }\n}\n```\n\nThe server exposes one tool, `run`, that shells out to the `agent-regress` CLI with the given\nsubcommand and arguments plus `--json`, and returns the parsed JSON result:\n\n```\nrun([\"compare\", \"--version-a-results\", \"a.json\", \"--version-b-results\", \"b.json\", \"--metric\", \"accuracy\"])\n```\n\nTransport is stdio, so there is nothing to host: the MCP client spawns the server as a local\nsubprocess. Source: [`src/agent_regress/mcp_server.py`](src/agent_regress/mcp_server.py).\n\n---\n\n## Why not DeepEval, Promptfoo, or Braintrust?\n\n| Capability | Agent Evaluation | DeepEval | Braintrust | Promptfoo |\n|---|---|---|---|---|\n| Statistical version comparison (p-values) | **Yes** | No | No | No |\n| Effect size reporting (Cohen's d) | **Yes** | No | No | No |\n| Bootstrap 95% confidence intervals | **Yes** | No | No | No |\n| Distributional shift detection | **Yes** | No | No | No |\n| Tau-bench pass^k harness (k=1,4,8) | **Yes** | No | No | No |\n| GAIA Level 1-3 split harness | **Yes** | No | No | No |\n| SWE-bench scaffold score harness | **Yes** | No | No | No |\n| Self-hostable, zero SaaS required | **Yes** | Partial | No | Yes |\n| Sample size warnings | **Yes** | No | No | No |\n| Core license | Apache 2.0 | MIT | Proprietary | MIT† |\n| Requires cloud account | No | Optional | Yes | No |\n| Test type | Distributional | Threshold | Threshold | Threshold |\n\n†Promptfoo acquired by OpenAI, March 2026; remains open source under its current license.\n\nDeepEval tests whether an individual agent response clears a quality bar. Agent Evaluation tests whether behavior changed significantly between two agent versions, a different statistical question that threshold testing cannot answer. The scipy Mann-Whitney U call at the core is one line, so any SaaS eval platform can add it. What accumulates over time through production use is version-specific regression history and a community-maintained benchmark leaderboard with independent result verification.\n\n---\n\n## Real regressions statistical testing catches that threshold testing misses\n\n**LangGraph**\n\n- [#5243](https://github.com/langchain-ai/langgraph/pull/5243): a new typed `context=` API replaced untyped `config['configurable']`. A single run on either invocation style still clears a threshold check; only a version-A-vs-B comparison shows whether the switch changed measured behavior.\n- [#4486](https://github.com/langchain-ai/langgraph/pull/4486): node/task-level result caching can silently mask repeated-sampling variance. Threshold checks don't care whether a result came from cache; a statistical comparison depends on genuinely independent samples, so `agent-eval` added cache-busting to protect that assumption.\n\n**OpenAI Agents SDK**\n\n- [#2463](https://github.com/openai/openai-agents-python/pull/2463): agent-as-tool calls were silently dropping the parent run's `RunConfig`. The nested call still returns a normal-looking response, so a single-response check clears; only inspecting config propagation across runs reveals the regression.\n- [#2214](https://github.com/openai/openai-agents-python/pull/2214): image/audio/file tool outputs were silently downgraded to text-only. A text-only threshold scorer has no way to notice a dropped attachment.\n\n**CrewAI**\n\n- [#6134](https://github.com/crewAIInc/crewAI/pull/6134): a security fix for file tools leaking absolute filesystem paths in responses. A quality scorer checks whether the answer is correct, not whether it also leaks a path, so the leak clears the bar.\n- [#6236](https://github.com/crewAIInc/crewAI/pull/6236): tools gained an optional Pydantic `output_schema`, moving from unstructured `str()` output to structured JSON. Both the old and new format can look \"reasonable\" to a threshold scorer even though the schema changed underneath.\n\nThese are the regressions that motivated this project. Full detail on all 14 individually-documented PRs (drawn from a 29-PR, 239-row validation campaign across LangGraph, CrewAI, and the OpenAI Agents SDK) is in [docs/pr-analysis.md](docs/pr-analysis.md).\n\n---\n\n## The problem this solves\n\nYou changed a prompt. Or switched from GPT-4o to GPT-4o-mini to cut costs. Or a dependency updated silently. Your evals still pass, because they test individual responses against fixed thresholds. They don't detect whether behavior shifted across the whole distribution.\n\nA 3-point drop in accuracy might be noise from LLM variance. Or it might be a real regression. Without statistical testing you cannot tell which. Teams either ignore small drops and miss real problems, or escalate everything and drown in false alarms.\n\nAgent Evaluation answers the distributional question with a p-value and effect size:\n\n```\n============================================================\nagent-regress Report -- tool_accuracy\n============================================================\nVerdict: REGRESSED\np-value: 0.0031\nCohen's d: -0.610\n95% CI: [-0.221, -0.067]\n\nVersion A: 0.8400 +/- 0.0601 (n=50)\nVersion B: 0.7000 +/- 0.0903 (n=50)\nDelta: -0.1400\n============================================================\n```\n\nWhen CI fails, the assertion error gives the deploy-blocking message:\n\n```\nAssertionError: REGRESSED: tool_accuracy dropped 16.7%\n(p=0.003, Cohen's d=-0.61, 95% CI [-0.22, -0.07])\nVersion A: 0.840 +/- 0.060 (n=50)\nVersion B: 0.700 +/- 0.090 (n=50)\n```\n\nWhen nothing changed:\n\n```\nVerdict: STABLE\np-value: 0.4100\nCohen's d: 0.021\n```\n\nDeepEval, Promptfoo, and Braintrust test whether individual responses meet thresholds. None of them answer whether a version's behavior distribution shifted significantly from the last. Agent Evaluation addresses that specific statistical question, which threshold testing cannot answer.\n\n---\n\n## Add to CI: fail the build on regression\n\nTwo patterns. Pick one.\n\n**`report.assert_stable()`** — inline, after you've already called `compare()`:\n\n```python\n# test_regression.py -- add to your existing test suite\nfrom agent_regress import compare\n\ndef test_no_regression():\n report = compare(\n version_a=production_agent,\n version_b=staging_agent,\n test_suite=load_test_suite(),\n n_runs=50,\n )\n report.assert_stable(\n p_threshold=0.05, # act on changes at p < 0.05\n min_effect=0.2, # Cohen's d threshold -- ignore noise below 0.2\n )\n```\n\n**`RegressionGate`** — reusable gate object, useful when you run multiple comparisons with the same thresholds:\n\n```python\nfrom agent_regress import compare, RegressionGate\n\ngate = RegressionGate(p_threshold=0.05, min_effect=0.2)\n\ndef test_tool_accuracy():\n report = compare(version_a=prod, version_b=staging, test_suite=suite, n_runs=50)\n gate.check(report) # raises AssertionError on regression; warns if n < 30\n\ndef test_routing_accuracy():\n report = compare(version_a=prod, version_b=staging, test_suite=routing_suite, n_runs=50)\n gate.check(report)\n```\n\n![agent-regress catching a P0 regression in CI and failing the build with a deploy-blocking AssertionError](https://raw.githubusercontent.com/RudrenduPaul/agent-eval/main/docs/assets/demo-2-p0-crash.gif)\n\nBoth patterns: warn (not fail) when `n < 30` per version, and treat `n < 10` as insufficient data and skip the gate. This CI-gate threshold (30) is intentionally lower than `compare()`'s own general low-power warning (`n < 50`, see [FAQ](#faq)) — it exists to stop a genuinely too-small sample from silently gating a build, not to guarantee 80% statistical power the way the 50-run recommendation does.\n\n```bash\nuv run pytest test_regression.py\n```\n\n---\n\n## Statistical methods\n\nAgent Evaluation uses three statistical tests, applied in combination:\n\n**Mann-Whitney U** compares two score distributions without assuming normality. LLM scores are not Gaussian. The U test is distribution-free and robust to the long tails and bimodal distributions that appear in real agent outputs.\n\n**Bootstrap confidence intervals** (1,000 resamples, seed=42) give a 95% CI on the mean score delta. The CI tells you how large the shift was: a CI of [-0.22, -0.07] means you can be 95% confident the true per-run accuracy drop is between 7 and 22 percentage points.\n\n**Cohen's d** (pooled standard deviation) separates statistical significance from operational significance. A shift at p=0.001 with d=0.04 is real but meaningless. A shift at p=0.06 with d=0.5 is operationally large but requires more data to confirm. The default CI gate acts only when both p < 0.05 and d >= 0.2.\n\nSee [docs/statistical-methods.md](docs/statistical-methods.md) for the full methodology.\n\n---\n\n## Benchmarks\n\nStatistical test overhead is the time to run the comparison itself, not the agent calls. Agent calls are the bottleneck; the statistics are not.\n\nMeasured on Apple M3 Pro, Python 3.14, scipy 1.15, numpy 2.2:\n\n| Operation | n=50 per version | n=1,000 per version |\n|---|---|---|\n| Mann-Whitney U | **0.34ms** | **0.47ms** |\n| Bootstrap CI (1,000 resamples) | **26ms** | **31ms** |\n| Full compare() statistical overhead | **~27ms** | **~32ms** |\n\nSee [docs/benchmarks.md](docs/benchmarks.md) to reproduce.\n\n---\n\n## Integration matrix\n\n| Framework | Status | Install |\n|---|---|---|\n| LangGraph | Shipped (v0.1) | `pip install agent-regress-cli[langgraph]` |\n| OpenAI Agents SDK | Shipped (v0.1) | `pip install agent-regress-cli[openai-agents]` |\n| CrewAI | Shipped (v0.1) | `pip install agent-regress-cli[crewai]` |\n| LangChain LCEL | Shipped (v0.1) | `pip install agent-regress-cli[langchain]` |\n| AutoGen | Planned (v0.3) | |\n| Vercel AI SDK (TypeScript) | Planned (v0.4) | |\n\nComparing two *installed versions* of the same framework (rather than two\nin-process configurations)? See\n[docs/cross-version-comparison.md](docs/cross-version-comparison.md) for the\n`subprocess_runner()` pattern.\n\n> [!WARNING]\n> The `[crewai]` extra: CrewAI's own memory/knowledge/RAG backend can pull in ChromaDB, which currently has an unpatched critical CVE ([GHSA-f4j7-r4q5-qw2c](https://github.com/advisories/GHSA-f4j7-r4q5-qw2c)) affecting any ChromaDB server run with `trust_remote_code=True` and exposed to the network. `agent-eval` never starts, configures, or exposes a ChromaDB server itself, so this only matters if your own `Crew` does — don't run a network-exposed ChromaDB instance with `trust_remote_code=True` until a fix ships.\n\n---\n\n## Standard benchmarks\n\nAgent Evaluation ships harnesses for the three standard agent benchmarks:\n\n**Tau-bench pass^k** measures reliability across k independent attempts. Single-run benchmarks miss degradation: an agent that succeeds 60% of the time at k=1 reaches 99.93% at k=8. The k=1 vs k=8 curve is the signal.\n\n```python\nfrom agent_regress.benchmarks.tau_bench import TauBenchHarness\n\nharness = TauBenchHarness(agent=my_agent, dataset=tau_bench_dataset)\nresults = harness.evaluate(k_values=[1, 4, 8])\n```\n\n**GAIA Level 1-3 split** stratifies by task difficulty. Overall accuracy hides per-difficulty regressions: a prompt change that helps Level 1 often hurts Level 3.\n\n```python\nfrom agent_regress.benchmarks.gaia import GAIAHarness\n\nharness = GAIAHarness(agent=my_agent, dataset=gaia_dataset)\nresults = harness.evaluate() # returns list[GAIALevelResult], one per level\nfor r in results:\n print(f\"Level {r.level}: {r.accuracy:.3f} ({r.n_correct}/{r.n_questions})\")\n```\n\n**SWE-bench scaffold score** isolates framework contribution from model contribution.\n\n```python\nfrom agent_regress.benchmarks.swebench import SWEBenchHarness\n\nharness = SWEBenchHarness(agent=my_agent, dataset=swe_dataset)\nresult = harness.evaluate()\nprint(f\"scaffold pass rate: {result.scaffold_pass_rate:.3f} ({result.n_resolved}/{result.n_instances})\")\n```\n\nSee [leaderboard/README.md](leaderboard/README.md) to submit results.\n\n---\n\n## Try it in Docker\n\n```bash\ngit clone https://github.com/RudrenduPaul/agent-eval\ncd agent-eval\ndocker compose up\n```\n\nStarts two services:\n\n- **web** (`http://localhost:8080`) — leaderboard UI served by `web/serve.py`, reading `leaderboard/results/*.json`\n- **example** — runs `examples/01-basic-comparison/example.py` and prints the comparison report to stdout\n\nGood for verifying the install works and seeing the leaderboard UI before wiring agent-regress into your own agent.\n\n---\n\n## Security\n\n- **Supply chain:** Releases are built and published from a GitHub Actions workflow, signed with Sigstore, and ship with a CycloneDX SBOM attached to every GitHub Release. (No SLSA provenance attestation is generated yet — that would require adopting `slsa-framework/slsa-github-generator`.)\n- **Vulnerability scanning:** Trivy scans on every CI run (HIGH/CRITICAL only, exit on unfixed). CodeQL static analysis on every push.\n- **Dependency pinning:** Dependabot keeps all GitHub Actions and Python dependencies current.\n- **Disclosure:** [SECURITY.md](SECURITY.md) — report vulnerabilities privately via GitHub Security Advisories.\n\n---\n\n## Leaderboard\n\nThe `leaderboard/` directory version-controls Tau-bench pass^k, GAIA, and SWE-bench results across models and frameworks. Submit by opening a PR with a JSON file matching `leaderboard/schema.json`. Results are independently reproduced before merging.\n\nSee [leaderboard/README.md](leaderboard/README.md).\n\n---\n\n## FAQ\n\n**What is agent-eval, and what makes it different from a normal LLM eval framework?**\n\nAgent Evaluation is a statistics library for detecting whether an agent's behavior actually changed between two versions. Run the same test suite 50 times on version A and 50 times on version B, and it reports a p-value (Mann-Whitney U), an effect size (Cohen's d), and a bootstrap 95% confidence interval on the score delta. Most eval frameworks check whether a single response clears a fixed quality threshold. Agent Evaluation instead answers a distributional question: did the score distribution shift significantly, or is a change just LLM-run-to-run noise.\n\n**How do I install it, and which platforms does it support?**\n\n`pip install agent-regress-cli` or `uv add agent-regress-cli`. It requires Python 3.10 through 3.13 (per the classifiers in `pyproject.toml`) and has no OS-specific code, so it runs anywhere those Python versions run. A Node/npx wrapper (`npx agent-regress-cli`, also published as `agent-regress-cli` on npm) is also available -- it shells out to this same Python package, so a Python toolchain (or `uv`/`pipx`, which can run it ephemerally without a manual `pip install` step) still needs to be available; it prints an actionable error if neither is found.\n\n**How does it compare to DeepEval, Promptfoo, or Braintrust?**\n\nThe full breakdown is in the [comparison table](#why-not-deepeval-promptfoo-or-braintrust) above. In short: DeepEval, Promptfoo, and Braintrust all test whether an individual response clears a fixed quality bar. None of the three report a p-value, an effect size, or a bootstrap confidence interval on whether behavior shifted between two versions, which is the specific statistical question agent-eval is built to answer.\n\n**I ran a comparison and got a warning about insufficient statistical power, or a verdict of INSUFFICIENT_DATA. What does that mean?**\n\nThe library warns (but does not fail) when either version has fewer than 50 runs, since that is the sample size needed for reliable detection of a moderate effect (Cohen's d of 0.2) at 80% power. Below 10 runs per version, it returns `INSUFFICIENT_DATA` instead of a REGRESSED/STABLE/IMPROVED verdict, since the sample is too small to trust any statistical conclusion. Re-run with `n_runs=50` or higher for a verdict you can act on.\n\n**Does agent-eval call my LLM or manage API keys for me?**\n\nNo. `compare()` takes two callables you provide, `version_a` and `version_b`, and runs your existing agent code against your test suite. Agent Evaluation never makes a model call itself, and the stats module (`src/agent_regress/stats/`) is required to stay pure Python and scipy with no LLM calls, so the statistical core has no network dependency and nothing to configure credentials for.\n\n**Which agent frameworks does it integrate with today?**\n\nLangGraph, the OpenAI Agents SDK, CrewAI, and LangChain LCEL are shipped as of v0.1 (see the [integration matrix](#integration-matrix) above), each installable as an extra, e.g. `pip install agent-regress-cli[langgraph]`. AutoGen and a Vercel AI SDK (TypeScript) integration are planned but not yet shipped.\n\n**Can I use agent-eval commercially, and what license is it under?**\n\nYes. It is licensed under Apache License 2.0, which permits commercial use, modification, and distribution, and includes an explicit patent grant. You need to preserve the copyright and license notices and state any changes you make; there is no warranty. See [LICENSE](LICENSE) for the full text.\n\n---\n\n## Contributing\n\n- Read [CONTRIBUTING.md](CONTRIBUTING.md) before opening a PR\n- Good first issues are labeled in GitHub\n- Stats module (`src/agent_regress/stats/`) must stay pure Python + scipy — no LLM calls, ever\n- All PRs require 95% coverage on `stats/`, 80% overall\n\nGitHub Discussions for design questions.\n\nApache 2.0. Contributions welcome.\n\n---\n\n## Cite this work\n\nIf you use Agent Evaluation in research, please cite:\n\n```bibtex\n@software{paul2026agenteval,\n author = {Paul, Rudrendu and Nandy, Sourav},\n title = {Agent Evaluation: Statistical Regression Testing for LLM Agents},\n year = {2026},\n url = {https://github.com/RudrenduPaul/agent-eval},\n license = {Apache-2.0}\n}\n```\n\n---\n\n*Built by Rudrendu Paul and Sourav Nandy*\n", "bytes": 25422, "sha": "9430915050880f3b32c210ee2ea48ec379a368e4f77f829e94a400e2fe63db7a", "repo_slug": "rudrendupaul/agent-eval", "fonte": "repo", "truncated": false, "api": "https://agentalog.com/api/listings/mcp_io_github_rudrendupaul_agent_eval_0e117396/readme" }