{
  "markdown": "<!-- mcp-name: io.github.hidai25/evalview-mcp -->\n<!-- keywords: AI agent testing, regression detection, golden baselines -->\n\n<p align=\"center\">\n  <img src=\"assets/logo.png\" alt=\"EvalView\" width=\"350\">\n  <br>\n  <strong>Snapshot testing for AI agents.</strong><br>\n  Record what your agent does today. Get told when it silently changes.\n</p>\n\n<p align=\"center\">\n  <a href=\"https://pypi.org/project/evalview/\"><img src=\"https://img.shields.io/pypi/v/evalview.svg?label=release\" alt=\"PyPI version\"></a>\n  <a href=\"https://pypi.org/project/evalview/\"><img src=\"https://img.shields.io/pypi/dm/evalview.svg?label=downloads\" alt=\"PyPI downloads\"></a>\n  <a href=\"https://github.com/hidai25/eval-view/actions/workflows/ci.yml\"><img src=\"https://github.com/hidai25/eval-view/actions/workflows/ci.yml/badge.svg?branch=main\" alt=\"Package CI\"></a>\n  <a href=\"https://github.com/hidai25/eval-view/actions/workflows/dogfood.yml\"><img src=\"https://github.com/hidai25/eval-view/actions/workflows/dogfood.yml/badge.svg?branch=main\" alt=\"Core Dogfood\"></a>\n  <a href=\"https://github.com/hidai25/eval-view/actions/workflows/dogfood-live.yml\"><img src=\"https://github.com/hidai25/eval-view/actions/workflows/dogfood-live.yml/badge.svg?branch=main&amp;event=workflow_dispatch\" alt=\"Live Provider Checks (manual)\"></a>\n  <a href=\"https://github.com/hidai25/eval-view/stargazers\"><img src=\"https://img.shields.io/github/stars/hidai25/eval-view?style=social\" alt=\"GitHub stars\"></a>\n  <a href=\"https://opensource.org/licenses/Apache-2.0\"><img src=\"https://img.shields.io/badge/License-Apache_2.0-blue.svg\" alt=\"License\"></a>\n</p>\n\n---\n\nYour agent returns `200` and looks fine. But a model update, a provider change, or a one-line prompt edit just made it skip a clarification, call the wrong tool, or quietly drop output quality. Your tests still pass. Your users notice before you do.\n\n**EvalView snapshots your agent's behavior — the tools it calls, in what order, with what output — and tells you the moment that behavior changes.** Like Jest snapshots, but for tool-calling, multi-turn agents.\n\n[![demo.gif](assets/demo.gif)](https://github.com/user-attachments/assets/96d8b5f7-3561-44a1-86a4-270fb0d1d8a6)\n\n<sub>↑ 30-second live demo — no API key needed</sub>\n\n## Quick Start\n\n> **OpenAI adapter migration:** OpenAI shut down the Assistants API on **August 26, 2026**.\n> The latest published EvalView release, **0.8.1**, still uses that API; the Responses API\n> migration is currently **unreleased source**. If you use `openai-assistants`, follow the\n> [migration guide](docs/OPENAI_MIGRATION.md) before running your tests. An `assistant_id`\n> alone cannot preserve your agent's configuration. Other adapters are unaffected.\n\n```bash\npip install evalview\n```\n\n```bash\nevalview snapshot    # Record your agent's current behavior as the baseline\nevalview check       # After any change, diff against the baseline\n```\n\nThat's the whole loop. `check` returns one of:\n\n```\n  ✓ login-flow        PASSED          behavior matches baseline\n  ⚠ refund-request    TOOLS_CHANGED   called a different tool, or in a different order\n  ✗ billing-dispute   REGRESSION      score dropped — output quality fell\n```\n\nIt diffs the **whole trajectory** — tool names, parameters, and order — not just the final string. The deterministic tool + sequence diff runs offline, with no API key. Add an LLM judge only when you want output-quality scoring.\n\nExecuting your agent can still incur backend API charges: `--no-judge` skips the\njudge, not those calls. Embedding-based semantic comparison is opt-in.\n\nNo agent yet? See it work in 30 seconds:\n\n```bash\nevalview demo\n```\n\n## Why snapshot testing (and not assertions)?\n\nMost eval tools ask you to *write down what \"good\" looks like* — assertions, metrics, rubrics. That's a lot of upfront work, and you can only catch the failures you thought to assert.\n\nEvalView inverts it: **it records what your agent actually does now, and flags any drift from that.** You catch regressions you never anticipated, with zero assertions written. When the new behavior is correct, `evalview snapshot` accepts it as the new baseline — same as updating a snapshot in Jest.\n\n| | EvalView | Assertion-based eval tools |\n|---|---|---|\n| Setup | Record current behavior | Write assertions/metrics first |\n| Catches | Any drift from baseline | Only what you asserted |\n| Non-determinism | Multi-variant baselines (up to 5 valid paths) | You handle it |\n| Unit of comparison | Full tool-call trajectory | Usually final output |\n\nThis makes EvalView a **merge-time regression gate**, which is a different job from observability (Langfuse, LangSmith) or metric scoring (promptfoo, DeepEval, Braintrust). Many teams run one of those for visibility **and** EvalView as the gate. [Honest comparisons →](docs/COMPARISONS.md)\n\n## EvalView tests itself in public, every day\n\nEvery day at 09:00 UTC, on pull requests, and on pushes to main,\n[Core Dogfood](.github/workflows/dogfood.yml) exercises the non-live test suite,\ntype checks, local mock-agent `snapshot` / `check`, `evalview demo`, end-to-end\nflows, and an `evalview monitor` smoke test. It uses no paid API credentials and\nmakes no paid inference calls. GitHub runner usage is separate.\n\n[Live Provider Checks](.github/workflows/dogfood-live.yml) test the real evaluator and\nchat assistant only when a maintainer explicitly opts into paid API use on main.\nThey have no automatic schedule. Their badge records the last manual run; a green\ncore badge does not establish live-provider health or rule out provider drift.\n\nPackage CI, core dogfood, and live checks have separate badges. Failed or incomplete\nchecks remain visible within their scope, with logs and reports preserved as\nartifacts. Rolling issues use separate `dogfood-core` and `dogfood-live` labels.\nA provider outage, exhausted quota, or missing credential means live health is\nunavailable; it does not prove an agent regression.\n\nThe historical [incident #264](https://github.com/hidai25/eval-view/issues/264)\nremains available for maintainer review of fresh evidence from both scopes. Neither\nworkflow automatically closes it. Trust warnings are evidence to investigate,\nnot proof of gaming or of a particular root cause.\n\n[Core runs →](https://github.com/hidai25/eval-view/actions/workflows/dogfood.yml) · [Manual live runs →](https://github.com/hidai25/eval-view/actions/workflows/dogfood-live.yml) · [Run and triage guide →](docs/INTERNAL_DOGFOODING.md#daily-failure-triage)\n\n## CI: block regressions in every PR\n\n```yaml\n# .github/workflows/evalview.yml\nname: EvalView\non: [pull_request]\njobs:\n  agent-check:\n    runs-on: ubuntu-latest\n    permissions: { pull-requests: write }\n    steps:\n      - uses: actions/checkout@v4\n      - uses: hidai25/eval-view@v0.8.1\n        with:\n          openai-api-key: ${{ secrets.OPENAI_API_KEY }}\n```\n\nYou get a PR comment with the diff, cost/latency deltas, and a pass/fail gate. [CI/CD guide →](docs/CI_CD.md)\n\n## Works with your stack\n\nLangGraph · CrewAI · OpenAI · Claude · Mistral · Ollama · MCP · **any HTTP API**.\n\n```bash\nevalview check --agent http://localhost:8000/invoke\n```\n\n[Framework details →](docs/FRAMEWORK_SUPPORT.md)\n\n## Use it as a library\n\n```python\nfrom evalview import gate\n\nresult = gate(test_dir=\"tests/\")\nresult.passed   # bool\nresult.diffs    # per-test scores and tool diffs\n```\n\n[Python API →](docs/CLI_REFERENCE.md#python-api)\n\n## More\n\nEvalView also does multi-turn testing, statistical/pass@k runs, record/replay cassettes, model-drift canaries, production monitoring with Slack alerts, and auto-generated regression tests from incidents. These are power-user features — start with `snapshot` and `check`, reach for the rest when you need them.\n\n→ [Full feature reference](docs/CLI_REFERENCE.md) · [Getting Started](docs/GETTING_STARTED.md) · [FAQ](docs/FAQ.md)\n\n→ [Documentation index](docs/README.md) · [OpenAI migration](docs/OPENAI_MIGRATION.md) · [Release process](docs/RELEASING.md)\n\n### Why I built EvalView\n\nAn agent that looked successful kept pulling entire documents into its context and made one question cost $42.93. That experience led me to build EvalView. I wrote about it in [“I Was Running an AI Casino. Then I Started Writing Tests for My Agents”](https://medium.com/@hidaibarmor/i-was-running-an-ai-casino-then-i-started-writing-tests-for-my-agents-93cb3468ce1e). The December 2025 post is the origin story; use the current docs for setup and commands.\n\n## Contributing\n\nThis is a young project built mostly by one developer. Issues, PRs, and \"I tried it and X was confusing\" feedback are all genuinely valuable.\n\n- [Open an issue](https://github.com/hidai25/eval-view/issues) · [Discussions](https://github.com/hidai25/eval-view/discussions) · [CONTRIBUTING.md](CONTRIBUTING.md)\n\n**License:** Apache 2.0\n\n---\n\n[![Star History Chart](https://api.star-history.com/svg?repos=hidai25/eval-view&type=Date)](https://star-history.com/#hidai25/eval-view&Date)\n",
  "bytes": 8944,
  "sha": "7e1b510a95040359dc43caa27cc99677d23ef3122429b75c4bc28c68d39ddacf",
  "repo_slug": "hidai25/eval-view",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/plg_hidai25_eval_view_evalview_4f21f761/readme"
}