{
  "markdown": "# @vercel/agent-eval\n\nTest AI coding agents on your framework. Measure what actually works.\n\n## Why?\n\nYou're building a frontend framework and want AI agents to work well with it. But how do you know if:\n- Your documentation helps agents write correct code?\n- Adding an MCP server improves agent success rates?\n- Sonnet performs as well as Opus for your use cases?\n- Your latest API changes broke agent compatibility?\n\n**This framework gives you answers.** Run controlled experiments, measure pass rates, compare techniques.\n\n## Quick Start\n\n```bash\n# Create a new eval project\nnpx @vercel/agent-eval init my-agent-evals\ncd my-agent-evals\n\n# Install dependencies\nnpm install\n\n# Add your API keys\ncp .env.example .env\n# Edit .env with your AI_GATEWAY_API_KEY and VERCEL_TOKEN\n\n# Preview what will run (no API calls, no cost)\nnpx @vercel/agent-eval --dry\n\n# Run all experiments\nnpx @vercel/agent-eval\n```\n\n## CLI\n\n### Run all experiments\n\n```bash\nnpx @vercel/agent-eval\n```\n\nWith no arguments, the CLI discovers every `experiments/*.ts` file and runs them all. Each experiment runs in parallel. Results with matching fingerprints are reused automatically (see [Result Reuse](#result-reuse)).\n\n### Run a single experiment\n\n```bash\nnpx @vercel/agent-eval cc\n```\n\nThe argument is the experiment filename without `.ts`. This resolves to `experiments/cc.ts`.\n\n### Flags\n\n| Flag                 | Description                                                                                |\n|----------------------|--------------------------------------------------------------------------------------------|\n| `--dry`              | Preview what would run without executing. No API calls, no cost.                           |\n| `--smoke`            | Quick setup verification. Picks the first eval alphabetically, runs once per model.        |\n| `--force`            | Ignore cached fingerprints and re-run everything. Only applies when running all.           |\n| `--ack-failures`     | Keep non-model failures as final results instead of deleting them.                         |\n\nFlags work with both modes:\n\n```bash\nnpx @vercel/agent-eval --dry          # preview all experiments\nnpx @vercel/agent-eval cc --dry       # preview a single experiment\nnpx @vercel/agent-eval --smoke        # smoke test all experiments\nnpx @vercel/agent-eval cc --smoke     # smoke test one experiment\n```\n\n### Other commands\n\n```bash\nnpx @vercel/agent-eval init <name>          # scaffold a new eval project\nnpx @vercel/agent-eval playground           # launch web-based results viewer\nnpx @vercel/agent-eval playground --watch   # live mode (watches for new results)\n```\n\n## Creating Evals\n\nEach eval tests one specific task an agent should be able to do with your framework.\n\n### Directory structure\n\n```\nevals/\n  create-button-component/\n    PROMPT.md           # Task for the agent\n    EVAL.ts             # Tests to verify success (or EVAL.tsx for JSX)\n    package.json        # Your framework as a dependency\n    src/                # Starter code\n```\n\n**PROMPT.md** -- what you want the agent to do:\n\n```markdown\nCreate a Button component using MyFramework.\n\nRequirements:\n- Export a Button component from src/components/Button.tsx\n- Accept `label` and `onClick` props\n- Use the framework's styling system for hover states\n```\n\n**EVAL.ts** -- how you verify it worked:\n\n```typescript\nimport { test, expect } from 'vitest';\nimport { readFileSync, existsSync } from 'fs';\nimport { execSync } from 'child_process';\n\ntest('Button component exists', () => {\n  expect(existsSync('src/components/Button.tsx')).toBe(true);\n});\n\ntest('has required props', () => {\n  const content = readFileSync('src/components/Button.tsx', 'utf-8');\n  expect(content).toContain('label');\n  expect(content).toContain('onClick');\n});\n\ntest('project builds', () => {\n  execSync('npm run build', { stdio: 'pipe' });\n});\n```\n\nUse **EVAL.tsx** when your tests require JSX syntax (React Testing Library, component rendering). You only need one eval file per fixture -- choose `.tsx` if any test needs JSX.\n\n### Asserting on agent behavior\n\nEVAL.ts tests can assert not just on the files the agent produced, but on *how* it worked — which shell commands it ran, which files it read, how many tool calls it made, etc. The framework automatically parses the agent's transcript and writes the results to `__agent_eval__/results.json` in the sandbox before your tests run.\n\n```typescript\nimport { test, expect } from 'vitest';\nimport { readFileSync } from 'fs';\n\ntest('agent used the correct scaffolding command', () => {\n  const results = JSON.parse(readFileSync('__agent_eval__/results.json', 'utf-8'));\n  const commands = results.o11y.shellCommands.map((c: { command: string }) => c.command);\n  expect(commands).toContain('npx create-next-app project');\n});\n\ntest('agent did not make excessive tool calls', () => {\n  const results = JSON.parse(readFileSync('__agent_eval__/results.json', 'utf-8'));\n  expect(results.o11y.totalToolCalls).toBeLessThan(50);\n});\n```\n\nThe `results.o11y` object is a `TranscriptSummary` with these fields:\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `shellCommands` | `{ command, exitCode?, success? }[]` | Shell commands the agent ran |\n| `filesRead` | `string[]` | Files the agent read |\n| `filesModified` | `string[]` | Files the agent wrote or edited |\n| `toolCalls` | `Record<ToolName, number>` | Count of each tool type used |\n| `totalToolCalls` | `number` | Total tool calls made |\n| `webFetches` | `{ url, method?, status?, success? }[]` | Web fetches made |\n| `totalTurns` | `number` | Conversation turns |\n| `errors` | `string[]` | Errors encountered |\n| `thinkingBlocks` | `number` | Thinking/reasoning blocks |\n\n> **Note**: If the agent's transcript is unavailable (e.g. the agent crashed before producing output), `results.o11y` will be `null`.\n\n### Agentic LLM judge\n\nFor open-ended quality checks that exact assertions can't express, EVAL.ts can run an **agentic LLM judge**. Each judge assertion re-invokes the *same agent* that did the codegen, **in the same sandbox**, to evaluate a criterion — then returns pass/fail. No fresh sandbox, no copying evidence around.\n\n```typescript\nimport { test, expect } from 'vitest';\nimport { environment, transcript } from '@vercel/agent-eval/eval';\n\n// Judge the final state: the agent explores the project (read/grep/run) for evidence.\ntest('uses server components', async () => {\n  await expect(environment).toSatisfyCriterion('uses Server Components for the product list');\n});\n\n// Judge the transcript: how the agent worked. It reads the transcript by path, so the\n// full transcript is never stuffed into a prompt.\ntest('diagnosed properly', async () => {\n  await expect(transcript).toSatisfyCriterion('diagnosed with DevTools, not trial-and-error edits');\n});\n\n// Numeric: the judge scores 0-1; assert a threshold (still pass/fail overall).\ntest('quality bar', async () => {\n  await expect(environment).toScoreAtLeast('production-quality error handling', 0.8);\n});\n```\n\nTwo subjects, imported from `@vercel/agent-eval/eval` — no paths, the subject is implicit:\n\n- `environment` — the judge agent explores the final sandbox state (cwd) with its own tools.\n- `transcript` — the judge agent reads the materialized transcript by path.\n\nTwo matchers, on either subject:\n\n- `toSatisfyCriterion(criterion)` — passes when the judge decides the criterion is satisfied.\n- `toScoreAtLeast(criterion, threshold)` — passes when the judge's 0–1 score is `>= threshold`.\n\nOne deterministic matcher, on `transcript` only — no judge run, free and exact:\n\n- `toContainText(needle)` — passes when the raw transcript contains the substring, or matches the `RegExp` (use `/…/i` for case-insensitive). Built for `.not`, i.e. asserting the agent *never* said or reached for something:\n\n```typescript\ntest('never suggested the pages router API', () => {\n  expect(transcript).not.toContainText('getServerSideProps');\n  expect(transcript).not.toContainText(/getserversideprops/i); // any casing\n});\n```\n\nA missing or empty transcript **throws** (fails the test even under `.not`) — an uncaptured transcript is an infra failure, not evidence of absence. Note the transcript is the agent's native format (for `claude-code`, raw session JSONL), so text containing quotes or newlines appears JSON-escaped there; stick to identifier-like needles or match the escaped form with a `RegExp`. For semantic \"never did X\" checks, phrase the negation *inside* a `toSatisfyCriterion` criterion instead — do not use `.not.toSatisfyCriterion(...)`, which would invert the judge's fail-closed default into a fail-open one.\n\nYou supply only the **criterion** string; the framework owns the judge prompt and the verdict contract. On failure the assertion message carries the judge's reasoning, e.g. `[judge:environment] FAIL (score 0.42): product list is a Client Component`, so a failed judge clause is distinguishable from a failed deterministic test or a crash.\n\nBy default the judge uses the **same agent and model** as the run under test (self-grading). Because each assertion is a real agent run, it costs time and tokens — keep criteria focused.\n\n**Pin the judge** to grade every run with one fixed agent + model — the apples-to-apples choice when comparing models, since the judge quality no longer varies with the model under test (and a model never grades itself):\n\n```typescript\nconst config: ExperimentConfig = {\n  agent: 'codex',\n  model: 'gpt-5.4',\n  // Grade with a fixed Claude judge regardless of the model under test.\n  judge: { agent: 'vercel-ai-gateway/claude-code', model: 'claude-opus-4-8' },\n};\n```\n\n- `judge.model` is required (pinning the model is the point).\n- `judge.agent` is optional and defaults to the codegen agent — omit it to keep the same harness and only pin the model. When it names a different agent, that agent's CLI is installed in the sandbox automatically and its key is resolved from its own env var (falling back to `VERCEL_OIDC_TOKEN`).\n- Pinning changes the eval fingerprint, so a pinned run won't reuse self-graded cached results.\n\n**TypeScript support is built in.** Importing from `@vercel/agent-eval/eval` types the `environment`/`transcript` subjects _and_ registers the `toSatisfyCriterion` / `toScoreAtLeast` matchers on Vitest's `expect` — no manual `declare module 'vitest'` augmentation needed. The package also re-exports the `JudgeSubject` and `JudgeVerdict` types for advanced use.\n\n> **Note**: requires `validation: 'vitest'` (the default). The framework gives the eval process the run's credentials automatically so the judge can call the agent CLI in-sandbox.\n\n## Configuration Reference\n\n### Experiment config\n\n```typescript\n// experiments/my-experiment.ts\nimport type { ExperimentConfig } from '@vercel/agent-eval';\n\nconst config: ExperimentConfig = {\n  // Required: which agent to use\n  agent: 'vercel-ai-gateway/claude-code',\n\n  // Model to use. Omit this to use the underlying agent CLI's native default.\n  // Provide an array to run the same experiment across multiple models.\n  model: 'opus',\n\n  // How many times to run each eval (default: 1)\n  runs: 10,\n\n  // Stop after first success? (default: true)\n  earlyExit: false,\n\n  // npm scripts that must pass after agent finishes (default: [])\n  scripts: ['build', 'lint'],\n\n  // Validation mode after the agent finishes (default: 'vitest')\n  // 'vitest' - run EVAL.ts/EVAL.tsx plus configured scripts\n  // 'none' - response-only mode; skip EVAL.ts/EVAL.tsx, run scripts if provided\n  validation: 'vitest',\n\n  // Timeout per run in seconds (default: 600)\n  timeout: 600,\n\n  // Filter which evals to run (default: '*' for all)\n  evals: '*',\n  // evals: ['specific-eval'],\n  // evals: (name) => name.startsWith('api-'),\n\n  // Setup function for sandbox pre-configuration\n  setup: async (sandbox) => {\n    await sandbox.writeFiles({ '.env': 'API_KEY=test' });\n    await sandbox.runCommand('npm', ['run', 'setup']);\n  },\n\n  // Rewrite the prompt before running\n  editPrompt: (prompt) => `Use the skill.\\n\\n${prompt}`,\n\n  // Custom post-run analysis hook. Can attach analysis/metadata to result.json.\n  // `runData.generatedFiles` maps path -> Buffer; call .toString('utf-8') for text.\n  onRunComplete: async ({ runData }) => ({\n    ...runData,\n    result: {\n      ...runData.result,\n      analysis: { mentionedBrands: ['Vercel'] },\n    },\n  }),\n\n  // Optional brands to compare in downstream analysis.\n  brands: [\n    {\n      id: 'vercel',\n      name: 'Vercel',\n      domain: 'vercel.com',\n      aliases: ['Vercel Platform'],\n      isYourBrand: true,\n    },\n  ],\n\n  // Sandbox backend (default: 'auto' -- Vercel if token present, else Docker)\n  sandbox: 'auto',\n\n  // Copy project files to results directory (default: 'none')\n  // 'none' - don't copy files\n  // 'changed' - copy only files modified by the agent\n  // 'all' - copy the entire project including original fixture files\n  copyFiles: 'changed',\n\n  // Pin the agentic LLM judge (see \"Agentic LLM judge\" above). Omit to self-grade\n  // with the codegen agent+model. `model` required; `agent` defaults to codegen.\n  judge: { agent: 'vercel-ai-gateway/claude-code', model: 'claude-opus-4-8' },\n};\n\nexport default config;\n```\n\n### Agent selection\n\n```typescript\n// Vercel AI Gateway (recommended -- unified billing and observability)\nagent: 'vercel-ai-gateway/claude-code'  // Claude Code via AI Gateway\nagent: 'vercel-ai-gateway/codex'        // OpenAI Codex via AI Gateway\nagent: 'vercel-ai-gateway/opencode'     // OpenCode via AI Gateway\nagent: 'vercel-ai-gateway/fx'           // fx via AI Gateway (research runs)\n\n// Direct API (uses provider keys directly)\nagent: 'claude-code'  // requires ANTHROPIC_API_KEY\nagent: 'codex'        // requires OPENAI_API_KEY\nagent: 'gemini'       // requires GEMINI_API_KEY\nagent: 'cursor'       // requires CURSOR_API_KEY\n```\n\n### Custom agents\n\nRegister an implementation of the exported `Agent` interface before exporting\nan experiment config. Custom IDs are valid anywhere a built-in agent ID is\naccepted:\n\n```typescript\nimport {\n  registerAgent,\n  type Agent,\n  type ExperimentConfig,\n} from '@vercel/agent-eval';\n\nconst definition = {\n  name: 'my-agent',\n  displayName: 'My Agent',\n  defaultModel: 'default-model',\n  o11yAgentName: 'claude-code',\n  runnerPath: '/path/to/run.mjs',\n  getApiKeyEnvVar: () => 'MY_AGENT_API_KEY',\n  install: () => [],\n  configFiles: () => [],\n  authEnv: () => ({}),\n} satisfies Agent['definition'];\n\nconst myAgent: Agent = {\n  name: definition.name,\n  displayName: definition.displayName,\n  getApiKeyEnvVar: definition.getApiKeyEnvVar,\n  getDefaultModel: () => definition.defaultModel,\n  run: async (fixturePath, options) => {\n    // Invoke the agent and return an AgentRunResult.\n    return {\n      success: true,\n      output: 'done',\n      duration: 1000,\n    };\n  },\n  definition,\n};\n\nregisterAgent(myAgent);\n\nconst config: ExperimentConfig = {\n  agent: 'my-agent',\n};\n\nexport default config;\n```\n\nAgent IDs must be non-empty. A later registration with the same ID replaces the\nprevious one, which allows shared registration modules to be evaluated by\nmultiple experiment files. Registered agents conform to the complete `Agent`\ncontract, including the definition used for installation, authentication,\ninvocation, transcript parsing, and pinned judging.\n\n### Multi-model experiments\n\nProvide an array of models to run the same experiment on each one. Results are stored under separate directories (`experiment-name/model-name`):\n\n```typescript\nconst config: ExperimentConfig = {\n  agent: 'vercel-ai-gateway/claude-code',\n  model: ['opus', 'sonnet'],\n  runs: 10,\n};\n```\n\n### Native agent defaults\n\nWhen `model` is omitted, Agent Eval does not pass a model override. The\nunderlying agent CLI chooses the same native default it would use for a normal\nuser run:\n\n```typescript\nconst config: ExperimentConfig = {\n  agent: 'vercel-ai-gateway/claude-code',\n  runs: 10,\n};\n```\n\nResults use `modelPolicy: 'native-default'`, `requestedModel` is omitted, and\n`observedModel` is populated when the agent CLI exposes the runtime model in its\ntranscript or logs. Provide `model` to force a specific model.\n\n### Opt-in runtime controls\n\nAgent Eval preserves each agent CLI's normal behavior by default. Recommendation\nand controlled-treatment evals can opt into a narrower runtime:\n\n```typescript\nconst config: ExperimentConfig = {\n  agent: 'vercel-ai-gateway/claude-code',\n  disableBundledSkills: true,\n  webResearch: true,\n};\n```\n\n`disableBundledSkills` disables skills shipped by Claude Code or Codex while\nleaving caller-installed project and user skills available. OpenCode and fx do\nnot ship bundled skill catalogs, so this option does not change their runtimes.\nThe option is omitted by default, preserving existing arguments and agent\nbehavior. Other built-in agents reject this option until they expose an\nequivalent control.\n\n`webResearch` remains opt-in. It allows Claude Code's `WebSearch`/`WebFetch`,\nenables Codex live search (including custom AI Gateway providers), and enables\nOpenCode's Exa-backed `websearch`/`webfetch` tools. Whether an agent chooses to\nuse an available research tool remains part of the measured behavior.\n\nThe fx adapter currently requires `webResearch: true`. fx does not yet expose a\nprompt-free way to disable every web tool while retaining unrestricted coding\ntools, so Agent Eval rejects non-research fx runs instead of silently changing\nthe treatment.\n\n### Run research evals with fx\n\nfx runs through Vercel AI Gateway and uses its native `web_search` and\n`web_fetch` tools. Agent Eval pins fx `0.0.5`, verifies the downloaded Linux\nbinary checksum, and captures the supported saved-session JSON transcript.\n\n```typescript\nimport type { ExperimentConfig } from '@vercel/agent-eval';\n\nconst config: ExperimentConfig = {\n  agent: 'vercel-ai-gateway/fx',\n  webResearch: true,\n  disableBundledSkills: true,\n};\n\nexport default config;\n```\n\nThe fx adapter supports Linux x86-64 and arm64 sandboxes. Agent Eval disables\nfx permission prompts inside the disposable sandbox, matching the execution\nmodel used by the other built-in coding agents. fx can self-grade its own runs,\nbut it cannot serve as a pinned judge for a different agent.\n\n### OpenCode model format\n\nOpenCode uses Vercel AI Gateway exclusively. The OpenCode CLI reads models as\n`{providerID}/{modelID}`, where the provider is the CLI's own `vercel` (AI\nGateway) provider — so both of these forms work:\n\n```typescript\nmodel: 'anthropic/claude-sonnet-4'        // canonical gateway id — vercel/ is added automatically\nmodel: 'vercel/anthropic/claude-sonnet-4' // OpenCode's native form — passed verbatim\n```\n\nWhen the prefix was added automatically, `observedModel` is reported back in\nthe request's namespace (`anthropic/claude-sonnet-4`), so requested-vs-observed\ncomparisons hold. Models targeting a provider configured via\n`agentOptions.extraProviders` are passed verbatim.\n\n### Response-only evals\n\nUse `validation: 'none'` for tasks where the important output is the agent's\nanswer rather than changed files passing `EVAL.ts`.\n\n```typescript\nconst config: ExperimentConfig = {\n  agent: 'vercel-ai-gateway/claude-code',\n  model: 'sonnet',\n  validation: 'none',\n  runs: 10,\n  earlyExit: false,\n  brands: [\n    { id: 'vercel', name: 'Vercel', aliases: ['Vercel Platform'], isYourBrand: true },\n    { id: 'netlify', name: 'Netlify' },\n    { id: 'railway', name: 'Railway' },\n  ],\n  onRunComplete: async ({ runData }) => {\n    // Add custom brand/recommendation analysis here.\n    return runData;\n  },\n};\n```\n\nResponse-only fixtures still need `PROMPT.md` and `package.json`, but they do\nnot need `EVAL.ts` or `EVAL.tsx`.\n\n## A/B Testing\n\nThe real power is comparing different approaches. Create multiple experiment configs:\n\n```typescript\n// experiments/control.ts\nimport type { ExperimentConfig } from '@vercel/agent-eval';\n\nconst config: ExperimentConfig = {\n  agent: 'vercel-ai-gateway/claude-code',\n  model: 'opus',\n  runs: 10,\n  earlyExit: false,\n};\n\nexport default config;\n```\n\n```typescript\n// experiments/with-mcp.ts\nimport type { ExperimentConfig } from '@vercel/agent-eval';\n\nconst config: ExperimentConfig = {\n  agent: 'vercel-ai-gateway/claude-code',\n  model: 'opus',\n  runs: 10,\n  earlyExit: false,\n  setup: async (sandbox) => {\n    await sandbox.runCommand('npm', ['install', '-g', '@myframework/mcp-server']);\n    await sandbox.writeFiles({\n      '.claude/settings.json': JSON.stringify({\n        mcpServers: { myframework: { command: 'myframework-mcp' } }\n      })\n    });\n  },\n};\n\nexport default config;\n```\n\n```bash\nnpx @vercel/agent-eval\n```\n\nCompare the results:\n```\ncontrol (baseline):     7/10 passed (70%)\nwith-mcp:              9/10 passed (90%)\n```\n\n| Experiment | Control | Treatment |\n|------------|---------|-----------|\n| MCP impact | No MCP | With MCP server |\n| Model comparison | Haiku | Sonnet / Opus |\n| Documentation | Minimal docs | Rich examples |\n| System prompt | Default | Framework-specific |\n| Tool availability | Read/write only | + custom tools |\n\n## Results\n\nResults are saved to `results/<experiment>/<timestamp>/`:\n\n```\nresults/\n  with-mcp/\n    2026-01-27T10-30-00Z/\n      create-button/\n        summary.json            # Pass rate, fingerprint, classification\n        classification.json     # Cached failure classification (if failed)\n        run-1/\n          result.json           # Individual run result + o11y summary\n          transcript.json       # Parsed/structured agent transcript\n          transcript-raw.jsonl  # Raw agent output (for debugging)\n          outputs/\n            eval.txt            # EVAL.ts test output\n            scripts/\n              build.txt         # npm script output\n          project/              # Agent-generated files (if copyFiles is set)\n            src/\n              Button.tsx        # Files created/modified by the agent\n```\n\n### summary.json\n\nEach eval directory contains a `summary.json` with:\n\n```json\n{\n  \"totalRuns\": 2,\n  \"passedRuns\": 0,\n  \"passRate\": \"0%\",\n  \"meanDuration\": 45.2,\n  \"fingerprint\": \"a1b2c3...\",\n  \"classification\": {\n    \"failureType\": \"infra\",\n    \"failureReason\": \"Rate limited (HTTP 429) — model never ran\"\n  },\n  \"valid\": false\n}\n```\n\nThe `fingerprint` field enables result reuse across runs. The `classification` and `valid` fields appear only for failed evals -- `valid: false` marks non-model failures so they are not reused by fingerprinting and are automatically retried.\n\n### Playground UI\n\nBrowse results in a web-based dashboard:\n\n```bash\nnpx @vercel/agent-eval playground\n```\n\nThis opens a local Next.js app with:\n- **Overview** dashboard with stats and recent experiments\n- **Experiment detail** with per-eval pass rates and run results\n- **Transcript viewer** to inspect agent tool calls, thinking, and errors\n- **Compare** two runs side-by-side with pass rate deltas\n\nOptions:\n```bash\nnpx @vercel/agent-eval playground --results-dir ./results --evals-dir ./evals --port 3001\n```\n\n### File Copying\n\nBy default, the framework only saves test outputs and transcripts. Use the `copyFiles` config option to also save the files generated by the agent:\n\n```typescript\nconst config: ExperimentConfig = {\n  copyFiles: 'changed',  // or 'all' or 'none' (default)\n};\n```\n\n**Options:**\n\n- **`none`** (default) — Don't copy any project files, only save outputs and transcripts\n- **`changed`** — Copy only files that were modified, created, or deleted by the agent\n- **`all`** — Copy the complete project including both the original fixture files and agent changes\n\nFiles are saved to `results/<experiment>/<timestamp>/<eval>/run-N/project/`. The framework uses git to track changes.\n\n## Result Reuse\n\nThe framework computes a SHA-256 fingerprint for each (eval, config) pair. The fingerprint covers all eval directory files and result-affecting config including `agent`, `model`, `scripts`, `timeout`, `earlyExit`, `runs`, `webResearch`, `disableBundledSkills`, and a pinned `judge`.\n\nOn subsequent runs, evals with a matching fingerprint and a valid cached result (at least one passing run) are skipped automatically. This means:\n\n- **Adding new evals** -- safe, no existing results to invalidate.\n- **Extending the model array** -- safe, each model gets its own experiment directory.\n- **Changing the `evals` filter** -- safe, the filter is not part of the fingerprint.\n- **Editing an eval file** -- only invalidates that specific eval.\n- **Changing config fields** (agent, model, timeout, etc.) -- invalidates all evals in that experiment.\n\nUse `--force` to bypass fingerprinting and re-run everything. Functions like `setup` and `editPrompt` cannot be hashed, so use `--force` when you change those.\n\nEach result also stores a `contentFingerprint` — a hash of the eval files **only**, independent of config. This separates \"the eval itself changed\" from \"a config field changed.\"\n\n### Carrying forward config-only changes\n\nA benign config change (e.g. bumping `timeout`) changes the combined fingerprint and would otherwise re-run every eval. `agent-eval refingerprint` carries those forward in the cached results **without masking a real eval change**:\n\n```bash\nagent-eval refingerprint            # all experiments\nagent-eval refingerprint cc --dry   # preview one experiment\n```\n\nFor each cached result it compares the eval's current `contentFingerprint` to the stored one: if the content is unchanged it re-stamps the combined fingerprint (the result stays cached); if the content **changed** it leaves the result stale so it re-runs. Opt-in runtime changes such as web research and bundled-skill isolation also store a `reuseCompatibilityFingerprint`; those boundaries are never carried forward. `agent-eval status` reports content or runtime-compatibility changes as work, while benign config-only edits stay cached. Run `refingerprint` after a benign experiment config edit to carry that change into the cache (`run` does this automatically).\n\n### After changing or syncing evals: status → pick what to run\n\nRun `agent-eval` with no arguments. It shows the work, then — in a terminal — lets you multi-select which experiments to run. It never re-runs everything:\n\n```bash\nagent-eval\n```\n```\nEvals needing work:\n  new      agent-026-no-serial-await\n  changed  agent-024-avoid-redundant-usestate\n\nWork to do — 6 run(s) across 3 experiment(s):\n  claude-opus-4.6      2 to run  (22 up to date)\n  ...\n\nPick experiments to run:\n   1  claude-opus-4.6\n   2  claude-sonnet-4.6\nNumbers (e.g. 1,3), \"all\", or Enter to skip:\n```\n\nStatus classifies each eval by **content**, so a benign config change (e.g. pinning a judge) is never reported as work. The same building blocks work non-interactively:\n\n```bash\nagent-eval status                  # read-only: what's new/changed, per experiment\nagent-eval status --check          # exit non-zero if anything is new/changed (simple CI gate)\nagent-eval status --json           # machine-readable, for custom CI policy\nagent-eval run claude-sonnet-4.6   # run the named experiment(s) — new/changed evals only\n```\n\n**Accepting staleness is the consumer's call, not the framework's.** `agent-eval` only *reports* — it has no `keep`/`acknowledge`. If you want to leave some experiments on an older eval while keeping others fresh, do that in your own CI: read `agent-eval status --json` (per-experiment `new`/`changed`) and fail only on experiments not in your accepted-stale list. (See next-evals-oss's `scripts/check-stale.mjs` for an example.)\n\n> `refingerprint` (carry config-only changes forward) runs automatically inside `run`; your sync script should call `agent-eval refingerprint` after pulling evals so committed results pick up benign config changes without re-running.\n\n## Failure Classification\n\nWhen evals fail, the framework optionally classifies each failure as one of:\n\n- **model** -- the agent tried but wrote incorrect code\n- **infra** -- infrastructure broke (API errors, rate limits, crashes)\n- **timeout** -- the run hit its time limit\n\nClassification uses Claude Sonnet 4.5 via the Vercel AI Gateway with sandboxed read-only tools to inspect result files. This requires `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN` to be set.\n\n### Classifier Status\n\n- **Enabled** (with `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN`): Classifications are cached in `classification.json`. Non-model failures are removed by default so they can be re-run; pass `--ack-failures` to keep them as final results.\n- **Disabled** (without keys): The classifier is skipped. All results are preserved as-is. Housekeeping will not remove non-model failures (only incomplete and duplicate results). Add `AI_GATEWAY_API_KEY` to `.env` to enable the classifier.\n\n## Housekeeping\n\nAfter each experiment completes, the framework automatically:\n- Removes duplicate results for the same eval (keeps the newest)\n- Removes incomplete results (missing `summary.json` or transcripts)\n- Removes empty timestamp directories\n\n## Environment Variables\n\nEvery run requires an API key for the agent and a token for the sandbox. Classifier is optional.\n\n| Variable             | Required when                          | Description                                                                                  |\n|----------------------|----------------------------------------|----------------------------------------------------------------------------------------------|\n| `AI_GATEWAY_API_KEY` | `vercel-ai-gateway/` agents or classifier | Vercel AI Gateway key -- required for `vercel-ai-gateway/` agents and failure classification |\n| `ANTHROPIC_API_KEY`  | `agent: 'claude-code'`                 | Direct Anthropic API key                                                                     |\n| `OPENAI_API_KEY`     | `agent: 'codex'`                       | Direct OpenAI API key                                                                        |\n| `GEMINI_API_KEY`     | `agent: 'gemini'`                      | Direct Google Gemini API key                                                                 |\n| `CURSOR_API_KEY`     | `agent: 'cursor'`                      | Direct Cursor API key                                                                        |\n| `VERCEL_TOKEN`       | Always (pick one)                      | Vercel personal access token -- for local dev                                                |\n| `VERCEL_OIDC_TOKEN`  | Always (pick one) OR for classifier    | Vercel OIDC token -- for CI/CD pipelines, or enables classifier without `AI_GATEWAY_API_KEY` |\n\nThe **classifier is optional**: if neither `AI_GATEWAY_API_KEY` nor `VERCEL_OIDC_TOKEN` is set, failure classification is skipped and all results are preserved as-is. Set either key to enable the classifier, which automatically identifies and removes non-model failures (infrastructure errors, rate limits, timeouts).\n\nOpenCode and fx only support Vercel AI Gateway\n(`vercel-ai-gateway/opencode` and `vercel-ai-gateway/fx`). There are no direct\nAPI variants for these agents.\n\n### Setup\n\nThe `init` command generates a `.env.example` file. Copy it and fill in your keys:\n\n```bash\ncp .env.example .env\n```\n\nThe framework loads `.env.local` first, then `.env` as a fallback, via [dotenv](https://github.com/motdotla/dotenv).\n\n### Vercel AI Gateway (recommended)\n\nOne key for all models:\n\n```bash\nAI_GATEWAY_API_KEY=your-ai-gateway-api-key\nVERCEL_TOKEN=your-vercel-token\n```\n\n### Direct API keys (no Vercel account required)\n\nIf you don't have a Vercel account, use provider API keys directly:\n\n```bash\nANTHROPIC_API_KEY=sk-ant-...      # For Claude Code\nOPENAI_API_KEY=sk-proj-...        # For Codex\n```\n\nAnd choose ONE sandbox option (no Vercel key needed):\n\n```bash\n# Option 1: Use Docker (free, no account needed)\n# Just set sandbox: 'docker' in your experiment config, that's it!\n\n# Option 2: Use Vercel (requires free account)\nVERCEL_TOKEN=your-vercel-token\n```\n\n#### Minimal setup example\n\nClaude Code via direct API with Docker sandbox:\n\n```typescript\n// experiments/my-eval.ts\nimport type { ExperimentConfig } from '@vercel/agent-eval';\n\nconst config: ExperimentConfig = {\n  agent: 'claude-code',  // Direct API (not vercel-ai-gateway/...)\n  model: 'opus',\n  runs: 1,\n  sandbox: 'docker',     // No VERCEL_TOKEN needed\n};\n\nexport default config;\n```\n\nThen just set:\n```bash\nANTHROPIC_API_KEY=sk-ant-...\n```\n\nThat's it! The classifier will be disabled (since you don't have `AI_GATEWAY_API_KEY`), but all features work fine — you'll just see a warning that non-model failure classification is skipped.\n\n## Tips\n\n**Start with `--dry`**: Always preview before running to verify your config and avoid unexpected costs.\n\n**Use `--smoke` first**: Verify API keys, model IDs, and sandbox connectivity before launching a full run.\n\n**Use multiple runs**: Single runs don't tell you reliability. Use `runs: 10` and `earlyExit: false` for meaningful data.\n\n**Isolate variables**: Change one thing at a time between experiments. Don't compare \"Opus with MCP\" to \"Haiku without MCP\".\n\n**Test incrementally**: Start with simple tasks, add complexity as you learn what works.\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for development workflow and release process.\n\n## License\n\nMIT\n",
  "bytes": 33179,
  "sha": "38d0ca1abef64df50ce891746fa9f5b038a88c4adf7e2a9227b9475b33a95a39",
  "repo_slug": "vercel-labs/agent-eval",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/skl_vercel_labs_agent_eval_frontend_design_efa20fcf/readme"
}