Author verifiable eval records through a draft → review → revise → submit loop with server-enforced graders; compile to JSONL/CSV/Inspect/lm-eval via MCP. STDIO or Streamable HTTP.\n
\n\n[](https://github.com/cyanheads/evals-mcp-server/releases/latest/download/evals-mcp-server.mcpb) [](https://cursor.com/en/install-mcp?name=evals-mcp-server&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBjeWFuaGVhZHMvZXZhbHMtbWNwLXNlcnZlciJdfQ==) [](https://vscode.dev/redirect?url=vscode:mcp/install?%7B%22name%22%3A%22evals-mcp-server%22%2C%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40cyanheads%2Fevals-mcp-server%22%5D%7D)\n\n[](https://www.npmjs.com/package/@cyanheads/mcp-ts-core)\n\n
\n\n---\n\n## Tools\n\nNine tools for authoring eval records — the draft loop (create, revise, discard, submit), the standalone deterministic checker, and read/list/export:\n\n| Tool | Description |\n|:---|:---|\n| `evals_describe_schema` | Return the required and optional fields plus grader options for a task type. Call before drafting. |\n| `evals_create_draft` | Create a draft eval record carrying its own grader; returns the parsed record, a review protocol, and a verification subagent prompt. |\n| `evals_get_record` | Read a draft or submitted record by id; the id is stable across submit. |\n| `evals_revise_draft` | Apply a surgical `set` / `append` / `unset` patch to a draft by dotted path; re-runs the self-consistency check. |\n| `evals_discard_draft` | Delete a draft record by id. Draft-only. |\n| `evals_run_check` | Run a grader spec against candidate answers and get PASS/REJECT per candidate, decoupled from any saved record. |\n| `evals_submit_draft` | Finalize a draft through the committability gate, then freeze it. |\n| `evals_list_records` | Browse and filter records by status, domain, task type, or tag. Returns a compact summary per record. |\n| `evals_export_records` | Compile submitted records to JSONL, CSV, Inspect AI, or lm-evaluation-harness and write the artifact under `exports/`. |\n\n### `evals_describe_schema`\n\nReturn what a record of a given `task_type` needs before you draft it.\n\n- Static — derived from the record and grader Zod schemas, no disk or runtime state\n- Per-type gold shape, appropriate grader kind(s), required/optional fields, and authoring notes\n- `task_type` is one of `numeric`, `exact_answer`, `set_answer`, `mcq`, `regex_answer`, `json_answer`, `free_response`\n\n---\n\n### `evals_create_draft`\n\nCreate and persist a draft eval record, then reflect it back as a review forcing function.\n\n- Validates against the `task_type` discriminated union (per-type rules: `mcq` requires `choices`; `free_response` requires an `llm_rubric` grader)\n- Runs a cheap self-consistency check — grader vs gold and each positive must PASS, vs each negative must REJECT\n- Returns the normalized record parroted back behind a divider, a per-field review protocol, a ready-to-paste verification subagent prompt, and what's still required before submit\n- Optional draft-time `verification` block and `captures` (EvalsIDs) when you already hold provenance\n- Stays `draft` — passing self-consistency proves the grader discriminates, not that the gold is right\n\n---\n\n### `evals_revise_draft`\n\nSurgically patch a draft so each change stays legible.\n\n- Explicit `set` (dotted-path → value), `append` (dotted-path → array items), and `unset` (dotted paths) operations — not full-record rewrites\n- Returns the updated record, an itemized list of what changed, and a re-run self-consistency verdict\n- Re-validates the full shape and cross-field constraints after the patch\n- Draft-only — submitted records are frozen; `task_type` cannot be patched (start a new draft to change the discriminant)\n\n---\n\n### `evals_run_check`\n\nRun a grader against one or more candidates without touching a saved record.\n\n- PASS/REJECT per candidate plus the resolved comparison value (e.g. the math.js-evaluated numeric target), so you see why each matched or missed\n- `candidates` accepts strings, numbers, objects, or arrays — whatever the grader kind expects\n- Supply `gold` for gold-relative kinds (`exact_match`); it is a no-op for target-embedding kinds like `numeric` and `mcq`\n- `llm_rubric` cannot run on this server — submission relies on recorded independent verification\n\n---\n\n### `evals_submit_draft`\n\nFinalize a draft through the committability gate, then freeze it.\n\n- The gate runs the grader against the gold (must PASS), requires ≥1 declared negative case to be REJECTED, and requires a recorded, decorrelated independent verification that agrees with the gold\n- Resolves and embeds any `captures` from `EVALS_CAPTURE_DIR`, cross-checking the gold against the authoritative captured value\n- Rejects duplicates (same `content_hash` already submitted)\n- On pass, flips the record to `submitted`, stamps `submitted_at` and a `checksum`, and freezes it; otherwise refuses with a typed error and the record stays a draft\n- `free_response` `llm_rubric` is admitted on recorded independent verification and flagged `server_verified: false`\n\n---\n\n### `evals_export_records`\n\nCompile submitted records to a downstream eval format.\n\n- `jsonl` (lossless, the lingua franca), `csv` (a flattened, lossy spreadsheet summary), `inspect` (UK AISI Inspect AI), `lm-eval` (EleutherAI lm-evaluation-harness)\n- Optional `domain` / `task_type` / `tag` filter\n- Only submitted records are exported — drafts are skipped\n- Writes the artifact under `exports/` and returns the file path, record count, byte size, and a short preview instead of dumping inline\n\n## Resource\n\n| Type | Name | Description |\n|:---|:---|:---|\n| Resource | `eval://record/{id}` | A single draft or submitted record by id — the same payload `evals_get_record` returns, for resource-capable clients. |\n\nAll record data is also reachable through the tool surface — `evals_get_record` for a single record, `evals_list_records` to browse. The resource is a convenience mirror for clients that support resources, not the access path.\n\n## Features\n\nBuilt on [`@cyanheads/mcp-ts-core`](https://www.npmjs.com/package/@cyanheads/mcp-ts-core):\n\n- Declarative tool and resource definitions — single file per primitive, framework handles registration and validation\n- Unified error handling — handlers throw, framework catches, classifies, and formats\n- Typed error contracts — tools declare their domain failures (`reason` + recovery), surfaced to the agent\n- Pluggable auth: `none`, `jwt`, `oauth`\n- Structured logging with optional OpenTelemetry tracing\n- STDIO and Streamable HTTP transports\n\nEval authoring:\n\n- A `draft → review → surgical-revise → submit` loop, with the server acting as both scribe (normalize, persist, compile) and adversarial checker (run the record's own grader, reject what doesn't hold up)\n- Records are a Zod `discriminatedUnion` keyed on `task_type` — `numeric`, `exact_answer`, `set_answer`, `mcq`, `regex_answer`, `json_answer`, `free_response`\n- A typed grader DSL serialized with each record — deterministic kinds (`numeric` via math.js, `exact_match`, `set_match`, `regex`, `mcq`, `json_match`) run server-side; `llm_rubric` relies on recorded independent verification\n- An enforced committability gate at submit: the gold must pass its own grader, ≥1 negative must be rejected, and a recorded decorrelated verification must agree with the gold\n- Optional fleet grounding via the `captures` EvalsID field — link framework-written tool-call dumps, resolved from `EVALS_CAPTURE_DIR` and cross-checked against the gold (no server-to-server calls)\n- Plain JSON files under `EVALS_DATA_DIR` — inspectable, diffable, version-controllable records\n- Compile to JSONL, CSV, Inspect AI, and lm-evaluation-harness formats\n\nAgent-friendly output:\n\n- The two instructional tools (`evals_create_draft`, `evals_revise_draft`) carry the loop's review mechanism in their responses — the parsed record parroted back, a per-field review protocol, and a ready subagent prompt\n- Self-consistency verdicts on every draft and revise — per-positive and per-negative results, not just a boolean\n- `evals_list_records` discloses truncation when the limit is hit, so a partial set is never mistaken for the whole corpus\n- The submit gate refuses with a typed `reason` + recovery hint, so a rejected record tells the agent exactly what to fix\n\n## Getting started\n\nAdd the following to your MCP client configuration file. Set `EVALS_DATA_DIR` to a writable folder — the server manages `drafts/`, `submitted/`, and `exports/` under it.\n\n```json\n{\n \"mcpServers\": {\n \"evals-mcp-server\": {\n \"type\": \"stdio\",\n \"command\": \"bunx\",\n \"args\": [\"@cyanheads/evals-mcp-server@latest\"],\n \"env\": {\n \"MCP_TRANSPORT_TYPE\": \"stdio\",\n \"MCP_LOG_LEVEL\": \"info\",\n \"EVALS_DATA_DIR\": \"/absolute/path/to/evals-data\"\n }\n }\n }\n}\n```\n\nOr with npx (no Bun required):\n\n```json\n{\n \"mcpServers\": {\n \"evals-mcp-server\": {\n \"type\": \"stdio\",\n \"command\": \"npx\",\n \"args\": [\"-y\", \"@cyanheads/evals-mcp-server@latest\"],\n \"env\": {\n \"MCP_TRANSPORT_TYPE\": \"stdio\",\n \"MCP_LOG_LEVEL\": \"info\",\n \"EVALS_DATA_DIR\": \"/absolute/path/to/evals-data\"\n }\n }\n }\n}\n```\n\nOr with Docker:\n\n```json\n{\n \"mcpServers\": {\n \"evals-mcp-server\": {\n \"type\": \"stdio\",\n \"command\": \"docker\",\n \"args\": [\n \"run\", \"-i\", \"--rm\",\n \"-e\", \"MCP_TRANSPORT_TYPE=stdio\",\n \"-e\", \"EVALS_DATA_DIR=/data\",\n \"-v\", \"evals-data:/data\",\n \"ghcr.io/cyanheads/evals-mcp-server:latest\"\n ]\n }\n }\n}\n```\n\nFor Streamable HTTP, set the transport and start the server:\n\n```sh\nMCP_TRANSPORT_TYPE=http MCP_HTTP_PORT=3010 EVALS_DATA_DIR=./evals-data bun run start:http\n# Server listens at http://localhost:3010/mcp\n```\n\n### Prerequisites\n\n- [Bun v1.4.0](https://bun.sh/) or higher (or Node.js v24+).\n- A writable directory for `EVALS_DATA_DIR`. No external API key is required.\n\n### Installation\n\n1. **Clone the repository:**\n\n```sh\ngit clone https://github.com/cyanheads/evals-mcp-server.git\n```\n\n2. **Navigate into the directory:**\n\n```sh\ncd evals-mcp-server\n```\n\n3. **Install dependencies:**\n\n```sh\nbun install\n```\n\n4. **Configure environment:**\n\n```sh\ncp .env.example .env\n# edit .env and set EVALS_DATA_DIR\n```\n\n## Configuration\n\nAll server configuration is validated at startup via Zod schemas in `src/config/server-config.ts`.\n\n| Variable | Description | Default |\n|:---|:---|:---|\n| `EVALS_DATA_DIR` | Root folder for record JSON; the store manages `drafts/`, `submitted/`, and `exports/` under it. | `./evals-data` |\n| `EVALS_REQUIRE_CONFIRMATION` | When `true`, `evals_submit_draft` requests human confirmation through multi-round input before finalizing. | `false` |\n| `EVALS_DEFAULT_LICENSE` | Default `metadata.license` applied when a draft omits one (e.g. `CC-BY-4.0`). | — |\n| `EVALS_CAPTURE_DIR` | Directory of framework-written tool-call captures; when set, `captures` EvalsIDs resolve to full dumps. | — |\n| `MCP_TRANSPORT_TYPE` | Transport: `stdio` or `http`. | `stdio` |\n| `MCP_HTTP_PORT` | Port for the HTTP server. | `3010` |\n| `MCP_AUTH_MODE` | Auth mode: `none`, `jwt`, or `oauth`. | `none` |\n| `MCP_LOG_LEVEL` | Log level (RFC 5424). | `info` |\n| `OTEL_ENABLED` | Enable [OpenTelemetry instrumentation](https://github.com/cyanheads/mcp-ts-core/tree/main/docs/telemetry). | `false` |\n\nSee [`.env.example`](./.env.example) for the full list of optional overrides.\n\n## Running the server\n\n### Local development\n\n- **Build and run:**\n\n ```sh\n # One-time build\n bun run rebuild\n\n # Run the built server\n bun run start:stdio\n # or\n bun run start:http\n ```\n\n- **Run checks and tests:**\n\n ```sh\n bun run devcheck # Lint, format, typecheck, security\n bun run test # Vitest test suite\n bun run lint:mcp # Validate MCP definitions against spec\n ```\n\n### Docker\n\n```sh\ndocker build -t evals-mcp-server .\ndocker run --rm -e MCP_TRANSPORT_TYPE=stdio -e EVALS_DATA_DIR=/data -v evals-data:/data evals-mcp-server\n```\n\nThe Dockerfile defaults to HTTP transport, stateless session mode, and logs to `/var/log/evals-mcp-server`. OpenTelemetry peer dependencies are installed by default — build with `--build-arg OTEL_ENABLED=false` to omit them.\n\n## Project structure\n\n| Directory | Purpose |\n|:---|:---|\n| `src/index.ts` | `createApp()` entry point — registers tools and the resource, inits the record-store and exporter services. |\n| `src/config` | Server-specific environment variable parsing and validation with Zod. |\n| `src/mcp-server/tools` | Tool definitions (`*.tool.ts`). |\n| `src/mcp-server/resources` | Resource definitions (`*.resource.ts`). |\n| `src/services/eval-record` | The record schema, draft builder, and submit gate. |\n| `src/services/grader` | Deterministic grader DSL execution and the committability check. |\n| `src/services/record-store` | On-disk JSON record CRUD, the draft→submitted move, and export writes. |\n| `src/services/exporter` | Compiling submitted records to JSONL/CSV/Inspect/lm-eval. |\n| `tests/` | Unit and integration tests mirroring `src/`. |\n\n## Development guide\n\nSee [`CLAUDE.md`/`AGENTS.md`](./CLAUDE.md) for development guidelines and architectural rules. The short version:\n\n- Handlers throw, framework catches — no `try/catch` in tool logic\n- Use `ctx.log` for request-scoped logging; records persist to disk via the `record-store` service, not `ctx.state`\n- Register new tools and resources in the `createApp()` arrays in `src/index.ts`\n- The server is the source of truth — validate inputs, run the grader as a hard gate, and never admit a record on assertion alone\n\n## Contributing\n\nIssues and pull requests are welcome. Run checks and tests before submitting:\n\n```sh\nbun run devcheck\nbun run test\n```\n\n## License\n\nApache-2.0 — see [LICENSE](LICENSE) for details.\n",
"bytes": 15300,
"sha": "51716cdef4d21a24430a52f9e9c13154364ba495b6d0a92e24137e0f6e667c2e",
"repo_slug": "cyanheads/evals-mcp-server",
"fonte": "repo",
"truncated": false,
"api": "https://agentalog.com/api/listings/mcp_io_github_cyanheads_evals_mcp_server_aa51cf57/readme"
}