{
  "markdown": "# LLMTest MCP Server\n\n<!-- mcp-name: io.github.tjacquesson/llmtest-mcp -->\n\n[![npm version](https://img.shields.io/npm/v/llmtest-mcp.svg)](https://www.npmjs.com/package/llmtest-mcp)\n[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)\n\nMCP server that benchmarks AI models on your actual prompts and finds cheaper, faster alternatives. Works with Claude Code, Cursor, Windsurf, and any MCP-compatible tool.\n\n## Quick Start\n\n### 1. Get your API key\n\nSign up at [llmtest.io](https://llmtest.io) and grab your API key from the dashboard.\n\n### 2. Add to your tool\n\n**Claude Code:**\n\n```bash\nclaude mcp add llmtest -- npx llmtest-mcp\n```\n\nThen set your key:\n\n```bash\nexport LLMTEST_API_KEY=llmt_your_key_here\n```\n\n**Cursor / Windsurf / Other MCP clients:**\n\nAdd to your MCP config file:\n\n```json\n{\n  \"mcpServers\": {\n    \"llmtest\": {\n      \"command\": \"npx\",\n      \"args\": [\"llmtest-mcp\"],\n      \"env\": {\n        \"LLMTEST_API_KEY\": \"llmt_your_key_here\"\n      }\n    }\n  }\n}\n```\n\n### 3. Talk to your AI\n\nJust ask in natural language:\n\n- \"Check my LLMTest status\"\n- \"Find cheaper models for my AI calls\"\n- \"Run a benchmark on my blog-writer flow\"\n- \"What models are trending?\"\n\n## How It Works\n\nLLMTest is a proxy that sits between your app and AI providers. Point your app at `https://llmtest.io/v1` instead of calling OpenAI/Anthropic directly, and LLMTest tracks your usage, benchmarks alternatives, and suggests cost savings.\n\nThis MCP server gives your AI assistant access to LLMTest's tools so it can manage everything for you.\n\n## Available Tools\n\n| Tool | Description |\n|------|-------------|\n| `status` | Show proxy status and activity summary |\n| `list_flows` | List all AI flows with cost and latency stats |\n| `get_suggestions` | Get pending model-switch recommendations |\n| `update_suggestion` | Accept or dismiss a suggestion |\n| `run_benchmark` | Benchmark a flow against challenger models |\n| `optimize_prompt` | Rewrite a flow's prompt and find a cheaper model that still works |\n| `seed_samples` | Add test prompts for pre-launch benchmarking |\n| `list_samples` | Show stored test samples per flow |\n| `list_new_models` | Show new and trending models |\n| `get_account` | Check credit balance and usage |\n| `get_autopilot_status` | Check whether autopilot is on and whether the account is eligible |\n| `enable_autopilot` | Turn on weekly auto-optimization with safety gates + drift-based auto-revert |\n| `disable_autopilot` | Turn off autopilot (existing optimizations stay active) |\n| `list_active_optimizations` | List auto-accepted optimizations still inside their 24h revert window |\n| `revert_optimization` | Roll an auto-accepted optimization back to the previous prompt |\n\n## Autopilot\n\nAutopilot automatically optimizes your flows on a weekly cadence. Changes that pass every safety gate go live with a 24-hour revert window. Drift detection keeps checking after that and rolls back if quality slips.\n\n**To enable from your IDE:** ask your AI assistant something like \"enable LLMTest autopilot\". It will call `enable_autopilot`. Use `get_autopilot_status` to confirm prerequisites.\n\n**Prerequisites** (checked per flow each cycle):\n\n- Autopilot enabled on the account\n- Email verified\n- Account age ≥ 14 days (trust ramp)\n- Flow has ≥ 20 real calls in the last 7 days\n- Flow not optimized by autopilot in the last 14 days (cooldown)\n- Positive credit balance (~$1–2 per run)\n\n**Safety gates** (all must pass for auto-accept): 95% CI lower bound > 50% win rate, multi-judge agreement ≥ 80%, ≥ 20% total savings, no length-bias warning, golden-set regression check.\n\n**Revert:** 24h window after auto-accept. After that, only drift detection can roll back.\n\n## Typical Workflow\n\n**Pre-launch (no traffic yet):**\n\n1. Tell your AI: \"I'm building a support chatbot using gpt-4o\"\n2. It seeds realistic test samples with `seed_samples`\n3. It runs `run_benchmark` to compare models\n4. It shows you `get_suggestions` with cheaper alternatives\n\n**Post-launch (with real traffic):**\n\n1. Route your AI calls through `https://llmtest.io/v1`\n2. LLMTest monitors usage and auto-benchmarks when flows hit 50+ calls\n3. Ask \"any cost-saving suggestions?\" to see recommendations\n4. Accept a suggestion and update your code\n\n## Environment Variables\n\n| Variable | Required | Description |\n|----------|----------|-------------|\n| `LLMTEST_API_KEY` | Yes | Your API key from [llmtest.io/dashboard](https://llmtest.io/dashboard) |\n| `LLMTEST_BASE_URL` | No | Custom API URL (defaults to `https://llmtest.io`) |\n\n## Links\n\n- [Website](https://llmtest.io)\n- [Dashboard](https://llmtest.io/dashboard)\n- [Docs](https://llmtest.io/docs)\n- [GitHub](https://github.com/tjacquesson/llmtest-mcp)\n- [npm](https://www.npmjs.com/package/llmtest-mcp)\n\n## License\n\nMIT\n",
  "bytes": 4771,
  "sha": "1a710a22eaac45f0d5e9477656a444ccc8fc6eb74817a76ff8fdcced01faef4b",
  "repo_slug": "tjacquesson/llmtest-mcp",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_tjacquesson_llmtest_mcp_4a4639e7/readme"
}