{
  "markdown": "<div align=\"center\">\n\n# bench\n\n**See what the best agents do differently.**\n\nOne line of code. Live dashboard, public profile, README badge.\n\n[![Live](https://img.shields.io/badge/live-bench.virajmishratakehome.workers.dev-e50914?style=for-the-badge)](https://bench.virajmishratakehome.workers.dev)\n[![npm](https://img.shields.io/npm/v/%40virajmishra1%2Fbench-sdk?style=for-the-badge&label=npm)](https://www.npmjs.com/package/@virajmishra1/bench-sdk)\n[![PyPI](https://img.shields.io/pypi/v/bench-observe?style=for-the-badge&label=PyPI)](https://pypi.org/project/bench-observe/)\n[![License](https://img.shields.io/badge/license-MIT-22c55e?style=for-the-badge)](./LICENSE)\n[![Built on](https://img.shields.io/badge/built_on-Cloudflare-f38020?style=for-the-badge)](https://workers.cloudflare.com/)\n\n<img src=\"./assets/hero.svg\" alt=\"Bench dashboard — live event stream, task history, eval scores, README badge\" width=\"100%\"/>\n\n</div>\n\n---\n\n## What is this?\n\nYou built an AI agent. You ran it a few times. But you have no idea if it's actually working well — which tasks fail silently, what it costs per run, or how it compares to anything else.\n\nBench fixes that. Wrap your agent with one function call. You get:\n\n- A public profile page showing runs, success rate, cost, and latency\n- An auto-score on every task (0–1, LLM-as-judge)\n- AI-generated summaries of your failure patterns\n- A README badge that stays live and updates as your agent runs\n- A public leaderboard so anyone can discover your agent\n\nIt's like GitHub for agents — observable, shareable, and public by default.\n\nWant to see it before signing up? Try the sandbox at [/try](https://bench.virajmishratakehome.workers.dev/try) — no signup needed.\n\n---\n\n## Setup (3 minutes)\n\nSign in at [bench.virajmishratakehome.workers.dev](https://bench.virajmishratakehome.workers.dev) with GitHub. The dashboard gives you a copyable setup bundle — install command, API key, and first task template. It listens for your first event and links straight to your profile when it arrives.\n\nOr do it manually:\n\n```bash\nnpm install @virajmishra1/bench-sdk\nexport BENCH_KEY=\"bk_...\"\n```\n\n```typescript\nimport { observe } from \"@virajmishra1/bench-sdk\";\n\nconst agent = observe({ apiKey: process.env.BENCH_KEY, agent: \"my-agent\" });\n\nawait agent.task(\"search\", { query }, async (t) => {\n  const result = await doSearch(query);\n  t.log(\"found\", result.length);\n  t.cost(0.004);\n  return result;\n});\n```\n\nThat's the whole SDK. Everything else is optional.\n\n**Python:**\n\n```bash\npip install bench-observe\nexport BENCH_KEY=\"bk_...\"\n```\n\n```python\nimport bench\n\nagent = bench.observe(api_key=os.environ[\"BENCH_KEY\"], agent=\"my-agent\")\n\nasync with agent.task_ctx(\"search\", {\"query\": query}) as task:\n    result = await do_search(query)\n    task.log(\"found\", len(result))\n    task.set_output(result)\n```\n\n**Already on OpenTelemetry?** Point your exporter at Bench instead:\n\n```bash\nexport OTEL_EXPORTER_OTLP_ENDPOINT=https://bench.virajmishratakehome.workers.dev\nexport OTEL_EXPORTER_OTLP_PROTOCOL=http/json\nexport OTEL_EXPORTER_OTLP_HEADERS=\"X-Bench-Key=bka_...,X-Bench-Agent=my-agent\"\n```\n\nBench understands standard `gen_ai.*` spans — `invoke_agent`, `execute_tool`, `chat`, `retrieval`, and more.\n\n**Prefer a CLI?** The stack-detecting CLI auto-instruments OpenAI, Anthropic, Vercel AI SDK, Mastra, and LangChain:\n\n```bash\nnpx @virajmishra1/bench-cli init --install\nnpx @virajmishra1/bench-cli login\n```\n\n---\n\n## What you get\n\n| Feature | Description |\n|---|---|\n| **Live dashboard** | Real-time event stream while your agent runs. WebSocket, zero polling. |\n| **Public profile** | `/u/you/your-agent` — shareable, OG-image ready, server-rendered |\n| **README badge** | Live SVG badge. Updates automatically. GitHub camo-friendly. |\n| **LLM eval** | Every task auto-scored 0–1 by a Llama 3.3 70B judge. Score logic is [open](./benchmarks/eval-prompts.md). |\n| **Failure insights** | k-means clustering + LLM description of what keeps going wrong |\n| **Leaderboard** | Browse public agents by runs, success rate, eval score, or cost |\n| **Compare** | `/vs/@a/agent1/@b/agent2` — side-by-side quality, cost, latency |\n| **Benchmarks** | Versioned benchmark suites with repeated runs and evidence trails. Separate from self-reported telemetry. |\n| **MCP discovery** | Public read-only MCP server — `search_agents`, `get_agent`, `list_benchmarks` |\n| **Embed widget** | `<iframe>`-ready mini-dashboard, 3 sizes, dark/light |\n| **Privacy controls** | Hide inputs/outputs, make agents private, per-key access |\n| **Permissioned reuse** | Publish capabilities with deny-by-default policies and quotas |\n\n---\n\n## Framework adapters\n\nDrop-in wrappers that auto-instrument your existing LLM calls:\n\n```typescript\n// Anthropic — wraps every messages.create() call\nimport { wrapAnthropic } from \"@virajmishra1/bench-anthropic\";\nconst client = wrapAnthropic(new Anthropic(), bench);\n\n// OpenAI — wraps chat completions, responses, and embeddings\nimport { wrapOpenAI } from \"@virajmishra1/bench-openai\";\nconst client = wrapOpenAI(new OpenAI(), bench);\n\n// Vercel AI SDK — wraps generateText / streamText / generateObject\nimport { track } from \"@virajmishra1/bench-vercel-ai\";\nconst result = await track(bench, \"summarize\", () =>\n  generateText({ model: anthropic(\"claude-sonnet-4-6\"), prompt: \"...\" })\n);\n\n// Mastra\nimport { wrapMastra } from \"@virajmishra1/bench-mastra\";\n```\n\n---\n\n## Let your AI find agents\n\nBench exposes a public MCP server at `/mcp`. Connect it to Claude Code:\n\n```bash\nclaude mcp add --transport http bench https://bench.virajmishratakehome.workers.dev/mcp\n```\n\nOr Codex:\n\n```bash\ncodex mcp add bench --url https://bench.virajmishratakehome.workers.dev/mcp\n```\n\nTools available: `search_agents`, `get_agent`, `list_benchmarks`. Search returns only public agents. Owner telemetry and benchmark evidence are labeled separately.\n\nSee [MCP.md](./MCP.md) for full tool schemas and the privacy model.\n\n---\n\n## Architecture\n\nBench runs entirely on Cloudflare. Each product is doing a specific job:\n\n```\nSDK (npm: @virajmishra1/bench-sdk)\n        |  batched events, X-Bench-Key\n        v\nPOST /ingest                              <- Workers (Hono)\n        |\n        +-> D1 --- users, agents, tasks, events\n        |\n        +-> AgentDO --- one Durable Object per agent\n        |           +- ring buffer (last 1k events, SQLite in DO storage)\n        |           +- latency histogram (p50, p95)\n        |           +- hibernating WebSocket -> live dashboards\n        |\n        +-> EvalWorkflow --- runs per task.end\n        |           +- Workers AI (Llama 3.3 70B) -> score 0-1 + reasoning\n        |              -> writes back to D1.tasks\n        |              -> updates agents.avg_eval_score\n        |\n        +-> ClusterWorkflow --- on-demand + hourly cron\n                    +- k-means on task embeddings -> cluster labels\n                       -> Workers AI LLM describes each cluster\n                       -> stored in agents.failure_clusters\n\nPublic surfaces:\n  /u/:login/:slug          -> profile page (server-rendered, OG image)\n  /badge/:login/:slug.svg  -> README badge (KV-cached 60s)\n  /embed/:login/:slug      -> iframe widget (3 sizes, dark/light)\n  /leaderboard             -> discovery (5 sort modes)\n  /vs/:a/:b                -> compare two agents\n  /try                     -> sandbox (no signup)\n  /benchmarks              -> verified benchmark registry\n  /mcp                     -> read-only MCP server\n  /api/agents/:l/:s/insights -> failure pattern analysis (JSON)\n```\n\nThe key design decision is the **actor model**: every agent gets its own Durable Object. That DO holds the last 1,000 events in SQLite, a latency histogram, and a hibernating WebSocket connection — zero idle cost, no polling.\n\n### Cloudflare products used\n\n| Product | Role |\n|---|---|\n| **Workers** | API, profile rendering, badge generation |\n| **Durable Objects** | One per agent — ring buffer, latency histogram, hibernating WebSocket |\n| **D1** | Users, agents, tasks, events |\n| **KV** | Token lookup cache, badge SVG cache, OG image cache |\n| **Workers AI** | Llama 3.3 70B — LLM judge + failure pattern descriptions |\n| **Workflows** | Durable retry for EvalWorkflow and ClusterWorkflow |\n| **Browser Rendering** | OG share images (SVG → PNG) |\n| **Assets** | Static frontend (landing, dashboard, JS, CSS) |\n\n---\n\n## SDK reference\n\n```typescript\nconst agent = observe({\n  apiKey: string;           // bk_xxx — from your dashboard\n  agent: string;            // slug, e.g. \"my-agent\"\n  displayName?: string;\n  endpoint?: string;        // default: bench.virajmishratakehome.workers.dev\n  flushIntervalMs?: number; // default: 2000\n  maxBatchSize?: number;    // default: 50\n});\n\n// Wrap a task — records start/end/duration/status/eval automatically\nawait agent.task(\"name\", input, async (task) => {\n  task.log(\"label\", value);    // attach a log event\n  task.cost(0.003);            // report LLM spend (owner-reported)\n  return result;               // returned value becomes the task output\n});\n\n// Fire a custom event\nagent.event(\"custom\", { key: \"value\" });\n\n// Flush immediately (auto-runs on batch full or interval)\nawait agent.flush();\n```\n\n`task.cost()` calls are labeled \"owner-reported\" in the UI. Framework adapters attach provider and token evidence, labeled separately.\n\nErrors are swallowed silently — observability should never crash your agent.\n\n---\n\n## Self-host\n\n```bash\ngit clone https://github.com/VirajMishra1/bench\ncd bench && npm install\n\ncd packages/worker\nnpx wrangler login\n\n# Create infrastructure\nnpx wrangler d1 create bench-db\nnpx wrangler kv namespace create CACHE\nnpx wrangler kv namespace create SESSIONS\n\n# Paste the returned IDs into wrangler.jsonc, then:\nnpx wrangler secret put SESSION_SECRET             # any random 32+ char string\nnpx wrangler secret put GITHUB_OAUTH_CLIENT_SECRET # from github.com/settings/developers\n\n# Apply schema and deploy\nnpm run db:remote\nnpm run deploy\n```\n\n---\n\n## File layout\n\n```\nbench/\n+-- packages/\n|   +-- sdk/                   <- @virajmishra1/bench-sdk\n|   +-- adapters/\n|   |   +-- anthropic/         <- @virajmishra1/bench-anthropic\n|   |   +-- openai/            <- @virajmishra1/bench-openai\n|   |   +-- vercel-ai/         <- @virajmishra1/bench-vercel-ai\n|   |   +-- mastra/            <- @virajmishra1/bench-mastra\n|   |   +-- langchain/         <- bench-langchain\n|   +-- worker/                <- Cloudflare Worker (all backend + frontend)\n|       +-- src/\n|       |   +-- index.ts       <- Hono routes\n|       |   +-- ingest.ts      <- POST /ingest\n|       |   +-- profile.ts     <- public profile page\n|       |   +-- badge.ts       <- SVG README badge\n|       |   +-- embed.ts       <- iframe widget\n|       |   +-- leaderboard.ts <- discovery page\n|       |   +-- compare.ts     <- /vs/:a/:b\n|       |   +-- do/agent.ts    <- AgentDO (actor per agent)\n|       |   +-- workflows/\n|       |       +-- eval.ts    <- LLM judge per task\n|       |       +-- cluster.ts <- failure clustering\n|       +-- public/            <- landing, dashboard, styles\n|       +-- migrations/        <- D1 schema history\n+-- benchmarks/\n|   +-- grounded-research-v1/  <- example benchmark suite + cases\n|   +-- eval-prompts.md        <- open-source judge prompts\n+-- examples/                  <- runnable example agents\n```\n\n---\n\n## License\n\nMIT — see [LICENSE](./LICENSE)\n\nBuilt by [@virajm1shra](https://x.com/virajm1shra) on Cloudflare.\n",
  "bytes": 11453,
  "sha": "626c5b621765d9c97dbd21eea0d3e7036d61d27bfab22c91c3f3163217992880",
  "repo_slug": "virajmishra1/bench",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_virajmishra1_bench_3faf766e/readme"
}