{
  "markdown": "# MetriLLM\n\n[![CI](https://github.com/MetriLLM/metrillm/actions/workflows/ci.yml/badge.svg)](https://github.com/MetriLLM/metrillm/actions/workflows/ci.yml)\n[![Node.js](https://img.shields.io/badge/node-%3E%3D20-brightgreen)](https://nodejs.org/)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)\n\n[![npm version](https://img.shields.io/npm/v/metrillm)](https://www.npmjs.com/package/metrillm)\n[![npm downloads](https://img.shields.io/npm/dw/metrillm)](https://www.npmjs.com/package/metrillm)\n[![GitHub stars](https://img.shields.io/github/stars/MetriLLM/metrillm)](https://github.com/MetriLLM/metrillm)\n\n**Benchmark your local LLM models in one command.** Speed, quality, hardware fitness — with a shareable score and public leaderboard.\n\n> Think Geekbench, but for local LLMs on your actual hardware.\n\n```bash\nnpm install -g metrillm@latest\nmetrillm bench\n```\n\n<p align=\"center\">\n  <img src=\"docs/images/cli1.png\" width=\"48%\" alt=\"MetriLLM CLI — interactive menu\" />\n  <img src=\"docs/images/cli2.png\" width=\"48%\" alt=\"MetriLLM CLI — hardware detection\" />\n</p>\n\n[![MetriLLM Leaderboard](docs/images/leaderboard.png)](https://metrillm.dev)\n\n## What You Get\n\n- **Performance metrics**: tokens/sec, time to first token, memory usage, load time\n- **Quality evaluation**: reasoning, coding, math, instruction following, structured output, multilingual (14 prompts, 6 categories)\n- **Global score** (0-100): 30% hardware fit + 70% quality\n- **Verdict**: EXCELLENT / GOOD / MARGINAL / NOT RECOMMENDED\n- **One-click share**: `--share` uploads your result and gives you a public URL + leaderboard rank\n\n## Real Benchmark Results\n\n> From the [public leaderboard](https://metrillm.dev) — all results below were submitted with `metrillm bench --share`.\n\n| Model | Machine | CPU | RAM | tok/s | TTFT | Global | Verdict |\n|-------|---------|-----|-----|------:|-----:|-------:|---------|\n| llama3.2:latest | Mac Mini | Apple M4 Pro | 64 GB | 98.9 | 125 ms | 77 | GOOD |\n| mistral:latest | Mac Mini | Apple M4 Pro | 64 GB | 54.3 | 124 ms | 76 | GOOD |\n| gemma3:4b | MacBook Air | Apple M4 | 32 GB | 35.9 | 303 ms | 72 | GOOD |\n| gemma3:1b | MacBook Air | Apple M4 | 32 GB | 39.4 | 362 ms | 72 | GOOD |\n| qwen3:1.7b | MacBook Air | Apple M4 | 32 GB | 37.9 | 3.1 s | 70 | GOOD |\n| llama3.2:3b | MacBook Air | Apple M4 | 32 GB | 27.8 | 285 ms | 69 | GOOD |\n| gemma3:12b | MacBook Air | Apple M4 | 32 GB | 12.3 | 656 ms | 67 | GOOD |\n| phi4:14b | MacBook Air | Apple M4 | 32 GB | 11.1 | 515 ms | 65 | GOOD |\n| mistral:7b | MacBook Air | Apple M4 | 32 GB | 13.6 | 517 ms | 61 | GOOD |\n| deepseek-r1:14b | MacBook Air | Apple M4 | 32 GB | 10.8 | 30.0 s | 25 | NOT RECOMMENDED |\n\n**Key takeaway**: Small models (1-4B) fly on Apple Silicon. Larger models (14B+) with thinking chains can choke even on capable hardware. [See full leaderboard &rarr;](https://metrillm.dev)\n\n## Install\n\n> Requires [Node 20+](https://nodejs.org/) and a local runtime:\n> [Ollama](https://ollama.com/) or [LM Studio](https://lmstudio.ai/).\n\n```bash\n# Install globally\nnpm install -g metrillm@latest\nmetrillm bench\n\n# Alternative package managers\npnpm add -g metrillm@latest\nbun add -g metrillm@latest\n\n# Homebrew\nbrew install MetriLLM/metrillm/metrillm\n\n# Or run without installing\nnpx metrillm@latest bench\n```\n\n## Usage\n\n```bash\n# Interactive mode — pick models from a menu\nmetrillm bench\n\n# Benchmark a specific model\nmetrillm bench --model gemma3:4b\n\n# Benchmark with LM Studio backend\nmetrillm bench --backend lm-studio --model qwen3-8b\n\n# Benchmark all installed models\nmetrillm bench --all\n\n# Share your result (upload + public URL + leaderboard rank)\nmetrillm bench --share\n\n# CI/non-interactive mode\nmetrillm bench --ci-no-menu --share\n\n# Force unload after each model (useful for memory isolation)\nmetrillm bench --all --unload-after-bench\n\n# Export results locally\nmetrillm bench --export json\nmetrillm bench --export csv\n```\n\n## Upload Configuration (CLI + MCP)\n\nBy default, production builds upload shared results to the official MetriLLM leaderboard (`https://metrillm.dev`).\n\n- No CI secret injection is required for standard releases.\n- Local/dev runs use the same default behavior.\n- Self-hosted or staging deployments can override endpoints with:\n  - `METRILLM_SUPABASE_URL`\n  - `METRILLM_SUPABASE_ANON_KEY`\n  - `METRILLM_PUBLIC_RESULT_BASE_URL`\n\nIf these variables are set to placeholder values (from templates), MetriLLM falls back to official defaults.\n\n## Windows Users\n\nPowerShell's default execution policy blocks npm global scripts. If you see `PSSecurityException` or `UnauthorizedAccess` when running `metrillm`, run this once:\n\n```powershell\nSet-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned\n```\n\nAlternatively, use `npx metrillm@latest` which bypasses the issue entirely.\n\n## Runtime Backends\n\n| Backend | Flag | Default URL | Required env |\n|---|---|---|---|\n| Ollama | `--backend ollama` | `http://127.0.0.1:11434` | `OLLAMA_HOST` (optional) |\n| LM Studio | `--backend lm-studio` | `http://127.0.0.1:1234` | `LM_STUDIO_BASE_URL` (optional), `LM_STUDIO_API_KEY` (optional) |\n\nShared runtime env:\n- `METRILLM_STREAM_STALL_TIMEOUT_MS` (optional): stream watchdog for all backends, default `30000`, `0` disables it\n\nLM Studio benchmark runs now use the native REST inference endpoint (`/api/v1/chat`) for both streaming and non-streaming generation.\nThe previous OpenAI-compatible inference path (`/v1/chat/completions`) has been retired from MetriLLM so tok/s and TTFT can rely on native LM Studio stats when available.\nIf a LM Studio response omits native token stats, MetriLLM still computes a score and shows the throughput as `estimated`.\n\nFor very large models, tune timeout flags:\n- `--perf-warmup-timeout-ms` (default `300000`)\n- `--perf-prompt-timeout-ms` (default `120000`)\n- `--quality-timeout-ms` (default `120000`)\n- `--coding-timeout-ms` (default `240000`)\n- `--stream-stall-timeout-ms` (default `30000`, `0` disables stall timeout for any backend)\n\nBenchmark Profile v1 (applied to all benchmark prompts):\n- `temperature=0`\n- `top_p=1`\n- `seed=42`\n- `thinking` follows your benchmark mode (`--thinking` / `--no-thinking`)\n- Context window stays runtime default (`context=runtime-default`) and is recorded as such in metadata.\n\nLM Studio non-thinking guard:\n- When benchmark mode requests non-thinking (`--no-thinking` or default), MetriLLM now aborts if the model still emits reasoning traces (for result comparability).\n- To disable it in LM Studio for affected models, put this at the top of the model chat template: `{%- set enable_thinking = false %}` then eject/reload the model.\n\n## How Scoring Works\n\n**Hardware Fit Score** (0-100) — how well the model runs on your machine:\n- Speed: 50% (tokens/sec relative to your hardware tier)\n- TTFT: 20% (time to first token)\n- Memory: 30% (RAM efficiency)\n\n**Quality Score** (0-100) — how well the model answers:\n- Reasoning: 20pts | Coding: 20pts | Instruction Following: 20pts\n- Structured Output: 15pts | Math: 15pts | Multilingual: 10pts\n\n**Global Score** = 30% Hardware Fit + 70% Quality\n\nHardware is auto-detected and scoring adapts to your tier (Entry/Balanced/High-End). A model hitting 10 tok/s on a 8GB machine scores differently than on a 64GB rig.\n\n[Full methodology &rarr;](https://metrillm.dev/methodology)\n\n## Share Your Results\n\nEvery benchmark you share enriches the [public leaderboard](https://metrillm.dev). No account needed — pick the method that fits your workflow:\n\n| Method | Command / Action | Best for |\n|--------|-----------------|----------|\n| **CLI** | `metrillm bench --share` | Terminal users |\n| **MCP** | Call `share_result` tool | AI coding assistants |\n| **Plugin** | `/benchmark` skill with share option | Claude Code / Cursor |\n\nAll methods produce the same result:\n- A **public URL** for your benchmark\n- Your **rank**: \"Top X% globally, Top Y% on [your CPU]\"\n- A **share card** for social media\n- A **challenge link** to send to friends\n\n[Compare your results on the leaderboard &rarr;](https://metrillm.dev)\n\n## MCP Server\n\nUse MetriLLM from Claude Code, Cursor, Windsurf, or any MCP client — no CLI needed.\n\n```bash\n# Claude Code\nclaude mcp add metrillm -- npx metrillm-mcp@latest\n\n# Claude Desktop / Cursor / Windsurf — add to MCP config:\n# { \"command\": \"npx\", \"args\": [\"metrillm-mcp@latest\"] }\n```\n\n| Tool | Description |\n|------|-------------|\n| `list_models` | List locally available LLM models |\n| `run_benchmark` | Run full benchmark (performance + quality) on a model |\n| `get_results` | Retrieve previous benchmark results |\n| `share_result` | Upload a result to the public leaderboard |\n\n[Full MCP documentation &rarr;](mcp/README.md)\n\n## Skills\n\nSlash commands that work inside AI coding assistants — no server needed, just a Markdown file.\n\n| Skill | Trigger | Description |\n|-------|---------|-------------|\n| `/benchmark` | User-invoked | Run a full benchmark interactively |\n| `metrillm-guide` | Auto-invoked | Contextual guidance on model selection and results |\n\nSkills are included in the [plugins](#plugins) below, or can be installed standalone:\n\n```bash\n# Claude Code\ncp -r plugins/claude-code/skills/* ~/.claude/skills/\n\n# Cursor\ncp -r plugins/cursor/skills/* ~/.cursor/skills/\n```\n\n## Plugins\n\nPre-built bundles (MCP + skills + agents) for deeper IDE integration.\n\n| Component | Description |\n|-----------|-------------|\n| MCP config | Auto-connects to `metrillm-mcp` server |\n| Skills | `/benchmark` + `metrillm-guide` |\n| Agent | `benchmark-advisor` — analyzes your hardware and recommends models |\n\n**Install:**\n```bash\n# Claude Code\ncp -r plugins/claude-code/.claude/* ~/.claude/\n\n# Cursor\ncp -r plugins/cursor/.cursor/* ~/.cursor/\n```\n\nSee [Claude Code plugin](plugins/claude-code/README.md) and [Cursor plugin](plugins/cursor/README.md) for details.\n\n## Integrations\n\n| Integration | Package | Status | Docs |\n|-------------|---------|--------|------|\n| CLI | [`metrillm`](https://www.npmjs.com/package/metrillm) | Stable | [Usage](#usage) |\n| MCP Server | [`metrillm-mcp`](https://www.npmjs.com/package/metrillm-mcp) | Stable | [MCP docs](mcp/README.md) |\n| Skills | — | Stable | [Skills](#skills) |\n| Claude Code plugin | — | Stable | [Plugin docs](plugins/claude-code/README.md) |\n| Cursor plugin | — | Stable | [Plugin docs](plugins/cursor/README.md) |\n\n## Development\n\n```bash\nnpm ci\nnpm run ci:verify     # typecheck + tests + build\nnpm run dev           # run from source\nnpm run test:watch    # vitest watch mode\n```\n\n## Homebrew Formula Maintenance\n\nThe tap formula lives in `Formula/metrillm.rb`.\n\n```bash\n# Refresh Formula/metrillm.rb with latest npm tarball + sha256\n./scripts/update-homebrew-formula.sh\n\n# Or pin a specific version\n./scripts/update-homebrew-formula.sh 0.2.1\n```\n\nAfter updating the formula, commit and push so users can install/update with:\n\n```bash\nbrew tap MetriLLM/metrillm\nbrew install metrillm\nbrew upgrade metrillm\n```\n\n## Contributing\n\nContributions are welcome! Please read the [Contributing Guide](CONTRIBUTING.md) before submitting a pull request. All commits must include a DCO sign-off.\n\n## License\n\n[Apache License 2.0](LICENSE) — see [NOTICE](NOTICE) for trademark information.\n",
  "bytes": 11185,
  "sha": "6ec2b38beaa2887e28f164b2f2c3f1f3c32561f2229ec6d2ebe7288adc49381e",
  "repo_slug": "metrillm/metrillm",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_metrillm_metrillm_8dd5a157/readme"
}