{
  "markdown": "![llmprobe](demo/tui-thumbnail.png?v=2)\n\n# llmprobe\n\n> Synthetic monitoring and CI smoke tests for LLM inference endpoints. Measure TTFT, latency, throughput, and errors. Single binary, zero SDKs.\n\n[![CI](https://github.com/Jwrede/llmprobe/actions/workflows/ci.yml/badge.svg)](https://github.com/Jwrede/llmprobe/actions/workflows/ci.yml)\n[![Go](https://img.shields.io/badge/go-1.25+-00ADD8?logo=go)](https://go.dev)\n[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)\n[![llmprobe MCP server](https://glama.ai/mcp/servers/Jwrede/llmprobe/badges/score.svg)](https://glama.ai/mcp/servers/Jwrede/llmprobe)\n\nllmprobe is a CLI tool for LLM serving reliability. It probes hosted APIs or\nOpenAI-compatible inference servers, then reports the metrics that matter for\nproduction user experience: time to first token (TTFT), total latency,\ngeneration throughput (tokens/sec), and error rates.\n\nUse it as a one-off health check, a continuous monitor, or a CI gate that\nblocks deploys when your LLM provider is degraded.\n\n![demo](demo/llmprobe-demo.gif)\n\n## Public benchmark\n\n[llm-bench](https://github.com/Jwrede/llm-bench) uses llmprobe to run a\ncontinuous public benchmark of major LLM APIs. It publishes a live dashboard at\n[bench.jonathanwrede.de](https://bench.jonathanwrede.de) and raw JSONL data in\n[Jwrede/llm-bench-data](https://github.com/Jwrede/llm-bench-data).\n\nThis is the intended use case: repeated synthetic probes that make LLM latency,\nTTFT regressions, throughput drops, and provider degradation visible before\nusers report them.\n\n## Install\n\n**Download a prebuilt binary** from the\n[latest release](https://github.com/Jwrede/llmprobe/releases/latest)\n(Linux, macOS, Windows; amd64 and arm64).\n\nOr install from source:\n\n```bash\ngo install github.com/Jwrede/llmprobe@latest\nllmprobe version\n```\n\n### Claude Code plugin\n\nInstall as a Claude Code plugin for `/llmprobe` skill and MCP tools:\n\n```bash\nclaude plugin install Jwrede/llmprobe\n```\n\nOr register the MCP server directly:\n\n```bash\nclaude mcp add --transport stdio llmprobe -- llmprobe mcp\n```\n\nllmprobe runs locally and only contacts LLM endpoints you configure.\nSee [PRIVACY.md](PRIVACY.md) for details.\n\n## Quick start\n\nllmprobe works with OpenAI, Anthropic, Google, Azure OpenAI, AWS Bedrock, and\nOpenAI-compatible endpoints such as vLLM, Ollama, OpenRouter, Groq, Together AI,\nFireworks, DeepSeek, and Mistral.\n\nCreate a `probes.yml` (or copy the included example):\n\n```yaml\nproviders:\n  - name: openai\n    api_key: ${OPENAI_API_KEY}\n    models:\n      - name: gpt-4o\n        thresholds:\n          max_ttft: 2s\n      - name: gpt-4o-mini\n        thresholds:\n          max_ttft: 500ms\n\n  - name: anthropic\n    api_key: ${ANTHROPIC_API_KEY}\n    models:\n      - name: claude-sonnet-4-20250514\n        thresholds:\n          max_ttft: 1s\n```\n\nRun a probe:\n\n```\n$ llmprobe probe\n\nProvider   Model                    Status    TTFT    Latency  Tok/s  Tokens  Error\n--------   -----                    ------    ----    -------  -----  ------  -----\nopenai     gpt-4o                   healthy   312ms   2100ms   68.4   42\nopenai     gpt-4o-mini              healthy   98ms    814ms    112.3  56\nanthropic  claude-sonnet-4-20250514 healthy   420ms   2831ms   52.1   38\nazure      gpt-4o                   healthy   289ms   1950ms   71.2   44\nbedrock    anthropic.claude-3-5...  degraded  1820ms  4510ms   28.1   38\n\n4 healthy, 1 degraded, 0 errors\n```\n\n## What it measures\n\n| Metric | What it means |\n|--------|--------------|\n| **TTFT** | Time from request send to first content token. This is what users feel as \"lag\" before the response starts streaming. |\n| **Latency** | Total time from request to stream close. |\n| **Tok/s** | Generation throughput: tokens produced per second after the first token. Calculated as `token_count / (latency - ttft)`. |\n| **Tokens** | Total output tokens. Prefers provider usage metadata when available, falls back to SSE event counting. |\n| **Status** | `healthy` if all thresholds pass, `degraded` if any threshold is exceeded, `error` if the request failed. |\n\n## Commands\n\n### `llmprobe probe`\n\nOne-off health check. Probes all configured endpoints and prints results.\n\n```bash\nllmprobe probe                        # table output\nllmprobe probe -f json                # JSON output\nllmprobe probe --fail-on degraded     # exit 1 if any endpoint is degraded\nllmprobe probe -c custom-config.yml   # custom config path\n```\n\nExit codes for CI:\n\n| `--fail-on` | Exit 0 | Exit 1 |\n|-------------|--------|--------|\n| `error` (default) | healthy or degraded | any error |\n| `degraded` | healthy only | degraded or error |\n| `none` | always | never |\n\n### `llmprobe watch`\n\nContinuous monitoring. Probes all endpoints on an interval and prints a\nsummary line per iteration.\n\n```bash\nllmprobe watch                          # default 60s interval\nllmprobe watch --interval 30s           # custom interval\nllmprobe watch --tui                    # live terminal dashboard with TTFT chart\nllmprobe watch --tui --load data.jsonl  # load historical data into the dashboard\nllmprobe watch -f json                  # JSONL output (one line per result)\nllmprobe watch --prometheus :9090       # expose Prometheus metrics\nllmprobe watch --otel localhost:4317     # export OpenTelemetry metrics via OTLP/gRPC\n```\n\nThe `--tui` flag launches a live terminal dashboard with a TTFT chart,\ncolor legend, and statistics table. Use `--load` to import historical\nJSONL data (from `llmprobe watch -f json > data.jsonl`).\n\n![llmprobe](demo/tui-screenshot.png?v=2)\n\n```\n$ llmprobe watch --interval 30s\n\nWatching 4 endpoints every 30s (Ctrl+C to stop)\n\n[14:01:02] All 4 endpoints healthy.\n[14:01:32] All 4 endpoints healthy.\n[14:02:02] 3 healthy, 1 degraded, 0 errors. DEGRADED: openai/gpt-4o (TTFT 1820ms)\n[14:02:32] All 4 endpoints healthy.\n```\n\n### `llmprobe report`\n\nGenerate a Markdown summary from JSONL probe data with p50/p95/p99\npercentiles for TTFT, latency, and throughput per endpoint.\n\n```bash\nllmprobe report data.jsonl\n```\n\nOutput:\n\n```\n| Provider | Model | Probes | Errors | TTFT p50 | TTFT p95 | ... | Tok/s p50 | ...\n|----------|-------|--------|--------|----------|----------|-----|-----------|----\n| openai   | gpt-4o | 100  | 2      | 115ms    | 188ms    | ... | 46.9      | ...\n```\n\n### `llmprobe baseline`\n\nCreate a baseline file from historical JSONL data for regression detection.\n\n```bash\nllmprobe baseline data.jsonl -o baseline.json\n```\n\nReference the baseline in your config to use multiplier-based thresholds:\n\n```yaml\nbaseline: baseline.json\n\nproviders:\n  - name: openai\n    api_key: ${OPENAI_API_KEY}\n    models:\n      - name: gpt-4o\n        thresholds:\n          max_ttft_multiplier: 2.0       # fail if TTFT > 2x baseline p50\n          max_latency_multiplier: 2.5    # fail if latency > 2.5x baseline p50\n```\n\nThis lets you detect regressions relative to your own historical data rather\nthan setting absolute thresholds.\n\n### `llmprobe version`\n\nPrint the installed binary version.\n\n```bash\nllmprobe version\n```\n\n## CI integration\n\nUse `llmprobe probe` as a pre-deploy gate:\n\n```yaml\n# .github/workflows/deploy.yml\n- name: Check LLM providers\n  env:\n    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}\n  run: |\n    go install github.com/Jwrede/llmprobe@latest\n    llmprobe probe --fail-on degraded\n```\n\nThis blocks the deploy if any LLM provider is experiencing degraded\nperformance right now.\n\nWhen a probe fails, the output shows only the failing endpoints:\n\n```\nFailed endpoints (1/4):\n  openai/gpt-4o  DEGRADED  TTFT=280ms  Latency=950ms  Tok/s=32.1\n```\n\n## MCP server\n\n[![llmprobe MCP server](https://glama.ai/mcp/servers/Jwrede/llmprobe/badges/card.svg)](https://glama.ai/mcp/servers/Jwrede/llmprobe)\n\nllmprobe includes a built-in [Model Context Protocol](https://modelcontextprotocol.io/)\nserver, allowing Claude Code and other MCP hosts to check LLM API health\ndirectly from an agent workflow.\n\n### Running the server\n\n```bash\nllmprobe mcp\n```\n\nThis starts the MCP server over stdio.\n\n### Registering with Claude Code\n\n```bash\nclaude mcp add --transport stdio llmprobe -- llmprobe mcp\n```\n\nOnce registered, Claude Code can call llmprobe tools during any conversation.\n\n### Available tools\n\n| Tool | Description |\n|------|-------------|\n| `probe_all` | Probe all configured endpoints from `probes.yml`. Returns TTFT, latency, throughput, and health status for every model. Accepts an optional `config` parameter for a custom config path. |\n| `probe_model` | Probe a single model without a config file. Requires `provider`, `model`, and `api_key_env`. Supports optional `base_url` for OpenAI-compatible endpoints and optional `label` for display. |\n| `list_providers` | List all providers and models in the config file with their thresholds. Use this to discover available models before probing. |\n| `get_config` | Return the full parsed configuration including defaults, providers, models, and thresholds. |\n\n**Example use case:** An agent calls `list_providers` to see what models\nare configured, then `probe_all` to verify they are healthy before\ndeploying changes.\n\n## Configuration\n\n```yaml\ndefaults:\n  prompt: \"Hello\"                                # probe prompt\n  max_tokens: 20                                 # max output tokens\n  timeout: 30s                                   # per-probe timeout\n  concurrency: 5                                 # max parallel probes\n\nproviders:\n  - name: openai                    # openai, anthropic, google, azure, bedrock\n    label: openai-prod              # optional display name; useful for multiple OpenAI-compatible endpoints\n    api_key: ${OPENAI_API_KEY}      # env var expansion\n    base_url: https://custom.api    # optional, override endpoint\n    models:\n      - name: gpt-4o\n        prompt: \"Say hello.\"        # override default prompt\n        max_tokens: 10              # override default max_tokens\n        response_format: json       # optional; OpenAI-compatible JSON mode\n        validate_json: true         # optional; mark degraded if returned content is not valid JSON\n        thresholds:\n          max_ttft: 2s              # alert if TTFT exceeds this\n          max_latency: 10s          # alert if total latency exceeds this\n          min_tokens_per_sec: 20    # alert if throughput drops below this\n          max_ttft_multiplier: 2.0  # optional; compare against baseline p50\n          max_latency_multiplier: 2.5\n\n  - name: azure\n    api_key: ${AZURE_OPENAI_API_KEY}\n    base_url: https://your-resource.openai.azure.com\n    api_version: \"2024-10-21\"       # optional, defaults to 2024-10-21\n    models:\n      - name: gpt-4o               # deployment name\n\n  - name: bedrock\n    access_key: ${AWS_ACCESS_KEY_ID}\n    secret_key: ${AWS_SECRET_ACCESS_KEY}\n    region: us-east-1\n    models:\n      - name: anthropic.claude-3-5-sonnet-20241022-v2:0\n```\n\nAPI keys and AWS credentials support `${ENV_VAR}` syntax. Only credential\nfields are expanded, so env var references in prompts or model names are\nleft as-is.\n\n### OpenAI-compatible providers\n\nMany providers (Groq, Together AI, Fireworks, DeepSeek, Mistral, OpenRouter,\nOllama, vLLM) expose an OpenAI-compatible API. These work out of the box\nby setting `base_url`. Use the `label` field to distinguish multiple\nOpenAI-compatible blocks:\n\n```yaml\nproviders:\n  # Groq\n  - name: openai\n    label: groq\n    api_key: ${GROQ_API_KEY}\n    base_url: https://api.groq.com/openai\n    models:\n      - name: llama-3.3-70b-versatile\n\n  # DeepSeek\n  - name: openai\n    label: deepseek\n    api_key: ${DEEPSEEK_API_KEY}\n    base_url: https://api.deepseek.com\n    models:\n      - name: deepseek-chat\n\n  # Together AI\n  - name: openai\n    label: together\n    api_key: ${TOGETHER_API_KEY}\n    base_url: https://api.together.xyz\n    models:\n      - name: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo\n\n  # Local Ollama\n  - name: openai\n    label: ollama\n    api_key: unused\n    base_url: http://localhost:11434\n    models:\n      - name: llama3.2\n```\n\nSee [examples/](examples/) for ready-to-use configs for vLLM, SGLang, and Ollama.\n\n### JSON response validation\n\nFor OpenAI-compatible endpoints, set `response_format: json` to request JSON\nmode and `validate_json: true` to mark the probe as `degraded` if the streamed\ncontent is not valid JSON.\n\n```yaml\nproviders:\n  - name: openai\n    label: vllm-json\n    api_key: unused\n    base_url: http://localhost:8000\n    models:\n      - name: meta-llama/Llama-3.1-8B-Instruct\n        prompt: 'Return {\"ok\": true} as JSON.'\n        response_format: json\n        validate_json: true\n```\n\n## Prometheus metrics\n\nRun with `--prometheus` to expose metrics for scraping:\n\n```bash\nllmprobe watch --interval 30s --prometheus :9090\n```\n\nAvailable metrics at `/metrics`:\n\n| Metric | Type | Labels |\n|--------|------|--------|\n| `llmprobe_ttft_seconds` | gauge | provider, model |\n| `llmprobe_latency_seconds` | gauge | provider, model |\n| `llmprobe_tokens_per_second` | gauge | provider, model |\n| `llmprobe_token_count` | gauge | provider, model |\n| `llmprobe_status` | gauge | provider, model |\n| `llmprobe_probes_total` | counter | provider, model |\n| `llmprobe_errors_total` | counter | provider, model |\n| `llmprobe_ttft_seconds_hist` | histogram | provider, model |\n| `llmprobe_latency_seconds_hist` | histogram | provider, model |\n| `llmprobe_tokens_per_second_hist` | histogram | provider, model |\n\nThe `llmprobe_status` gauge encodes health as: 1 = healthy, 0.5 = degraded,\n0 = error. Use this for alerting in Grafana or Alertmanager.\n\n## OpenTelemetry metrics\n\nRun with `--otel` to export probe metrics to an OTLP/gRPC collector.\n\n```bash\nllmprobe watch --interval 30s --otel localhost:4317\n```\n\nExported metric names:\n\n| Metric | Description |\n|--------|-------------|\n| `llmprobe.ttft.seconds` | Time to first token in seconds |\n| `llmprobe.latency.seconds` | Total request latency in seconds |\n| `llmprobe.tokens_per_second` | Generation throughput |\n| `llmprobe.token_count` | Output token count from the last probe |\n| `llmprobe.status` | 1 = healthy, 0.5 = degraded, 0 = error |\n| `llmprobe.probes.total` | Total probes executed |\n| `llmprobe.errors.total` | Total probe errors |\n\nAll metrics include `provider` and `model` attributes.\n\n## Architecture\n\n```\nprobes.yml\n  -> Config loader (YAML + env var expansion)\n    -> Probe engine (concurrent goroutines per provider/model)\n      -> Provider clients (raw HTTP + SSE parsing, no SDKs)\n        -> Results (TTFT, latency, tokens/sec, status)\n          -> Output (table, JSON, JSONL)\n```\n\nEach provider client is a thin HTTP wrapper that sends a streaming request\nand parses the response. No LLM SDKs are imported. The SSE parser handles\nboth data-only events (OpenAI, Google) and named events (Anthropic). The\nBedrock client implements SigV4 signing and AWS binary event stream parsing\nfrom scratch.\n\nTTFT is measured from the moment the HTTP request is sent to the first\nevent that contains actual content text (not role assignments or metadata).\n\n## Providers\n\n| Provider | Endpoint | Auth | Streaming format |\n|----------|----------|------|-----------------|\n| OpenAI | `/v1/chat/completions` | `Authorization: Bearer` | SSE, `[DONE]` sentinel |\n| Anthropic | `/v1/messages` | `x-api-key` header | named-event SSE |\n| Google | `/v1beta/models/{model}:streamGenerateContent?alt=sse` | `key` query param | SSE |\n| Azure OpenAI | `/openai/deployments/{model}/chat/completions` | `api-key` header | SSE, `[DONE]` sentinel |\n| AWS Bedrock | `/model/{model}/converse-stream` | SigV4 | AWS binary event stream |\n| OpenAI-compat | `/v1/chat/completions` (custom base_url) | `Authorization: Bearer` | SSE |\n\nOpenAI-compatible covers: Groq, Together AI, Fireworks, DeepSeek, Mistral,\nOpenRouter, Ollama, vLLM, and any endpoint that speaks the OpenAI chat\ncompletions API.\n\n## Roadmap\n\n- More provider-specific examples for self-hosted OpenAI-compatible endpoints\n- More report formats for long-running monitoring windows\n- Optional runbook templates for common LLM endpoint failures\n\n## License\n\nMIT\n",
  "bytes": 16084,
  "sha": "f49d4c985eef0775a77f43d680adc0b6f54787acac17e3c65fe85b0a78629219",
  "repo_slug": "jwrede/llmprobe",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/plg_jwrede_llmprobe_llmprobe_6f382ddd/readme"
}