{
  "markdown": "# LLM Preflight\n\n<!-- mcp-name: io.github.feronovak/llm-preflight -->\n\n**Last reviewed:** 2026-08-31 · **As of:** v2.10.0\n\n[![PyPI](https://img.shields.io/pypi/v/llm-preflight)](https://pypi.org/project/llm-preflight/)\n[![Tests](https://github.com/feronovak/llm-preflight/actions/workflows/tests.yml/badge.svg)](https://github.com/feronovak/llm-preflight/actions/workflows/tests.yml)\n[![License](https://img.shields.io/github/license/feronovak/llm-preflight)](LICENSE)\n\n![llm-preflight running the no-key demo: init, benchmark run, results table, quality gate, and decision block](https://raw.githubusercontent.com/feronovak/llm-preflight/main/docs/images/readme-demo.gif)\n\nKnow whether an AI-generated LLM integration is safe before it reaches\nproduction. LLM Preflight is a local contract preflight for LLM integration\nchanges: model, prompt, structured-output, and provider-call changes. It runs\na small cross-provider preflight and compares validated output, response\nspeed, tokens, and estimated cost.\n\n## Try it in 60 seconds\n\nCreate and run a deterministic local benchmark—no API key or network request:\n\n```bash\npython3 -m pip install llm-preflight\nllm-preflight init\nllm-preflight benchmark.json --no-save\n```\n\nFrom a source checkout:\n\n```bash\npython3 -m llm_preflight init\npython3 -m llm_preflight benchmark.json --no-save\n```\n\n`init` never overwrites an existing config. It creates a mock benchmark so\nyou can see the report and exit behavior before making a paid request.\nIts result is intentionally `inconclusive` (exit code `3`): a local mock\nvalidates configuration and output handling, but cannot approve a live model.\n\n## Choose your path\n\n- **Validate a change.** Compare an approved model, prompt, schema, or provider\n  route with a candidate using the [model-change guide](https://github.com/feronovak/llm-preflight/blob/main/docs/guides/model-change.md).\n- **Review a new model.** Discover metadata, deliberately probe a route, then\n  prepare a bounded candidate smoke with the [model-catalogue guide](https://github.com/feronovak/llm-preflight/blob/main/docs/guides/model-catalog.md).\n- **Automate an established contract.** Add the no-spend\n  [GitHub Marketplace Action](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/github-action.md)\n  or use [CI and JSON output](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/ci.md).\n- **Configure a coding agent.** Start the local MCP server from a trusted\n  repository with `llm-preflight-mcp --workspace \"$PWD\"`, then use the\n  [MCP server guide](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/mcp.md)\n  for your client configuration.\n\n## Safety boundary\n\n```mermaid\nflowchart LR\n    A[Integration change] --> B[No-spend validation\\ndoctor, pricing, dry run]\n    B --> C{Human reviews\\nevidence and cost bound}\n    C -->|Explicit approval| D[Bounded paid smoke]\n    C -->|No approval or missing evidence| E[Inconclusive: fix or stop]\n    D --> F[Local evidence for\\nproduction approval]\n```\n\nLLM Preflight is local evidence, not production approval. It is not a hosted\nevaluation platform, tracing system, RAG framework, or public leaderboard.\nIts results apply to your account, network, prompts, and validation rules.\n\n> [!WARNING]\n> Live benchmarks make paid API requests. Start with the no-key demo, preview\n> the plan before a live run, and keep limits and repetitions small.\n\n## CLI, CI, and MCP\n\nWorks as a CLI, GitHub Action, and local MCP server. Every path starts with\nno-spend validation and planning; a live provider run remains an explicit,\nbounded human-approved step. See the [GitHub Action guide](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/github-action.md)\nor the [MCP server guide](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/mcp.md).\n\nFor earlier releases, see the [changelog](CHANGELOG.md).\n\n## Purpose\n\n**Mission:** make every LLM integration change evidence-based before production.\n\n**Vision:** AI-assisted software delivery where an agent can validate its LLM\nchanges as routinely as it runs tests, while people retain control of spend and\nproduction approval.\n\n**Positioning:** LLM Preflight is the fast, local, cross-provider contract\npreflight for AI-powered application changes. It is not a general evaluation,\nobservability, or autonomous-deployment platform.\n\nIt is built for engineers and coding agents working on AI features: teams that\nneed to check a real application contract against live model APIs before a\nmodel ID, prompt, parser, tool definition, or provider option ships. Read the\n[north star](https://github.com/feronovak/llm-preflight/blob/main/docs/NORTH_STAR.md)\nand the [AI implementation testing guide](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/agent-validation.md)\nfor the intended workflow and boundaries.\n\n## Common jobs\n\n- **Switch a model or provider.** Run the bounded\n  [migration check](#change-a-model-safely), then add the contract test your\n  feature needs.\n- **Check a prompt, schema, parser, or tool change.** Define an explicit\n  [output contract](https://github.com/feronovak/llm-preflight/blob/main/docs/guides/output-contracts.md)\n  before the smoke.\n- **Review a newly discovered model.** Refresh metadata, then prepare—not run—\n  a bounded candidate plan:\n\n  ```bash\n  llm-preflight catalog refresh benchmarks/watch.json\n  llm-preflight catalog prepare benchmarks/watch.json \\\n    --against benchmarks/approved.json --output benchmarks/candidates.json\n  llm-preflight benchmarks/candidates.json --migration-check --dry-run\n  ```\n\n  Only explicitly approved, fully evidenced models proceed to paid work; see\n  the [model catalogue guide](https://github.com/feronovak/llm-preflight/blob/main/docs/guides/model-catalog.md).\n- **Investigate a provider or price change.** Run `--doctor`,\n  `--pricing-check`, and a dry-run; report a suspected regression through the\n  redacted issue forms.\n- **Automate a known contract.** Use the no-spend GitHub Action or the\n  [CI guide](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/ci.md)\n  with a saved baseline and `--ci`.\n\nIt measures deterministic test validity, end-to-end latency (p50/p95), time to\nfirst token, throughput when the stream is incremental and usage is available,\ntoken totals, and estimated cost. Result files retain request metadata and per-request observations for\nreproducibility.\n\n\"Deterministic\" describes the validator, not the model: every response is\nchecked against explicit structural rules — a regular expression, a JSON shape,\nan exact routing label — so the same response always produces the same verdict.\nThe tool does not score semantic quality; that is your task-specific\nevaluation, and it stays out of scope on purpose.\n\n## What live evidence looks like\n\nA completed preflight retains per-request observations and a machine-readable\ndecision: contract validity, latency (including TTFT where observable), token\nusage, estimated cost, pricing evidence, and blocking warnings. The terminal\nsummary is a convenience; automation should consume the saved JSON decision.\n\nThat evidence applies to your account, network, prompts, and validator at one\ntime—not a universal model ranking. For a complete interactive example, see\n[interactive runs](https://github.com/feronovak/llm-preflight/blob/main/docs/guides/interactive-runs.md).\n\n## First live run\n\nPython 3.10+ is required. There are no third-party runtime dependencies:\n`pip install llm-preflight` installs this package and nothing else, and the\nCLI runs on the Python standard library alone. Development tools (pytest,\nruff, mypy) are optional extras that never reach a production install.\n\n```bash\ncp benchmark.example.json benchmark.json\ncp .env.example .env.production\n# Edit benchmark.json and add only the provider keys you use.\npython3 -m llm_preflight benchmark.json --dry-run\npython3 -m llm_preflight benchmark.json\n```\n\nThe CLI reads `.env.production` beside the config without overriding environment\nvariables already set by your shell. Use `--no-env-file` or `--env-file PATH`\nwhen needed. Runs print a terminal report and, unless `--no-save` is used,\nwrite JSON and Markdown results under `results/`.\n\nInstall the command globally in a virtual environment if preferred:\n\n```bash\npython3 -m pip install llm-preflight\nllm-preflight --init\n```\n\nRun `--doctor` and `--dry-run` before the final command. They make no generation\nrequests; the final command is the paid work.\n\n## Change a model safely\n\nThis is the core workflow. Put your approved model and candidate model in one\nconfig, then run the small response-and-contract preflight:\n\n```bash\nllm-preflight benchmark.json --migration-check --dry-run\nllm-preflight benchmark.json --migration-check\n```\n\nIt sends three short representative cases to each selected model, once each.\nIt answers: did the API work, did each response meet the basic contract, and\nhow quickly did the provider start and finish responding? It is a cheap\ncompatibility check, not a statistical performance conclusion.\n\nWhen that passes, run the task-specific checks that match your application—for\nexample `exact-routing-check` or `structured-output-check`—before approving a\nswitch.\nUse [custom contract tests](https://github.com/feronovak/llm-preflight/blob/main/docs/guides/output-contracts.md) to express the outputs your\nown feature must preserve.\n\n## Using a coding agent\n\nGive an agent the same evidence you would use yourself: a reviewed config, an\nexplicit output contract, and a dry run before paid work. Start with the\nrecommended five-check suite:\n\n```bash\n# No generation request: inspect credentials, model selection, and paid-work plan.\nllm-preflight benchmark.json --doctor --json\nllm-preflight benchmark.json --tests agent-smoke --smoke --dry-run --json\n\n# Paid run, only after reviewing the plan.\nllm-preflight benchmark.json --tests agent-smoke --smoke --json --no-save\n```\n\nAn agent should not infer model IDs, weaken a validator to turn a failure into\na pass, or approve a model without an explicit instruction. The compact\n[LLM and coding-agent guide](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/coding-agents.md)\ncovers commands, result JSON, exit codes, and automation guardrails. The\n[AI implementation testing guide](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/agent-validation.md)\nshows how to make this validation an agent's default testing step.\n\n## MCP for coding agents\n\nUse the local stdio MCP server when an agent needs the preflight evidence\nwithout shell parsing or arbitrary command execution:\n\n```json\n{\n  \"mcpServers\": {\n    \"llm-preflight\": {\n      \"command\": \"llm-preflight-mcp\",\n      \"args\": [\"--workspace\", \"/absolute/path/to/repository\"]\n    }\n  }\n}\n```\n\nIt exposes only four tools: validate a config, prepare a dry-run plan, run an\nexplicitly confirmed preflight, and compare saved baselines. The first, second,\nand fourth tools never contact providers or load credentials. A live run still\nneeds an explicit paid-run confirmation. See the [MCP server guide](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/mcp.md) for tool\nsemantics, workspace boundaries, and the safe agent workflow.\n\n## Useful commands once you know your path\n\n```bash\n# Inspect configuration, credentials, and model selection without generation.\n# --doctor provides pricing advisory; use --pricing-check as the fail-closed coverage gate.\nllm-preflight benchmark.json --doctor\nllm-preflight benchmark.json --pricing-check\nllm-preflight benchmark.json --dry-run\n\n# Run a reduced live benchmark.\nllm-preflight benchmark.json --smoke\n\n# Run a single ad hoc prompt.\nllm-preflight --quick \"Return only valid JSON with a status field.\" \\\n  --models openai:gpt-5.4-mini\n```\n\nFor advanced discovery, interactive runs, CI, baselines, replay, and stop\nmodes, see [workflows](https://github.com/feronovak/llm-preflight/blob/main/docs/guides/model-change.md). For models, environment files,\ncustom prompts, and provider-specific options, see\n[configuration](https://github.com/feronovak/llm-preflight/blob/main/docs/reference/configuration.md).\n\n## What makes a comparison useful\n\n- Keep prompts, system instructions, temperature, and output limits fixed.\n- Validate outputs: a fast malformed response is a failed result.\n- Run from the same host; network distance and provider load affect latency.\n- Treat single-user latency and load testing as separate experiments.\n- Prefer dated model IDs over moving aliases.\n\nThe CLI distinguishes `API FAIL` (transport, credentials, provider, or request\nfailure) from `API OK / TEST FAIL` (a response that fails your validator).\nRecommendations only consider models that pass every selected test.\n\n## How it compares\n\nSeveral good tools live near this space. Use them when their job is your job:\n\n- **promptfoo, deepeval** — full evaluation suites: scored quality metrics,\n  red-teaming, large ongoing test matrices in CI. Use them to grade prompt and\n  model quality over time.\n- **Braintrust, LangSmith** — hosted platforms: tracing, dashboards, team\n  collaboration, production observability.\n- **`llm` (Simon Willison)** — a general multi-provider CLI for running\n  prompts, not a comparison harness.\n\nLLM Preflight does one narrower job: the local go/no-go check in the moment\nbefore an LLM integration change. Your prompt, candidate models, structural\nvalidation, latency, and cost — one command, one report, no hosted service, no\ntelemetry, and no vendor between you and the verdict.\n\n## Documentation\n\nStart at the [documentation homepage](https://github.com/feronovak/llm-preflight/blob/main/docs/index.md), then choose the path that matches your work:\n\n- **Start safely:** [safe demo](https://github.com/feronovak/llm-preflight/blob/main/docs/getting-started/safe-demo.md) and\n  [model change](https://github.com/feronovak/llm-preflight/blob/main/docs/guides/model-change.md).\n- **Validate a change:** [output contracts](https://github.com/feronovak/llm-preflight/blob/main/docs/guides/output-contracts.md),\n  [model catalogue](https://github.com/feronovak/llm-preflight/blob/main/docs/guides/model-catalog.md), and\n  [pricing and safety](https://github.com/feronovak/llm-preflight/blob/main/docs/guides/pricing-and-safety.md).\n- **Automate:** [CI and JSON output](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/ci.md),\n  [coding agents](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/coding-agents.md), and\n  [MCP](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/mcp.md).\n- **Look up details:** [CLI reference](https://github.com/feronovak/llm-preflight/blob/main/docs/reference/cli.md),\n  [configuration](https://github.com/feronovak/llm-preflight/blob/main/docs/reference/configuration.md),\n  [result JSON](https://github.com/feronovak/llm-preflight/blob/main/docs/reference/results.md), and\n  [troubleshooting](https://github.com/feronovak/llm-preflight/blob/main/docs/operations/troubleshooting.md).\n- **Understand the product:** [north star](https://github.com/feronovak/llm-preflight/blob/main/docs/NORTH_STAR.md),\n  [product decisions](https://github.com/feronovak/llm-preflight/blob/main/docs/DECISIONS.md), and\n  [AI implementation testing](https://github.com/feronovak/llm-preflight/blob/main/docs/automation/agent-validation.md).\n- [Contributing](https://github.com/feronovak/llm-preflight/blob/main/CONTRIBUTING.md) — development setup and the TDD workflow.\n- [Security](https://github.com/feronovak/llm-preflight/blob/main/SECURITY.md) — reporting vulnerabilities.\n\n## Contributing and license\n\nContributions are welcome; see [CONTRIBUTING.md](https://github.com/feronovak/llm-preflight/blob/main/CONTRIBUTING.md). Released\nunder the [MIT License](https://github.com/feronovak/llm-preflight/blob/main/LICENSE).\n",
  "bytes": 15867,
  "sha": "dd0fd4f9adaf66f94fcad4216f63a3a2b76473740ee62b7c06669ba868b2b341",
  "repo_slug": "feronovak/llm-preflight",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_feronovak_llm_preflight_ae7d170d/readme"
}