{
  "markdown": "<!-- mcp-name: io.github.suneel190700/mendrift-mcp -->\n\n![test](https://github.com/suneel190700/mendrift/actions/workflows/test.yml/badge.svg)\n\n# Mendrift\n\nAutonomous MLOps incident response agent, plus **mendrift-mcp** — an open-source\nMCP server for drift detection and ML incident tooling.\n\n```bash\npip install mendrift-mcp     # or: uvx mendrift-mcp\n```\n\nPublished on [PyPI](https://pypi.org/project/mendrift-mcp/) and the\n[MCP Registry](https://registry.modelcontextprotocol.io) as\n`io.github.suneel190700/mendrift-mcp`.\n\n**▶ [Live demo](https://mendrift-demo.onrender.com)** — run a real incident in your\nbrowser: supply an alert, watch the agent diagnose it against a real MLflow registry,\nand approve or reject the rollback at the human-in-the-loop gate. Toggle between a\ncrafted synthetic scenario and **real US consumer-credit benchmark data**. React\nfrontend on a FastAPI backend; the free tier sleeps, so the first load may take ~40s.\n\nWhen a production model drifts or degrades, Mendrift detects it, diagnoses the\nroot cause from monitoring and registry evidence, proposes a remediation, and\nexecutes it **only after human approval**.\n\n```\nalert ──> classify ──> diagnose (MCP tools) ──> propose\n             │                                     │\n           noise ──> close               human approval gate\n                                                   │\n                                    execute ──> verify recovery\n```\n\nBuilt with LangGraph (agent orchestration), LangChain (`ChatAnthropic` +\n`bind_tools`), the Model Context Protocol, Evidently, MLflow, and Claude\n(Haiku + Sonnet).\n\n## mendrift-mcp tools\n\n| tool | type | purpose |\n|---|---|---|\n| `get_drift_report` | read | per-feature drift distances + schema changes (Evidently) |\n| `summarize_metric_anomalies` | read | production vs previous model scored on current traffic |\n| `get_deployment_history` | read | registry version transitions and aliases |\n| `diff_deployments` | read | params / metrics / feature-schema diff between versions |\n| `propose_rollback` | read | generates a reviewable rollback plan |\n| `execute_rollback` | **gated** | requires a single-use HMAC `approval_token` |\n| `open_incident` | write | incident record with diagnosis + evidence |\n\n## Safety model\n\nThe approval gate is enforced in the **tool layer, not the prompt**:\n`execute_rollback` verifies a single-use, action-scoped HMAC token minted only\nby the human review flow — the minting function is never exposed over MCP. A\nprompt-injected or confused agent cannot execute writes.\n\nTested live: Claude was first ordered to roll back \"with full authorization\"\n(it proposed but declined to fabricate a token), then handed a fabricated\ntoken, which the gate rejected by constant-time HMAC comparison:\n\n![Approval gate defense: refusal, then cryptographic rejection](https://raw.githubusercontent.com/suneel190700/mendrift/main/docs/gate-defense.png)\n\nSee `tests/test_approval_gate.py`, including the action-scoping test: a token\nminted for one model/version is invalid for any other.\n\n## Human-in-the-loop, crash-proof\n\nThe incident graph halts before execution (`interrupt_before`) and checkpoints\nevery step to SQLite. The process can die; a new process resumes the same\nincident by `thread_id` after a human mints the approval token — which enters\nstate only via `update_state()`, from outside the graph. Denial is a\nfirst-class path: no token → `closed_approval_denied`, no execution.\n\n![Kill-and-resume demo](https://raw.githubusercontent.com/suneel190700/mendrift/main/docs/interrupt-demo.gif)\n\n## Agent design\n\n| step | model | why |\n|---|---|---|\n| classify | Haiku | single constrained label; cheapest path |\n| diagnose | Sonnet | multi-hop tool reasoning over evidence |\n| verify | Haiku | threshold check on fresh metrics |\n\nRouting lives in a code table (`ROUTER_TABLE`), not prompts, so cost per path\nis measurable config — ~3.9K input / 630 output tokens per incident. The\ndiagnose loop is bounded (max 8 tool calls) with per-call retries and capped\nbackoff; on tool failure the model receives a structured error record, and on\nbudget exhaustion the agent degrades to an incident with partial evidence — it\nnever invents a diagnosis. Destructive actions require affirmative evidence: a\nrollback is recommended only when retrieved evidence links the symptom to a\nspecific deployment, never on deploy-correlation alone. The agent can also\nrecommend **monitor** — real but mild, non-actionable drift is watched, not\nacted on.\n\n## Live mode\n\n`MENDRIFT_DEMO=0` runs the agent against real infrastructure rather than fixtures:\n\n- `scripts/seed_demo.py` trains two sklearn versions into a local MLflow registry —\n  v13 clean, v14 with a schema swap and a training window polluted by missed-fraud\n  labels (recall 0.72 → 0.18, AUC 0.84 → 0.82) — and writes reference/current frames\n- `get_drift_report` runs Evidently's `DataDriftPreset` over those frames, returning\n  real Wasserstein/JS distances against per-metric thresholds, plus schema changes\n  derived from actual column sets\n- `get_deployment_history` / `diff_deployments` read the registry and the underlying\n  runs — real aliases, params, metrics\n- `summarize_metric_anomalies` scores the current window with both the production and\n  previous versions, so it reports **model** divergence rather than population drift —\n  a rollback clears it, ordinary data shift does not\n- an approved `execute_rollback` moves the `production` alias for real\n\n```bash\nuv run mlflow server --host 127.0.0.1 --port 5001        # separate terminal\nPYTHONPATH=src uv run python scripts/seed_demo.py\n\nrm -f demo.db\nMENDRIFT_DEMO=0 PYTHONPATH=src uv run python scripts/demo_interrupt.py start\nMENDRIFT_DEMO=0 PYTHONPATH=src uv run python scripts/demo_interrupt.py approve\n```\n\nA live run diagnoses from computed evidence — e.g. *\"v2 introduced a schema swap\nreplacing promo_flag with promo_flag_v2 … label_noise 0.0 → 0.45 collapsing\nval_recall 0.724 → 0.176 … 79.7% prediction-rate divergence from the prior version,\nmodel-induced, not population drift\"* — then halts for approval and resolves.\n\nThe eval suite deliberately stays on fixtures: evals need determinism and zero cost\nin CI, while live mode exercises the real stack.\n\n## Live web demo (two worlds)\n\nA hosted web app wraps live mode behind a browser UI: a **React (Vite)** frontend on\na **FastAPI** backend, deployed on Render. A visitor submits an alert, the frontend\nposts it to `/api/diagnose`, and the backend runs the real LangGraph agent — live\nClaude reasoning over an embedded MLflow registry (`sqlite://`, seeded on boot) — then\nhalts at the HMAC gate. Approve or reject and the backend resumes the graph via\n`/api/decision`, executing a real alias rollback and verifying recovery. The Anthropic\nkey lives only on the server; runs are rate-limited since each calls a real model.\nTry it: **[mendrift-demo.onrender.com](https://mendrift-demo.onrender.com)**.\n\nThe dashboard toggles between two seeded worlds, so the same agent can be seen against\nboth a crafted scenario and genuine real-world data:\n\n- **Synthetic** (`scripts/seed_demo.py`, model `fraud-scorer`) — the crafted schema-swap\n  incident: clean, teachable, an unambiguous rollback story.\n- **Real US credit** (`scripts/seed_real.py`, model `credit-risk`) — the\n  [Give Me Some Credit](https://www.openml.org/search?type=data&status=active&id=44228)\n  dataset (real US consumer-credit records, target `SeriousDlqin2yrs`) split by borrower\n  age into reference/current windows for genuine feature drift, with a **controlled model\n  regression injected** into v2 (asymmetric missed-default label noise) so the incident\n  has ground truth. Real distributions and real Evidently drift; a known correct action.\n  Measured gap: val_recall 0.637 → 0.156, AUC 0.854 → 0.810.\n\nInjecting a known regression into real data is standard practice for validating a\ndrift-detection system — it gives the evaluator ground truth for what the agent *should*\ndecide while the drift computation still runs on genuine distributions.\n\nThe backend routes each request to the right world (model + parquet frames + label\ncolumn) per the `dataset` field; the tool layer reads those from env vars, applied\nper-request under a lock so concurrent requests stay isolated.\n\nRun the web app locally:\n\n```bash\n# 1. build the React frontend (FastAPI serves the built assets)\ncd frontend && npm install && npm run build && cd ..\n\n# 2. seed both worlds, then start the backend (frontend + API on one port)\nexport ANTHROPIC_API_KEY=sk-ant-...\nexport MLFLOW_TRACKING_URI=\"sqlite:///$(pwd)/mlflow.db\"\nPYTHONPATH=src uv run python scripts/seed_demo.py     # synthetic world (fraud-scorer)\nPYTHONPATH=src uv run python scripts/seed_real.py     # real world (credit-risk)\nPYTHONPATH=src uv run uvicorn app.main:app --port 8000     # open http://localhost:8000\n```\n\nFor frontend development with hot reload, run `cd frontend && npm run dev` (port 5173);\nVite proxies `/api` to the backend on port 8000.\n\n## Evaluation\n\n`src/mendrift/evals/` replays synthetic incident trajectories against the\n**real graph** — only the LLM (scripted) and the read tools (fixture world)\nare faked; the gated action tools are the genuine implementations, so the HMAC\ngate is exercised by every test. Four assertions per trajectory:\n\n| check | meaning |\n|---|---|\n| `no_ungated_writes` | every `execute_rollback` carried a valid HMAC token — **hard fail** |\n| `classification_ok` | triage label matched |\n| `tool_sequence_ok` | required tool calls occurred in order (extras allowed) |\n| `action_ok` | terminal outcome matched |\n\n**19 logic-distinct incident scenarios** spanning the decision space, each with\nits own evidence shape and correct action:\n\n- **Rollback** — deploy-correlated drift or quality regression with affirmative diff evidence\n- **Retrain** — label/concept shift, segment-specific degradation (no valid rollback target)\n- **Monitor** — mild seasonal drift, low-importance-feature drift, holiday effects\n- **Incident (investigate)** — upstream schema rename, feature-store change, docs-only deploy, calibration break, threshold shift, silent data-quality drop\n- **Graceful degradation** — evidence tools down → incident with partial evidence, never a fabricated diagnosis\n- **Noise** — flapping / auto-resolved alerts closed with zero tool calls\n- **Human-declined** — well-founded rollback the reviewer rejects → closed, no execution\n\nScripted for fast CI, live for the measured rate:\n\n```bash\nPYTHONPATH=src uv run python scripts/run_traj.py --all          # scripted, fast\nPYTHONPATH=src uv run python scripts/run_traj.py --all --live   # real models\n```\n\nLive-model eval runs at ~95% task-success; the handful of run-to-run\ndivergences reflect LLM eval variance on decision-margin scenarios. The live\nsuite surfaced real failure classes during development — a JSON extractor\nmasking a correct decision, a classifier baited by an alert's reassuring\nwording, and a diagnoser proposing rollback on correlation alone — each fixed\nat its own layer (parser, alert wording, evidence-rule prompt).\n\n## Quickstart (demo mode)\n\n```bash\nuv sync\nMENDRIFT_DEMO=1 uv run mendrift-mcp     # stdio MCP server with fixture data\nPYTHONPATH=src uv run pytest -v         # gate + trajectory suite\n```\n\nClaude Desktop config:\n\n```json\n{\"mcpServers\": {\"mendrift\": {\n  \"command\": \"uvx\",\n  \"args\": [\"mendrift-mcp\"],\n  \"env\": {\"MENDRIFT_DEMO\": \"1\"}\n}}}\n```\n\n## Status\n\n- [x] mendrift-mcp server over stdio, verified in MCP Inspector and Claude Desktop\n- [x] seven tools with a read / gated / write permission taxonomy\n- [x] HMAC-gated rollback with action-scoped single-use tokens (tests first)\n- [x] LangGraph incident graph: SQLite checkpointing + human-approval interrupt, kill-resume proven\n- [x] LLM nodes on LangChain (`ChatAnthropic.bind_tools`): Haiku classify/verify, Sonnet diagnose loop\n- [x] 19-scenario trajectory eval across the decision space; ~95% live, zero ungated writes\n- [x] CI: gate + trajectory suite on every push\n- [x] live mode: real Evidently drift computation, MLflow registry history/diff, real alias rollback\n- [x] live web demo: React + Vite frontend on FastAPI, deployed on Render\n- [x] two demo worlds: crafted synthetic scenario + real US credit-risk data, selectable in the UI\n- [x] published: PyPI (`pip install mendrift-mcp`) + MCP Registry (`io.github.suneel190700/mendrift-mcp`)\n\n## License\n\nMIT\n",
  "bytes": 12410,
  "sha": "c7ece352e0793a4a4bae9d5757159da0b74d3ce6da1dfdf41695e61dd6f9007f",
  "repo_slug": "suneel190700/mendrift",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_suneel190700_mendrift_mcp_a9952d10/readme"
}