{
"markdown": "# Quellgeist\n\n[](https://github.com/Rajeev-Shyam/Quellgeist/actions/workflows/ci.yml)\n[](https://github.com/Rajeev-Shyam/Quellgeist/actions/workflows/security.yml)\n[](docs/case-studies/wave4-qwen-finetune.md)\n[](LICENSE)\n[](https://www.python.org/downloads/)\n\n> First-line incident triage you can trust: ranked root-cause hypotheses where **every claim cites a real evidence handle** — and the agent **abstains rather than guess**.\n\nQuellgeist is a model-agnostic AI agent for first-line production-incident triage.\nIt runs a legible JSON-action ReAct loop over read-only tools (structured logs +\nrecent deploys + metric time-series), then emits a structured **Diagnosis**:\nconfidence-ranked root-cause hypotheses, each backed by a structured evidence\nhandle (`LogRef.id` / `CommitRef.sha` / `MetricRef.id`) the agent actually saw —\nnever free text. Two ideas set it apart:\n\n- **Cite-by-structured-handle.** Evidence is a checkable handle, not a sentence,\n so a fabricated citation is *measurable* and **deterministically rejected** by a\n keyless fabrication check — not a matter of fuzzy string-matching.\n- **Abstain-over-hallucinate.** A confidently-stated wrong cause is the worst\n possible answer, so *\"insufficient evidence\"* is a first-class outcome.\n\n> **Status: Wave 4 complete — the fine-tune works.** The DR-0020 QLoRA fine-tune\n> of the local reasoner (Qwen3-4B, served via Ollama) took it from the base's\n> **0/16 holdout to 12/16** — zero fabrication, zero speculative-filtering, and\n> *cheaper* than the base — while **beating a 31B frontier** (Gemma-4-31B, 10/16)\n> on the same holdout at **$0, fully offline**. Non-memorisation is triangulated\n> three ways (fixtures ≈ holdout; core-fresh ≥ core-overlap; structure probe 7/10).\n> Two honest limits: the `resource_exhaustion` class didn't transfer (0/N; the\n> frontier passes it), and adversarial-abstention recall is **6/12** at the system\n> level — a ceiling the 31B frontier *shares* (also 6/12), not a fine-tune\n> regression. When this agent misses it's *incomplete* or *too cautious*, never\n> confidently fabricating. See\n> [Status & roadmap](#status--roadmap) · [fine-tune case study](docs/case-studies/wave4-qwen-finetune.md).\n\n## Why it's different\n\n| | |\n|---|---|\n| **Evidence is a handle** | Each hypothesis cites a log row's source-stable `id` or a commit `sha`, copied verbatim from a tool result — the unit the deterministic fabrication check looks up. Prose lives in a display-only `note`. (DR-0009) |\n| **Abstention is a feature** | When signals are weak the agent returns `abstained=true` with a reason and an empty hypotheses list — enforced by the schema. |\n| **Model-agnostic by construction** | The loop parses JSON actions from plain chat text, so it's identical on Gemini's free tier and a local 4-bit Qwen — no dependence on any backend's native function-calling. Swap models with one config change. (DR-0008, DR-0010) |\n| **Reliability is gated, not asserted** | A keyless, deterministic CI gate (ruff + black + `pytest`, including the fixture-backed eval harness) runs on every push. |\n\n### What it is / what it's NOT\n\n- **It is:** a first-line *triage* agent — ranked, evidence-cited root-cause\n hypotheses (or an honest abstention) from read-only logs/deploys/metrics, over a\n model-agnostic loop that runs on a hosted frontier model **or** a local 4B.\n- **It is NOT:** an autonomous remediator (it never mutates prod — resolution\n verification is a deferred, cut-first wave); a production-hardened service (the\n demo is a deliberate toy); or a general-purpose agent. The holdout it's measured\n on is *out-of-vocabulary but in-structure* — not a claim about unseen incident\n shapes or real production data.\n\n## Quickstart (~30 seconds to a broken service + structured logs)\n\nRequires [uv](https://docs.astral.sh/uv/) and Python 3.12+.\n\n**See a real-shaped diagnosis in one keyless command** (no model, no API key):\n\n```bash\nuv sync && uv run quellgeist diagnose --demo # renders the demo incident's cited postmortem\n```\n\nThen run the full loop against the live toy service:\n\n```bash\nuv run uvicorn demo.app.main:app # 1. start the toy service (leave running)\n\n# --- in a second shell, from the repo root ---\nuv run python -m demo.chaos.bad_deploy # 2. inject a simulated bad deploy\ncurl -s localhost:8000/login # 3. trip /login -> 500s + structured error logs\nuv run quellgeist diagnose --show-trace # 4. diagnose live (needs a model; see below)\n\nuv run python -m demo.chaos.reset # back to a green slate\n```\n\nThe live step needs a reasoner — see [Running the model](#running-the-model).\nWithout a key, `quellgeist diagnose` exits 1 with a one-line error + hint (never a\ntraceback); `--demo` always works keyless and renders the same output shape\ndeterministically from gold.\n\n## Architecture\n\nA custom, legible loop is the orchestration layer; the three read-only tools are\nthe evidence interface; the `Diagnosis` schema is the contract that the\npostmortem renderer and the eval judge both read.\n\n```mermaid\nflowchart TD\n trigger([\"incident trigger - CLI\"]) --> loop\n model[\"reasoner via LiteLLM
(Gemini or local Qwen, swappable)\"] -. \"chat completion\" .-> loop\n\n subgraph loopbox[\"model-agnostic JSON-action ReAct loop\"]\n loop[\"run_loop()
decide, call tool, observe, repeat\"]\n end\n\n loop -- \"query_logs\" --> logs[\"logs tool
structured JSONL, stable ids\"]\n loop -- \"get_recent_commits\" --> commits[\"commits tool
deploy_log.json, shas\"]\n loop -- \"query_metrics\" --> metrics[\"metrics tool
time-series, named series\"]\n logs -- \"rows + ids\" --> loop\n commits -- \"commits + shas\" --> loop\n metrics -- \"series + names\" --> loop\n\n loop --> diag[\"Diagnosis (schema.py)
ranked hypotheses citing
LogRef.id / CommitRef.sha / MetricRef.id, or abstains\"]\n diag --> pm[\"postmortem renderer
deterministic Markdown\"]\n diag --> judge[\"eval judge
fixture scenarios, CI gate\"]\n```\n\nAll three tools are also exposed as **MCP servers** over stdio\n(`python -m quellgeist.servers.logs_mcp`, `…commits_mcp`, `…metrics_mcp`). The\nagent currently reuses the same tool *functions* in-process behind a `ToolSpec`\nregistry; a stdio MCP-*client* path (the agent driving the servers over the\nwire) is on the roadmap (DR-0010).\n\n> **Deep dive:** [`docs/architecture.md`](docs/architecture.md) walks the full\n> pipeline (loop → tools → verifier → postmortem), a sequence diagram, the module\n> map, and the cross-cutting design decisions.\n\n\n\nThe servers publish to the **Official MCP Registry** on each tagged release (see\n[`docs/publishing.md`](docs/publishing.md)); once published each is runnable with\n`uvx --from quellgeist quellgeist-logs-mcp` (or `…-commits-mcp` / `…-metrics-mcp`).\n\n## Example session\n\nInject the bad deploy — it drops a marker that flips `verify_token` into a\nNoneType regression and writes a `deploy_log.json` whose offending commit landed\njust before the errors (illustrative `stdout` — the timestamp reflects when you\nrun it; paths shown relative to the repo root):\n\n```text\n$ uv run python -m demo.chaos.bad_deploy\ninjected bad deploy a1b2c3d (touched demo/app/auth.py) at 2026-06-24T12:22:43Z\n marker: demo/.bad_deploy\n deploy log: demo/deploy_log.json\nnext: hit /login to generate the 500s, then `quellgeist diagnose`\n```\n\nWith a reasoner configured, `quellgeist diagnose` reads the logs + deploys and\nemits a postmortem. The CI environment has no validated model key (DR-0012), so\nthe diagnosis below is **rendered from gold** — built deterministically from the\nfixture's labelled cause and evidence handles via `render_postmortem`, *not*\nlive model output:\n\n```text\n# Incident Postmortem (rendered from gold)\n\n## Root-cause hypotheses\n\n### 1. Bad deploy a1b2c3d (10:01:50Z) refactored auth.py and introduced a NoneType error in verify_token; /login 500s begin ~20s later at 10:02:12Z. (confidence: 1.00)\n\nEvidence:\n- log #2\n- commit a1b2c3d\n```\n\nReproduce that render yourself (no model needed):\n\n```bash\nuv run python - <<'PY'\nfrom evals.scenarios.generator import load_scenario\nfrom quellgeist.agent.schema import Diagnosis, Hypothesis\nfrom quellgeist.output.postmortem import render_postmortem\n\ns = load_scenario(\"evals/scenarios/fixtures/bad_deploy_0001.json\")\ngold = Diagnosis(hypotheses=[\n Hypothesis(cause=s.gold_cause, confidence=1.0, evidence=s.gold_evidence_refs)\n])\nprint(render_postmortem(gold, title=\"Incident Postmortem (rendered from gold)\"))\nPY\n```\n\nThe point isn't the prose — it's that **`log #2`** and **`commit a1b2c3d`** are\nexact handles into the real signals, not paraphrases. A live run additionally\nfills in a one-line summary and suggested actions, and abstains outright when the\nevidence is too weak to name a confident cause.\n\nWrite the postmortem to a file with `--out postmortem.md`, or as a self-contained\nHTML page with `--out postmortem.html` (or `--format html`) — same deterministic\nrender, no external assets.\n\n## Running the model\n\nThe reasoner is any [LiteLLM](https://docs.litellm.ai/) model string, selected by\n`--model` or the `QG_MODEL` env var (default `gemini/gemini-3.5-flash`). Provider\nkeys are read from the environment by LiteLLM; nothing is stored in the repo.\n\n```bash\nexport QG_MODEL=\"gemini/gemini-3.5-flash\"\nexport GEMINI_API_KEY=\"…\"\nuv run quellgeist diagnose --show-trace\n```\n\nOr fully local and offline via [Ollama](https://ollama.com) — the intended home\ndefault (DR-0008; exact artifact pinned in DR-0019), no API key involved:\n\n```bash\nollama pull qwen3:4b-instruct-2507-q4_K_M\nexport QG_MODEL=\"ollama_chat/qwen3:4b-instruct-2507-q4_K_M\"\nuv run quellgeist diagnose --show-trace\n```\n\n> **Base vs tuned — important.** The `ollama pull` above is the **base** Qwen3-4B:\n> the honest safe *floor* — it scores **0/16 on the holdout and abstains on\n> everything**, never fabricating (DR-0019). The **12/16** headline is the\n> **DR-0020 fine-tune** (`quellgeist-qwen3-dr0020`), which you build + serve via\n> [`finetune/README.md`](finetune/README.md) (a free-Colab QLoRA run →\n> `ollama create`). Until that tuned GGUF is published for a one-line pull, the\n> base model is what a plain `ollama pull` gives you — safe, not yet useful. Use a\n> hosted model (above) or the fine-tune to see live diagnoses.\n\nHeads-up (DR-0012): a Gemini key on an unvalidated, no-billing project returns\n`429 limit: 0` on current models, so the shipped CI gate is deliberately\n**keyless** and model-driven evals are key-gated and run **out-of-band**\n(DR-0015). At home the intended default reasoner is a local **Qwen3-4B** via\nOllama (DR-0008).\n\n### Running the eval (reasoner + verifier + LLM-judge)\n\nThe fixture eval scores the reasoner with a deterministic keyword judge + a\nzero-fabrication check (the keyless gate), and can additionally run two model\nlayers (DR-0016): a **verifier** that confirms cited evidence supports each\nhypothesis (forcing abstention otherwise) and an advisory **LLM-judge** rubric.\n\n```bash\nexport GEMINI_API_KEY=\"…\"\nexport QG_MODEL=\"gemini/gemini-3.5-flash\"\nQG_VERIFY=1 QG_JUDGE_LLM=1 \\\nQG_MIN_CALL_INTERVAL_S=6 \\ # pace calls under the free-tier RPM (avoids 429 bursts)\n uv run python -m evals.run_evals\n```\n\n`QG_VERIFIER_MODEL` / `QG_JUDGE_MODEL` override the model per layer (default\n`QG_MODEL`). An unreachable backend (quota/503/timeout) **or** a rejected\ncredential (missing/invalid/stale key) is reported as a **skip**, not a failure\n(DR-0015/DR-0017), so the out-of-band eval never reddens on a free-tier hiccup.\nThe LLM-judge's scores are **advisory** (they never gate). On a human-labelled\ngold subset it agreed with human verdicts at **Cohen's kappa 0.81** using an\nindependent judge (`groq/llama-3.1-8b-instant` ≠ the reasoner) — validated on that\nsubset (DR-0018); still self-grading whenever `QG_JUDGE_MODEL` equals the reasoner.\n\n> **CI's out-of-band eval runs on Groq** (`groq/llama-3.3-70b-versatile`, gated on\n> `GROQ_API_KEY`): Gemini's free tier proved unusable from cloud CI (429 → 503 →\n> timeout → invalid-key), so the reasoner was swapped with one env var — the\n> model-agnostic thesis in action (DR-0017). The intended *home* default remains a\n> local Qwen3-4B (DR-0008).\n\n## Using it on your real data\n\nThe demo eats three canonical files; your production signals don't look like that.\n`quellgeist ingest` is the adapter — point it at real sources and it writes the\ncanonical files the tools read:\n\n```bash\nquellgeist ingest \\\n --logs /var/log/myapp/ # file or directory; JSONL, JSON, plain text, or mixed\n --deploys deploys.json # JSON array, GitHub payload, or `git log` text\n --metrics prom.json # a Prometheus response or a canonical array\n --out-dir ./signals\n# prints the `export QG_*` lines; then:\nquellgeist diagnose --show-trace --strict-citations # add --model / a provider key\n```\n\nIt tolerates messy real data (foreign field names are aliased onto the schema,\ntimestamps normalised to UTC, a malformed line coerced rather than crashing the\nrun), and `query_logs` caps how many rows one observation returns (`QG_MAX_ROWS`,\ndefault 200) so a large log can't blow the context window. The deterministic\ncite-or-abstain guarantee runs **at real-use time**: `diagnose` verifies every\ncited handle against your real signals and warns on a fabrication (`--strict-citations`\nexits non-zero for CI). Full guide: [`docs/ingestion.md`](docs/ingestion.md).\n\n## Run the live service (v2)\n\nv2 wraps the same frozen core in a live, concurrent, observable incident-response\nservice: a **signed webhook** triggers an investigation, a worker pool runs the\n*unchanged* loop over an isolated per-incident snapshot, every run is persisted to\nSQLite with its trace and cost, an operator **approves / steers / rejects** before it\nposts to Slack + a self-contained HTML page, and after a sandbox fix the agent re-reads\nsignals to confirm recovery. Everything additive; the frozen measurement surface is\nuntouched. The whole stack runs from one file:\n\n```bash\ncp .env.example .env # set QG_WEBHOOK_SECRET, QG_OPERATOR_TOKEN, QG_VERIFIER_MODEL…\ndocker compose up --build # demo service + agent service + Ollama\ndocker compose exec ollama ollama pull qwen3:4b-instruct-2507-q4_K_M # first run only\n\n# break it, trigger an investigation, review, then confirm the fix in the sandbox:\ndocker compose exec demo python -m demo.chaos.bad_deploy\n# … POST a signed incident to :8000/incidents, approve it on the HTML page …\ndocker compose exec demo python -m demo.chaos.fix_deploy # heal without wiping the log\n# … POST /incidents/{id}/verify-resolution → recovered | not_recovered | inconclusive\n```\n\nSecrets stay env-only (public repo); the service is **fail-closed** — no webhook secret\nrejects every request, no operator token closes the operator surface, and it never posts a\nfabricated *or* unverified diagnosis. Design: [DR-0023](docs/quellgeist-adr-log.md) +\n[spec](docs/quellgeist-v2-spec.md).\n\n## Status & roadmap\n\nBuilt in **rolling waves** — only the current wave is implemented in detail\n(see [`docs/quellgeist-plan-rolling-wave.md`](docs/quellgeist-plan-rolling-wave.md)).\nThe full decision history lives in the\n[**ADR log**](docs/quellgeist-adr-log.md).\n\n| Wave | Scope | Status |\n|---|---|---|\n| 0 | De-risk the model bet (4B can orchestrate the loop) | ✅ done — default = Qwen3-4B (DR-0008) |\n| 1 | Bad-deploy slice: demo → break → diagnose → postmortem; eval harness + CI | ✅ done — spine built & unit-tested |\n| 2 | Reliability core: verifier pass, deterministic fabrication check, abstention, LLM-as-judge | ✅ built — keyless deterministic gate + opt-in verifier/judge; first real run passed with zero fabrication (DR-0016/DR-0017). Judge validation + a reliability *rate* carry into Wave 3 |\n| 3 | Breadth: config/env + resource-exhaustion classes, metrics, ~50 scenarios | ✅ done — 3 classes across a 65-scenario suite; first full run **61/65, 0 fabricated**; judge validated (kappa 0.81). See the [reliability](docs/case-studies/wave3-reliability-rate.md) + [judge](docs/case-studies/wave3-judge-validation.md) case studies |\n| **4** | **Cost / fine-tune: QLoRA Qwen3-4B vs base vs frontier, with/without verifier** | ✅ **done** — base **0/16 → tuned 12/16** holdout (0 fabricated, 0 speculative-filter, cheaper than base); frontier-competitive vs Gemma-4-31B (beats it 10/16 on capability, ties 6/12 on abstention); `resource_exhaustion` unlearned + adversarial abstention a shared 6/12 ceiling ([case study](docs/case-studies/wave4-qwen-finetune.md), DR-0019/DR-0020) |\n| **5** | Polish & ship: HTML render, security pass, MCP registry, launch | 🚧 **engineering complete — release-gated** (HTML render + security scanners + threat model + registry/OIDC scaffolding done; the release tag + launch are the remaining steps) |\n| 6 | Resolution-verification loop | ⤳ folded into v2 (Wave 9) |\n| **v2 (7–9)** | Live incident-response service (webhook → concurrent workers → persisted runs → HITL review → Slack/HTML → sandbox resolution re-check) + Dockerfile/compose | ✅ **built** — Waves 7–9 shipped: signed webhook → concurrent workers → persisted cited runs → fail-closed HITL review gate → Slack/HTML → deterministic sandbox resolution re-check; non-root Docker + compose; **339 keyless tests**, frozen diff empty ([DR-0023](docs/quellgeist-adr-log.md)/[0027](docs/quellgeist-adr-log.md)/[0028](docs/quellgeist-adr-log.md), [spec](docs/quellgeist-v2-spec.md)) |\n| v2 Track B (10) | Reliability track: timing-aware verifier + structure-varied / out-of-structure generalisation eval | 🚧 scoped — DR-0024–0026 |\n\nThe wave boundary is deliberate, not unfinished: only the current wave is built in\ndetail, and later waves are scoped but intentionally unimplemented. **v2 is additive\nover the proven v1 core — the frozen fine-tune measurement surface is never touched\n([DR-0023](docs/quellgeist-adr-log.md); guarded by `tests/frozen/`).**\n\n## Reliability gate\n\nThe deterministic CI gate is the reliability contract: **339 tests** (ruff +\nblack via pre-commit, then `pytest` — covering the loop's never-crash /\ngraceful-abstention behaviour, the deterministic fabrication check and\ncite-based judge gate, the verifier and advisory LLM-judge, parameterised\nscenario generation, the judge-validation harness, the server filters, the\npostmortem renderer, the fixture-backed eval harness, the real-data ingestion +\nrobustness layer, an end-to-end real-incident harness, and the v2 live service —\nsigned webhook, concurrent per-incident isolation, HITL review gate, and the\ndeterministic sandbox resolution check) on Python 3.12 and 3.13.\n\nOut of band, the **model-driven eval** runs the reasoner over the 65-scenario\nsuite. The latest full run scored **61/65 passed, 0 fabricated evidence**\n(Cerebras Gemma-4-31B) — per-class breakdown + the failure analysis in the\n[reliability case study](docs/case-studies/wave3-reliability-rate.md).\n\n```bash\nuv run pytest tests/ -q\nuv run pre-commit run --all-files\n```\n\n## Development & contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for the dev setup, conventions, and the\nwave model; [SECURITY.md](SECURITY.md) for reporting and the no-secrets /\ntoy-demo policy; and [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) for community\nexpectations. Bug reports and feature requests use the\n[issue templates](.github/ISSUE_TEMPLATE); PRs follow the\n[PR template](.github/PULL_REQUEST_TEMPLATE.md).\n\n## License\n\n[MIT](LICENSE) © Rajeev Shyam Kumar.\n",
"bytes": 20236,
"sha": "29995667a407d7bc2a6ca626b6f35bd98d306e4578439f5ed7f83420a8427666",
"repo_slug": "rajeev-shyam/quellgeist",
"fonte": "repo",
"truncated": false,
"api": "https://agentalog.com/api/listings/mcp_io_github_rajeev_shyam_quellgeist_metric_9804b3ce/readme"
}