{
  "markdown": "# k8s-troubleshoot-mcp\n\nA read-only [MCP (Model Context Protocol)](https://modelcontextprotocol.io/)\nserver that lets an AI assistant diagnose a Kubernetes cluster. Ask why a pod is\ncrash-looping instead of running six `kubectl` commands and correlating the\noutput by hand.\n\n**Read-only is a structural property, not a promise.** There are no write tools,\nand the RBAC manifests grant no write verbs. See [Security model](#security-model).\n\n## What It Does\n\nThe server exposes 16 diagnostic tools over stdio. Connect it to Claude Desktop,\nVS Code, or Kiro, and you can ask things like:\n\n- \"Why is the `checkout` pod in `staging` not ready?\"\n- \"Show me the last 50 lines of logs from the `api` container\"\n- \"Which nodes are not Ready, and what are their taints?\"\n- \"Is the `web` HPA scaling, and what does its current metric say?\"\n- \"What events fired in `production` in the last few minutes?\"\n\nThe assistant calls the tools, the server queries the Kubernetes API with a\nscoped ServiceAccount token, and every response comes back as a structured dict —\nincluding errors, which are never raised as exceptions into the MCP layer.\n\n## Architecture\n\n```\n┌──────────────────┐    stdio (JSON-RPC)   ┌──────────────────────────┐\n│   MCP Client     │◄─────────────────────►│   MCP Server             │\n│  Claude Desktop  │                       │   (this project)         │\n│  VS Code / Kiro  │                       │                          │\n└──────────────────┘                       │  config.py   ─ validate  │\n                                           │  server.py   ─ 16 tools  │\n                                           │  tools/*.py  ─ read+shape│\n                                           │  response.py ─ escape +  │\n                                           │                structure │\n                                           └───────────┬──────────────┘\n                                                       │ HTTPS, explicit\n                                                       │ KUBECONFIG only\n                                           ┌───────────▼──────────────┐\n                                           │  Kubernetes API server   │\n                                           │  ── RBAC boundary ──     │\n                                           │  ServiceAccount:         │\n                                           │  get/list/watch only     │\n                                           └──────────────────────────┘\n```\n\nConfiguration is validated once at startup. If anything is wrong — `KUBECONFIG`\nunset, the file unreadable or malformed, `ALLOWED_NAMESPACES` missing or\ncontaining a wildcard — the process writes one line to stderr and exits 1. It\nnever starts in a partially-valid state.\n\n## Available Tools\n\nArguments marked `?` are optional.\n\n| Tool | Description | Parameters |\n|------|-------------|------------|\n| `get_pod_status` | Phase, conditions, container statuses, QoS class and node for a pod | `pod_name`, `namespace` |\n| `get_pod_logs` | Recent log lines from a pod container. Content is untrusted — see [Reading `get_pod_logs` output](#reading-get_pod_logs-output) | `pod_name`, `namespace`, `container?`, `previous?`, `tail_lines?` |\n| `get_pod_events` | Recent events for a pod, newest first, with `total_available` | `pod_name`, `namespace` |\n| `list_pods` | Pods in a namespace with phase, restart count and readiness | `namespace`, `label_selector?` |\n| `get_node_status` | Conditions, capacity, allocatable, taints and roles for a node | `node_name` |\n| `list_nodes` | Cluster nodes with readiness, roles, age and kubelet version | none |\n| `get_deployment_status` | Replica counts, conditions and rollout strategy | `deployment_name`, `namespace` |\n| `list_deployments` | Deployments in a namespace with replica counts and availability | `namespace` |\n| `get_statefulset_status` | Replica counts, revisions and update strategy | `statefulset_name`, `namespace` |\n| `get_daemonset_status` | Scheduling counts and update strategy | `daemonset_name`, `namespace` |\n| `get_service` | Type, ClusterIP, ports, selector and ready endpoint count | `service_name`, `namespace` |\n| `get_endpoints` | Ready and not-ready endpoint addresses backing a service | `service_name`, `namespace` |\n| `get_pvc_status` | Phase, capacity, binding and resize state for a PVC | `pvc_name`, `namespace` |\n| `get_hpa_status` | Replica bounds, current metrics and conditions for an HPA | `hpa_name`, `namespace` |\n| `get_namespace_events` | Recent events across a namespace, newest first, with `total_available` | `namespace`, `limit?` |\n| `list_namespaces` | The namespaces this server is permitted to read | none |\n\nEvery namespaced tool validates its `namespace` argument **before** making any\nAPI call, so a disallowed namespace produces a structured error and no network\nrequest.\n\n### Deliberately absent\n\nNo `get_secrets`, `get_configmap`, `exec_into_pod`, `port_forward`, or any\n`create`/`update`/`patch`/`delete` tool. These are excluded from all versions\nunless a new threat-model review is conducted and documented — they are not\nbacklog items. The reasoning for each is in\n[SECURITY.md](SECURITY.md#what-this-server-is-not--deliberate-exclusions).\n\n## Security model\n\nFull detail is in [SECURITY.md](SECURITY.md). The summary:\n\n### The boundary is Kubernetes RBAC\n\n**Everything this server does in application code is defense-in-depth. The\nenforcement boundary is the ServiceAccount's RBAC bindings.** If the bindings\ngrant more than intended, the application-layer allowlist is all that stands in\nthe way, and it is not a boundary you should rely on.\n\nProvisioning is split by scope so that the cluster-scoped grant is minimal:\n\n| Manifest | Scope | Grants |\n|----------|-------|--------|\n| `clusterrole.yaml` + `clusterrolebinding.yaml` | cluster | `get`/`list`/`watch` on `nodes` and `namespaces` only |\n| `role.yaml` | namespace | `get`/`list`/`watch` on the diagnostic resources |\n| `rolebinding.yaml.template` | namespace | binds the Role, one namespace at a time |\n\nApplying the cluster-scoped pair makes **no namespace readable**. A namespace\nbecomes readable only when a Role *and* a RoleBinding exist in it. A namespace\nlisted in `ALLOWED_NAMESPACES` but never bound stays unreadable — RBAC wins.\n\n`pods/log` is granted in its own rule block, never folded into the `pods` rule,\nbecause Kubernetes subresources do not inherit from their parent.\n\n### Defense-in-depth layers\n\n| Layer | What it does | What it is not |\n|-------|--------------|----------------|\n| **RBAC** | Grants read verbs on diagnostic resources in bound namespaces only | — this *is* the boundary |\n| **Explicit kubeconfig** | Reads `KUBECONFIG` from an exact path; no `~/.kube/config`, no in-cluster config, no fallback chain | Not a permission check — it prevents silently picking up an ambient credential |\n| **Namespace allowlist** | Rejects wildcards, strips `kube-system`/`kube-public`, validates before every call | Advisory; a bug here is contained by RBAC |\n| **Output escaping** | All cluster-authored free text routed through `serialize_log_content` | Prevents breaking out of a JSON string; cannot stop a model acting on legible instructions |\n| **Structured errors** | Every failure returns a dict; no exception reaches the MCP layer | — |\n\n### Prompt injection is mitigated, not eliminated\n\nPod logs and event messages are written by workloads in the cluster. A container\ncan print anything, including text shaped like instructions to the model reading\nit. Escaping keeps injected text inside its JSON string; it cannot stop a model\nfrom acting on instructions it reads as data. **Treat tool output as untrusted\ninput to whatever consumes it.** This residual risk is accepted and documented.\n\n## Prerequisites\n\n1. **A Kubernetes cluster** and a `kubectl` context with enough permission to\n   create a ServiceAccount, Role, RoleBinding, ClusterRole and\n   ClusterRoleBinding — you need this once, to provision. The server itself\n   never uses your admin credential.\n2. **Kubernetes 1.24+** — `scripts/generate-kubeconfig.sh` mints a token via the\n   TokenRequest API, not a legacy auto-mounted Secret.\n3. **Python 3.11+**\n4. **uv** — `curl -LsSf https://astral.sh/uv/install.sh | sh` (or use Docker,\n   which needs neither Python nor uv on the host)\n\n## Setup\n\n```bash\ngit clone https://github.com/NanaGyamfiPrempeh30/k8s-troubleshoot-mcp.git\ncd k8s-troubleshoot-mcp\n\n# Install dependencies (uv creates .venv automatically)\nuv sync\n\n# Run the test suite\nuv run pytest tests/ -q\n```\n\n### Provision RBAC and mint a kubeconfig\n\n```bash\nscripts/generate-kubeconfig.sh /secure/path/k8s-mcp-kubeconfig.yaml staging production\n```\n\nThe first argument is where to write the kubeconfig; the rest are the namespaces\nthe server may read. Pass the same set you intend to put in\n`ALLOWED_NAMESPACES` — RBAC is the enforcement boundary, and a namespace bound\nhere but absent from the allowlist (or the reverse) is a mismatch between real\npermission and configured capability.\n\nThe script applies the cluster-scoped manifests together, then applies\n`role.yaml` with an explicit `-n <namespace>` and renders a RoleBinding per\nnamespace. On success it prints the kubeconfig path to stdout and nothing else;\nall diagnostics go to stderr. It also asserts after provisioning that\n`kubectl auth can-i get secrets` returns `no`, and aborts if it does not.\n\n> **Do not run `kubectl apply -f kubernetes/`.** It does not fail — it reports\n> success while creating `role.yaml` in the *current* namespace and skipping\n> `rolebinding.yaml.template` entirely, because `kubectl apply -f <dir>` only\n> reads `.yaml`/`.yml`/`.json`. The result is a server that looks provisioned\n> and can read nothing. Verified against a v1.35 API server with\n> `--dry-run=server`: 5 resources applied, not 6.\n\nThe generated kubeconfig is written with `umask 077` and `chmod 600`. Keep it\nout of the repository — the script warns if the output path is inside a\nrepository and not covered by `.gitignore`.\n\n### Run it\n\n```bash\nKUBECONFIG=/secure/path/k8s-mcp-kubeconfig.yaml \\\nALLOWED_NAMESPACES=staging,production \\\nuv run k8s-troubleshoot-mcp\n```\n\nThe server speaks JSON-RPC on stdin/stdout, so it will appear to hang — that is\ncorrect. It is waiting for a client.\n\n## Connecting to Claude Desktop\n\nAdd to `claude_desktop_config.json`\n(`%APPDATA%\\Claude\\claude_desktop_config.json` on Windows,\n`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS):\n\n```json\n{\n  \"mcpServers\": {\n    \"k8s-troubleshoot\": {\n      \"command\": \"uv\",\n      \"args\": [\"run\", \"k8s-troubleshoot-mcp\"],\n      \"cwd\": \"/path/to/k8s-troubleshoot-mcp\",\n      \"env\": {\n        \"KUBECONFIG\": \"/secure/path/k8s-mcp-kubeconfig.yaml\",\n        \"ALLOWED_NAMESPACES\": \"staging,production\"\n      }\n    }\n  }\n}\n```\n\nRestart Claude Desktop fully (quit from the system tray, reopen), then check\nSettings → Developer → `k8s-troubleshoot` shows **running**.\n\nOn Windows, if the server shows as disconnected, use a batch file wrapper —\nClaude Desktop has working-directory issues with direct interpreter invocation:\n\n```bat\n@echo off\ncd /d C:\\Users\\YourUsername\\k8s-troubleshoot-mcp\nuv run k8s-troubleshoot-mcp\n```\n\n```json\n{\n  \"mcpServers\": {\n    \"k8s-troubleshoot\": {\n      \"command\": \"cmd.exe\",\n      \"args\": [\"/c\", \"C:\\\\Users\\\\YourUsername\\\\k8s-troubleshoot-mcp\\\\run_mcp.bat\"],\n      \"env\": {\n        \"KUBECONFIG\": \"C:\\\\secure\\\\path\\\\k8s-mcp-kubeconfig.yaml\",\n        \"ALLOWED_NAMESPACES\": \"staging,production\"\n      }\n    }\n  }\n}\n```\n\n## Running with Docker\n\n```bash\ndocker build -t k8s-troubleshoot-mcp .\n\ndocker run -i --rm \\\n  -v /secure/path/k8s-mcp-kubeconfig.yaml:/kubeconfig:ro \\\n  -e KUBECONFIG=/kubeconfig \\\n  -e ALLOWED_NAMESPACES=staging,production \\\n  k8s-troubleshoot-mcp\n```\n\n`-i` is required — JSON-RPC travels on stdin/stdout. `:ro` is not decoration:\nthis server performs no writes of any kind, so a writable mount would grant\nprivilege it has no use for.\n\nThe image runs as **non-root, UID 10001**, and contains no credentials.\n\nTwo things that will bite you:\n\n- **The kubeconfig must be readable by UID 10001.** A file created mode `600`\n  and owned by your host user is not, and bind mounts preserve host ownership.\n  Either grant group/other read, or run with `--user \"$(id -u)\"`.\n- **A cluster on the host's loopback** (minikube, kind) needs `--network host`\n  on Linux. On Docker Desktop that is not enough — see\n  [Local testing with minikube](#local-testing-with-minikube).\n\nTo use the container from Claude Desktop, set `\"command\": \"docker\"` and put the\nwhole `run -i --rm …` invocation in `\"args\"`.\n\n## Environment Variables\n\n| Variable | Required | Default | Description |\n|----------|----------|---------|-------------|\n| `KUBECONFIG` | **Yes** | — | Exact path to the kubeconfig. No fallback to `~/.kube/config` and no in-cluster config; a missing, unreadable or malformed file is a startup failure |\n| `ALLOWED_NAMESPACES` | **Yes** | — | Comma-separated namespaces the server may read. Wildcards (`*`, `all`) are rejected; `kube-system` and `kube-public` are stripped with a warning even if listed |\n| `LOG_LEVEL` | No | `INFO` | `DEBUG`, `INFO`, `WARNING`, `ERROR`. An unrecognized value falls back to `INFO` |\n| `API_TIMEOUT_SECONDS` | No | `30` | Must be a positive integer |\n| `MAX_LOG_LINES` | No | `200` | Must be a positive integer. Clamped to a hard ceiling of `1000` with a warning |\n\nAll logging goes to **stderr**. stdout carries the JSON-RPC stream exclusively —\na single stray `print()` would corrupt the protocol, so there are none in `src/`.\n\n## Reading `get_pod_logs` output\n\nThe `content` field is **escaped log text, not log text**. One `json.loads` of\nthe MCP response is not enough to get printable output.\n\n`serialize_log_content` escapes the log before the response envelope is\nJSON-encoded, so the escaping survives transport. Decoding the response undoes\nonly the transport layer:\n\n```python\nresp = json.loads(raw_mcp_response)\ncontent = resp[\"data\"][\"content\"]\n\nprint(content)\n# line one\\nline two\\tsaid \\\"hi\\\" \\u003cb\\u003e\\n   <- one physical line\n\ncontent.count(\"\\n\")   # 0  -- there are no real newlines in it\n```\n\nTo recover the original text, decode the field a second time:\n\n```python\ntext = json.loads('\"' + content + '\"')\n\nprint(text)\n# line one\n# line two    said \"hi\" <b>\n```\n\nThis is intentional, not a bug. The escaping is the structural prompt-injection\nmitigation (REQ-020, REQ-027) — if a single decode restored real control\ncharacters and raw `<`, the mitigation would be gone before the content reached\na model. Decode the second time only where the output is going somewhere that\nwill not interpret it, such as a terminal or a log file.\n\nIn MCP Inspector's raw view you will see `\\\\n` (two backslashes). That is\ncorrect: the transport layer escaping the backslash of an already-escaped `\\n`.\n\n**Do not compute a line count from `content`.** Splitting it on a newline\nreturns 1 for a log of any length. Use `lines_returned`, which is why it exists.\n\n## Local testing with minikube\n\n### Stale kubeconfig after a minikube restart\n\nminikube exposes the API server through a proxy address whose port is assigned\nper session. That address — visible via `kubectl cluster-info` — can change\nacross `minikube stop` / `minikube start` cycles, and across WSL2 restarts.\n\nA kubeconfig minted by `scripts/generate-kubeconfig.sh` during a previous\nsession pins the old `host:port`. The server will start normally and then fail\nevery tool call with a `connection_error` naming an endpoint nothing is\nlistening on:\n\n```\nKubernetes control plane is running at https://127.0.0.1:54489\n                                                        ^^^^^ per-session\n```\n\nTo recover:\n\n1. Confirm the current endpoint:\n\n   ```bash\n   kubectl cluster-info\n   ```\n\n2. Mint a fresh kubeconfig against it:\n\n   ```bash\n   scripts/generate-kubeconfig.sh /path/to/kubeconfig.yaml <namespace> [namespace...]\n   ```\n\n3. **Fully restart the MCP server process.** The server reads `KUBECONFIG` once\n   at startup (REQ-002) and never re-reads it, so overwriting the file\n   underneath a running server changes nothing. There is also no fallback to\n   `~/.kube/config` (REQ-003) — a working `kubectl` on the same machine will\n   not rescue a stale kubeconfig.\n\nThe same failure looks identical whether the cause is a stale port, a revoked\ntoken, or a genuinely unreachable cluster. `kubectl cluster-info` distinguishes\nthem: if it succeeds while the server reports `connection_error`, the\nkubeconfig is stale.\n\n### Reaching minikube from a container on Docker Desktop\n\n`--network host` joins the **Docker VM's** network namespace, not the WSL\ndistribution's, so the published minikube port is not on that loopback and the\ncontainer gets `ConnectionRefused`. `host.docker.internal` is reachable, but\nrewriting the kubeconfig's `server` to it fails TLS verification —\n`host.docker.internal` is not among the minikube API server certificate's SANs.\n\nPoint the URL at `host.docker.internal` **and** add `tls-server-name` to the\ncluster entry, which is the kubeconfig field that exists for exactly this:\n\n```yaml\nclusters:\n- name: minikube\n  cluster:\n    server: https://host.docker.internal:54489\n    tls-server-name: localhost\n    certificate-authority-data: ...\n```\n\n## Project Structure\n\n```\nk8s-troubleshoot-mcp/\n├── .github/workflows/\n│   ├── build-and-push.yml       # Docker build + push to Docker Hub\n│   └── secret-scan.yml          # TruffleHog secret scanning\n├── kubernetes/\n│   ├── namespace.yaml\n│   ├── serviceaccount.yaml      # automountServiceAccountToken: false\n│   ├── clusterrole.yaml         # nodes + namespaces, read verbs only\n│   ├── clusterrolebinding.yaml\n│   ├── role.yaml                # pods/log in its own rule block\n│   └── rolebinding.yaml.template\n├── scripts/\n│   ├── generate-kubeconfig.sh   # provisions RBAC, mints a scoped token\n│   └── check-namespaces.py      # CI guard: GitHub vs Docker Hub handles\n├── docs/\n│   ├── PUBLISHING.md            # Docker Hub + MCP Registry release runbook\n│   └── dockerhub-overview.md    # Docker Hub repository description\n├── server.json                  # MCP Registry listing metadata\n├── src/k8s_troubleshoot_mcp/\n│   ├── __main__.py              # startup sequence, fail-closed\n│   ├── config.py                # env validation (REQ-001..010, 069..071)\n│   ├── k8s_client.py            # explicit-path client factory\n│   ├── server.py                # FastMCP instance + 16 tool registrations\n│   ├── response.py              # serialize_log_content + structured errors\n│   ├── pagination.py            # total_available / continue-token detection\n│   └── tools/                   # pods, nodes, workloads, services,\n│                                #   storage, autoscaling, events, namespaces\n├── tests/\n│   ├── unit/\n│   └── property/                # P1-P18, Hypothesis-driven\n├── requirements.md              # EARS-format requirements\n├── design.md                    # architecture + 18 correctness properties\n├── SECURITY.md                  # threat model, RBAC boundary, exclusions\n├── DEBUG_LOG.md                 # every error found, root cause, resolution\n├── Dockerfile                   # multi-stage, non-root UID 10001\n├── uv.lock                      # 67 pinned dependencies\n└── README.md                    # this file\n```\n\n## Troubleshooting\n\n**`KUBECONFIG environment variable is not set`** — the server refuses to start\nwithout an explicit path and will not fall back to `~/.kube/config`. This is\ndeliberate (REQ-003). Run `scripts/generate-kubeconfig.sh` if you have not yet.\n\n**`is not a valid kubeconfig`** — the file exists and is readable but is\nmalformed. The message names the path, line and column, never the file's\ncontents; a kubeconfig holds a bearer token.\n\n**`connection_error` on every tool while `kubectl` works** — a stale kubeconfig.\nSee [Local testing with minikube](#local-testing-with-minikube).\n\n**`namespace_not_allowed`** — the namespace is not in `ALLOWED_NAMESPACES`, or\nit is `kube-system`/`kube-public`, which are stripped at startup even if listed.\n\n**`kubernetes_api_error` with `http_status: 403` while the namespace *is*\nallowed** — the allowlist and the RBAC bindings have diverged.\n`ALLOWED_NAMESPACES` grants nothing; a namespace is only readable once a Role\nand RoleBinding exist in it. Re-run `generate-kubeconfig.sh` with the full\nnamespace set.\n\nErrors come back as one of three codes: `namespace_not_allowed` (rejected before\nany API call), `kubernetes_api_error` (carries `http_status` and `reason` from\nthe API server), and `connection_error`.\n\n**Permission denied reading the kubeconfig in Docker** — the container runs as\nUID 10001 and bind mounts preserve host ownership. See\n[Running with Docker](#running-with-docker).\n\n**Log content looks like one long line with `\\n` in it** — that is the escaping\nworking. See [Reading `get_pod_logs` output](#reading-get_pod_logs-output).\n\n**MCP Inspector shows `Logging ✗`** — expected, not a defect. FastMCP does not\nregister a `set_logging_level` handler, so `get_capabilities()` omits the\ncapability. That MCP feature sends log records to the *client*; this server logs\nto stderr, which is unrelated. Inspector shows Resources and Prompts as\nsupported for the mirror-image reason — FastMCP registers those handlers\nunconditionally even though none are defined.\n\n## How this was built\n\n[DEBUG_LOG.md](DEBUG_LOG.md) records every error encountered during development\n— root cause and resolution for each, including several found only by running\nagainst a real cluster after the entire test suite was green.\n\nIt is worth reading if you are evaluating whether to trust this server with a\ncluster credential, because a recurring theme runs through it: **the test mocks\nagreed with each other rather than with the cluster.** Five separate defects\nwere found that way, each with a passing test asserting the opposite. The\nverification steps taken in response are summarized in\n[SECURITY.md](SECURITY.md#how-the-claims-in-this-document-were-verified).\n\n## Development\n\n```bash\nuv sync --extra dev\nuv run pytest tests/ -q\n```\n\nThe property tests (`tests/property/`) enumerate tools from a shared registry,\n`NAMESPACED_TOOLS` in `tests/property/strategies.py`. Any new tool must be added\nthere in the same change — a tool missing from the registry causes P4/P6/P7 to\nsilently stop covering it while still reporting green.\n\nInstall the TruffleHog pre-commit hook before your first commit (`pre-commit` is\nnot a project dependency — it is a developer tool installed alongside):\n\n```bash\npip install pre-commit\npre-commit install\n```\n\n## Roadmap\n\n- [x] 16 read-only diagnostic tools\n- [x] 18 correctness properties (P1-P18), Hypothesis-driven\n- [x] RBAC manifests + scoped kubeconfig generation\n- [x] Live-cluster validation\n- [x] Docker packaging (multi-stage, non-root, no baked credentials)\n- [x] TruffleHog secret scanning (pre-commit + GitHub Actions)\n- [x] Docker Hub + MCP Registry listing prepared ([docs/PUBLISHING.md](docs/PUBLISHING.md))\n- [ ] Migrate `get_endpoints` to `discovery.k8s.io/v1` EndpointSlice\n- [ ] Publish to Smithery\n- [ ] `get_ingress_status` and `get_networkpolicy` tools\n- [ ] HTTP transport for network-based deployment\n\n## Credits\n\n- [Model Context Protocol](https://modelcontextprotocol.io/)\n- [FastMCP](https://github.com/jlowin/fastmcp)\n- [Kubernetes Python client](https://github.com/kubernetes-client/python)\n- Built by [Yaw Nana Gyamfi Prempeh](https://github.com/NanaGyamfiPrempeh30)\n\n## License\n\nMIT\n",
  "bytes": 23517,
  "sha": "62d4b27cef1583500b6be81298808fbf7b094aec908acc0c3c399b5d67924ba0",
  "repo_slug": "nanagyamfiprempeh30/k8s-troubleshoot-mcp",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_nanagyamfiprempeh30_k8s_troubl_faa2980a/readme"
}