io.github.kubently/kubently
Troubleshoot Kubernetes agentically: natural-language cluster diagnosis via ask_kubently
Open source Open in the app JSON README (API)
About
Troubleshoot Kubernetes agentically: natural-language cluster diagnosis via ask_kubently
Details
- Kind
- MCP servers
- Topic
- Cloud & DevOps
- Publisher
- kubently
- Origin
- official
- Category
- ferramentas
- Transport
- local
- Version
- 2.3.1
- Stars
- 45
- Forks
- 4
- Open pull requests
- 7
- Last push
- 2026-08-31T04:15:04Z
- Repository state
- ativo
- Language
- Python
- License
- Apache-2.0
- Added
- 2026-08-29 04:00:22
- Updated
- 2026-08-29 04:00:22
- Origin id
io.github.kubently/kubently
README
# Kubently
[](LICENSE)
[](https://www.python.org/)
[](https://kubernetes.io/)
[](https://a2a-protocol.org/)
[](https://www.docker.com/)
[](https://helm.sh/)
[](SECURITY.md)
[](CONTRIBUTING.md)
Kubently - Troubleshooting Kubernetes Agentically
## Overview
Kubently (*Kubernetes + Agentically*) is a **free, self-hosted, vendor-neutral multi-cluster Kubernetes troubleshooter**. Ask one question, get AI-diagnosed answers from every cluster in your fleet in parallel — including clusters you can't reach directly: executors dial **outbound** to the central API, so there's no inbound ingress, no shared kubeconfig, and no per-cluster credentials to distribute.
Agents collaborate over the [A2A (Agent-to-Agent) protocol](https://a2a-protocol.org/latest/), and any MCP client (Claude Code, Cursor, Claude Desktop) can use Kubently as a tool out of the box.
## Key Features
- **Multi-Cluster Fleet Troubleshooting**: One question fans out across all registered clusters in parallel
- **Outbound-Dial Executors**: Reach clusters behind firewalls/NAT — no inbound ingress, no shared kubeconfig
- **Natural Language Interface**: Conversational Kubernetes troubleshooting and debugging
- **Comprehensive Analysis**: Automated issue detection, root cause analysis, and solution recommendations
- **Multi-LLM Support**: Compatible with Google Gemini, OpenAI, Anthropic, and other providers
- **A2A Protocol**: Industry-standard agent-to-agent communication for complex workflows
- **MCP Server**: Optional [Model Context Protocol](docs/MCP.md) endpoint so MCP clients (Claude Desktop, Cursor, custom agents) get direct tool access
- **Security-First**: API key authentication, OAuth/OIDC support, and TLS with cert-manager
- **Persistent Sessions**: Redis-backed conversation history and context management
- **Extensive Tool Suite**: kubectl integration, log analysis, resource inspection, and more
## Quick Start
### For Users: Get Started in 5 Minutes
Point `kubectl` at any cluster (kind, minikube, or real) and run:
```bash
npm install -g @kubently/cli
kubently install
```
That's it. The CLI installs Kubently via Helm, wires up secrets and the
executor, port-forwards the API, and drops you into a debug chat:
```
kubently> why is my nginx pod crashlooping?
```
You'll need an LLM API key (Anthropic, OpenAI, or Google) — the installer
prompts for it, or reads `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` /
`GOOGLE_API_KEY` from your environment. Use `--provider` to pick the LLM,
`--chart ./deployment/helm/kubently` to install from a local checkout, and
`kubently install --help` for everything else.
### Use from Claude Code / Cursor (MCP)
Already ran `kubently install`? Add Kubently to Claude Code:
```bash
claude mcp add kubently -- kubently mcp
```
Or connect directly over HTTP (no bridge process):
```bash
claude mcp add --transport http kubently http://localhost:8080/mcp/ \
--header "X-API-Key: <your-api-key>"
```
Then ask Claude things like *"use kubently to figure out why payments pods are
crashlooping"*. Any MCP client works — see [docs/MCP.md](docs/MCP.md) for
Cursor and generic configuration.
### Proactive diagnosis (Alertmanager → Slack)
Set `api.env.SLACK_WEBHOOK_URL` to a Slack incoming-webhook URL and point
Alertmanager at Kubently:
```yaml
receivers:
- name: kubently
webhook_configs:
- url: https://<your-kubently-host>/webhooks/alertmanager
http_config:
http_headers: # Alertmanager >= 0.28
X-API-Key:
secrets: ["<your-api-key>"]
```
Each firing alert is diagnosed by the agent and the result is posted to Slack —
the bot often explains the root cause before you've opened your laptop.
### Scheduled fleet health digest
Alerts are reactive. A digest sweeps *every* registered cluster on a schedule and
posts one summary to the same Slack webhook — healthy clusters collapse to a
single line, so what's left is what needs you.
```yaml
fleetReport:
enabled: true
schedule: "0 13 * * 1-5" # weekday mornings
```
Preview it before you schedule it — `dry_run` returns the digest and posts
nothing:
```bash
curl -X POST https://<your-kubently-host>/webhooks/fleet-report \
-H "X-API-Key: <your-api-key>" -H 'Content-Type: application/json' \
-d '{"dry_run": true}'
```
The digest question is yours to change. Pass `query` in that request to try one
immediately, then keep the wording you like via `fleetReport.query` in values:
```yaml
fleetReport:
query: |-
Check every cluster for pods restarting more than 5 times and for PVCs above
85% usage. One line per healthy cluster. No preamble.
```
To run the real scheduled path once — image, secrets, in-cluster URL and all:
```bash
kubectl create job --from=cronjob/kubently-fleet-report fleet-report-test -n kubently
```
### Deployment verification (did that deploy actually work?)
Tell Kubently what you just deployed and it watches the rollout settle, then
runs a real investigation — pods ready? events clean? errors in the new logs?
metrics regressed vs the pre-deploy window (when Prometheus is configured)? —
and posts a PASS/FAIL verdict with the evidence to Slack. Wire it into the last
step of your CI pipeline:
```bash
curl -X POST https://<your-kubently-host>/webhooks/verify-deployment \
-H "X-API-Key: <your-api-key>" -H 'Content-Type: application/json' \
-d '{"cluster": "prod-east", "namespace": "shop",
"workload": "deploy/checkout-api", "context": "v1.42.0"}'
```
Add `"dry_run": true` to get the verdict back synchronously without posting.
No CI access? Label the workload instead — `kubently.io/verify=enabled` — and
enable `verifyDeployment.watch` in values: Kubently notices every generation
change and verifies the rollout unprompted.
### Scheduled checks (your recurring questions, on cron)
The digest asks one broad question. Scheduled checks let you ask *your*
questions on *their* schedules — each check is a named prompt with a cron
schedule and optional target clusters, run by the agent and posted to Slack:
```yaml
scheduledChecks:
enabled: true
checks:
- name: cert-expiry
schedule: "0 8 * * 1" # Monday mornings
prompt: |-
Check TLS secrets for certificates expiring within 21 days.
List each as namespace/name with days remaining.
- name: pvc-pressure
schedule: "0 */6 * * *"
clusters: [prod-east]
prompt: Find PersistentVolumeClaims above 85% usage.
```
A passing check posts **nothing** — silence means green (set `notifyOnPass:
true` to hear about passes too). Failures always post, evidence included.
Iterate on a check without waiting for cron:
```bash
curl -X POST https://<your-kubently-host>/webhooks/scheduled-check \
-H "X-API-Key: <your-api-key>" -H 'Content-Type: application/json' \
-d '{"check": "cert-expiry", "dry_run": true}'
```
**📖 See [QUICK_START.md](docs/QUICK_START.md) for full quick-start guide**
**📚 See [GETTING_STARTED.md](docs/GETTING_STARTED.md) for production deployment**
### For Developers: Local Testing
```bash
# Deploy to a local kind cluster (builds images from HEAD)
ANTHROPIC_API_KEY=sk-... ./deployment/scripts/kind-e2e.sh
# Run comprehensive test suite
./test-automation/run_tests.sh test-and-analyze --api-key test-api-key
```
**📖 See [CLAUDE.md](CLAUDE.md) for development guidelines**
## Configuration
### LLM Providers
Pick a provider with `LLM_PROVIDER` and supply that provider's key. There is
no default provider — the agent refuses to start without `LLM_PROVIDER`. For
local development with `deployment/docker-compose.yaml`, put both in `.env`
(see `deployment/.env.example`):
```bash
# Anthropic
LLM_PROVIDER=anthropic-claude
ANTHROPIC_API_KEY=your-anthropic-api-key
# OpenAI (also matches Azure and OpenAI-compatible endpoints)
LLM_PROVIDER=openai
OPENAI_API_KEY=your-openai-api-key
# Google Gemini
LLM_PROVIDER=google-gemini
GOOGLE_API_KEY=your-gemini-api-key
```
In Kubernetes the keys come from the `kubently-llm-secrets` secret and
`LLM_PROVIDER` goes under `api.env`.
### Helm Deployment
Customize deployment using Helm values:
Kubently ships as a single chart. Its components are switched on and off with
`api.enabled`, `redis.enabled` and `executor.enabled` — an executor-only
install on a remote cluster is the same chart with the first two disabled.
```bash
# From a checkout
helm install kubently ./deployment/helm/kubently -n kubently \
-f deployment/helm/test-values.yaml
# Or from the published chart repository
helm repo add kubently https://kubently.github.io/kubently
helm install kubently kubently/kubently -n kubently -f my-values.yaml
```
`LLM_PROVIDER` is required and has no chart default — set it under `api.env`
(`anthropic-claude`, `openai`, or `google-gemini`). See
[ENVIRONMENT_VARIABLES.md](docs/ENVIRONMENT_VARIABLES.md) for the full
configuration surface, and [GETTING_STARTED.md](docs/GETTING_STARTED.md) for
the production walkthrough.
### Operator Runbooks
Feed your organization's tribal knowledge into investigations. Runbooks are
hand-written markdown files with lightweight frontmatter; when an
investigation (a chat question, an Alertmanager alert, or an A2A call)
matches a runbook's criteria, the agent receives it as "the operator's
runbook for this situation", follows it where applicable, notes deviations,
and cites it by name in the diagnosis.
A worked example:
```markdown
---
name: Payments CrashLoopBackOff
match:
alerts: ["KubePodCrashLooping", "PaymentsPod*"] # alert-name globs
namespaces: ["payments", "payments-*"] # namespace selectors
workloads: ["payment-api*"] # matches derived pod names too
topics: ["crashloop", "OOMKilled", "payment service"] # free-text tags
---
1. Check recent deploys first: payment-api ships through ArgoCD, and 90% of
crashloops here follow a bad config sync.
2. OOMKilled almost always means the JVM heap flag drifted from the container
memory limit — compare `-Xmx` against `resources.limits.memory` before
blaming traffic.
3. If the DB connection pool is exhausted, do NOT restart the pods; escalate
to #payments-oncall (restarts thundering-herd the database).
```
Deploy runbooks as Helm values (they become a ConfigMap mounted into the API
pod; edits go live without a pod restart):
```yaml
# production-values.yaml
runbooks:
payments-crashloop.md: |
---
name: Payments CrashLoopBackOff
match:
alerts: ["KubePodCrashLooping"]
namespaces: ["payments"]
---
1. Check recent deploys first ...
```
Matching is scored: an alert-name hit outranks namespace/workload selector
hits, which outrank topic hits. The best match is injected first, and the
total injected size is capped (`KUBENTLY_RUNBOOKS_MAX_CHARS`, default 8000
characters) — one complete, best-matching runbook beats fragments of many.
Outside Helm, point `KUBENTLY_RUNBOOKS_DIR` at any directory of `.md` files.
### Incident History
Past diagnoses become searchable institutional memory. Whenever an
investigation concludes with a root cause, Kubently stores a compact record —
date, cluster, resources involved, symptom keywords, the root-cause
one-liner, and the resolution when one was stated — in Redis, isolated per
authenticated caller (the same namespace boundary as conversation memory, so
in multi-tenant deployments one tenant's incidents are never visible to
another).
The history is used two ways:
- The agent's **`search_past_incidents`** tool answers "have we seen this
before?" — keyword search over resources, clusters, symptoms and
root-cause text, newest first.
- **Auto-surface**: when a new investigation strongly matches a past incident
(same resources/symptoms/cluster), a one-line
`SIMILAR PAST INCIDENT (date): <root cause>` note is injected into
context — framed as something to *verify against fresh evidence*, never to
assume. When a past incident materially informs the diagnosis, the RCA
cites it ("same root cause as the 2026-07-03 incident").
This is retrieval over stored summaries, not a learning system: records are
plain data with a TTL (default 90 days, `KUBENTLY_INCIDENT_TTL_SECONDS`) and
a per-tenant cap (default 200, `KUBENTLY_INCIDENT_MAX_PER_NAMESPACE`,
oldest evicted). The feature is on by default; set
`KUBENTLY_INCIDENT_HISTORY=false` (Helm: under `api.env`) to disable both
recording and retrieval.
## Architecture
- **API Server**: FastAPI-based REST API for cluster management and authentication
- **A2A Server**: Implements A2A protocol with LangGraph for workflow orchestration
- **Test Automation**: Comprehensive testing framework with 20+ Kubernetes scenarios
- **CLI Tools**: Modern Node.js CLI for interactive debugging
### Agent Toolset
The diagnostic agent investigates with a small set of read-only tools:
- **`list_clusters`** — enumerate registered clusters
- **`execute_kubectl`** — read-only kubectl against one cluster (whitelist-enforced on the executor)
- **`execute_kubectl_multi`** — one read-only kubectl command fanned out across many clusters
- **`get_recent_changes`** — "what changed?" timeline for a workload or namespace: rollouts (ReplicaSet revisions + change-causes), Helm release history *(opt-in: `changeCorrelation.helmHistory.enabled`)*, ArgoCD sync history *(optional: `changeCorrelation.argocd.url`)*, and Normal+Warning events — correlated against first-error timestamps in the RCA
- **`get_events_for_resource`** — chronological events for a resource and its children (deployment → replicasets → pods)
- **`search_pod_logs`** — structured log search across every pod/container matching a label selector (substring or regex, time bounds, previous-container support). Logs are filtered on the cluster's executor so only matching lines — capped, with explicit truncation notes — come back
- **`query_loki`** *(optional)* — LogQL range queries against a cluster's Loki for aggregated/historical log search, including logs from pods that have restarted or been deleted. Enabled by setting `loki.url` in Helm values (unset by default); queries execute on each cluster's executor through the same outbound channel as kubectl commands
- **`query_prometheus`** *(optional)* — instant and range PromQL queries for latency, saturation, OOM-trend and restarts-over-time evidence. Enabled by setting `prometheus.url` in Helm values (unset by default); queries execute on each cluster's executor through the same outbound channel as kubectl commands
- **`search_past_incidents`** — keyword search over this deployment's incident history (see below). On by default when Redis is available; disable with `KUBENTLY_INCIDENT_HISTORY=false`
- **`get_manifest_file`** *(optional)* — read-only fetch of a file from the configured GitOps manifests repo, so proposed fixes are diffed against the real manifest instead of a hallucinated one. Enabled with `gitRemediation` in Helm values (off by default)
- **`propose_fix_pr`** *(optional)* — proposes a high-confidence manifest fix as a **pull request** against the configured GitOps manifests repo (GitHub or GitLab): branch → commit → PR with the investigation evidence in a body clearly marked machine-proposed. The agent **never merges** — a human reviews and merges, and your GitOps controller applies. Size-capped (files/changed lines), token never enters model context, cluster access stays read-only. See [GitOps PR Remediation](docs/GITOPS_REMEDIATION.md)
- **`query_cloud_logs`**, **`query_cloud_metrics`**, **`get_recent_cloud_changes`** *(optional)* — read-only cloud telemetry for a cluster: CloudWatch Logs Insights / CloudWatch metrics / CloudTrail on AWS, Cloud Logging / Cloud Monitoring / GKE audit logs on GCP. The executor answers them using the workload identity **you** grant its ServiceAccount (EKS Pod Identity, IRSA, or GKE Workload Identity) — no cloud key is ever stored by Kubently, and revoking the IAM role kills the capability instantly. Enabled per cluster with `executor.cloud.enabled` in Helm values (off by default); each call re-checks that the target cluster's executor actually reports an identity. Operations are additionally limited by a code-level allowlist. See [Cloud Telemetry](docs/CLOUD_TELEMETRY.md)
- **`mcp_<server>_*`** *(optional)* — tools from external MCP servers (streamable HTTP, e.g. Grafana Cloud's or Datadog's remote MCP) configured via `mcpServers` in Helm values (unset by default). Tool names are prefixed with the server name to avoid collisions; results are treated as untrusted input (framed and size-capped) and credentials stay in Kubernetes secrets. **Connect read-scoped servers/credentials only** — Kubently cannot enforce read-only semantics on a remote server's tools. See `docs/MCP_CLIENT_TOOLS.md`
## Documentation
### Getting Started
- **[Quick Start Guide](docs/QUICK_START.md)** - Get running in 5 minutes
- **[Getting Started](docs/GETTING_STARTED.md)** - Complete setup for production use
- **[Deployment Guide](docs/DEPLOYMENT.md)** - Detailed deployment options and configuration
### Usage & Operations
- **[CLI Admin Guide](docs/GETTING_STARTED.md#step-5-register-and-deploy-executors)** - Managing clusters and executors
- **[Test Queries](docs/TEST_QUERIES.md)** - Example API requests and A2A protocol usage
- **[MCP Connect Guide](docs/MCP.md)** - Connect MCP clients (Claude Desktop, Cursor, custom agents)
- **[Environment Variables](docs/ENVIRONMENT_VARIABLES.md)** - Configuration reference
- **[GitOps PR Remediation](docs/GITOPS_REMEDIATION.md)** - Agent-proposed fix PRs (human-merged, default off)
- **[Cloud Telemetry](docs/CLOUD_TELEMETRY.md)** - Read-only CloudWatch / Cloud Logging access via workload identity (default off)
- **[External MCP Tools](docs/MCP_CLIENT_TOOLS.md)** - Mounting third-party MCP servers into the agent, including per-request injection
### Architecture & Development
- **[Architecture](docs/ARCHITECTURE.md)** - System design and components
- **[A2A Protocol Spec](https://a2a-protocol.org/latest/)** - Official protocol documentation
- **[Development Guide](CLAUDE.md)** - Guidelines for contributors
### Troubleshooting
- **[Getting Started - Common Issues](docs/GETTING_STARTED.md#common-issues)** - Troubleshooting guide
## Contributing
See [CLAUDE.md](CLAUDE.md) for development guidelines and contribution instructions.
## Maintainer
**Kubently Team** - [hello@kubently.io](mailto:hello@kubently.io)
## License
Apache 2.0 License - See LICENSE file for details