Kubernetes Health Checks with an SRE Agent Team

We pointed an AURA agent team at a Kubernetes cluster and asked a deliberately boring question: "Check the health of my cluster. Are all pods running?"

We expected a baseline. Instead, it came back with a pod in CrashLoopBackOff with 788 restarts. Nobody had asked about it.

Then it did something more useful than finding the problem. When part of its metrics work hit a provider rate limit, it didn't guess. It reported the analysis as incomplete.

This page shows the config behind that run, how to run it yourself, and why a coordinator with scoped specialists behaves differently from one AI assistant with every tool.

TL;DR

• What it is: An open-source, read-only Kubernetes SRE agent team. A coordinator plans the investigation and delegates to two specialists: one inspects the cluster, the other queries Prometheus.
• What you get back: Findings tied to the cluster and metric data they came from, plus an explicit statement when the evidence is incomplete.
• What it can change: Nothing, in this example. Both MCP servers run read-only, and each specialist can call only the tools on its allowlist.
• What you need: kubectl access to a cluster, a Prometheus endpoint, Docker, Node.js, and a model provider API key.
• Cost: AURA is free and open source (Apache 2.0). You pay only for your model usage.
• Source: examples/complete/kubernetes-sre-orchestrated-multi-model.toml

What this example does

One config file defines a three-agent team:

1. Coordinator. Breaks your question into sub-tasks, routes each one to the right specialist, and writes the final answer. It can answer simple questions directly or ask you to clarify.
2. Kubernetes discovery specialist. Lists namespaces, workloads, pods, nodes, events, and resource usage. It's limited to 14 read-only tools.
3. Prometheus analyst. Runs instant and range queries, checks scrape targets, and reads metric metadata. It's limited to 6 tools.

The config also builds in these guardrails:

• Default lookback. Telemetry checks cover the last 5 minutes unless you ask for a different window.
• Bounded planning. Planning is capped at 2 cycles, and each specialist is capped at 8 turns.
• Time limit. Each coordinator phase and each worker task has a 180-second budget.
• No guessing. Each specialist's instructions say to use its tools and never fabricate cluster state or metric values.
• A model for each role. The coordinator and the specialists run on different models. Changing any of them is a one-line config edit.

Who it's for

Jobs to be done

• On-call engineer: When something feels off in the cluster, I want one question to check both workload state and metrics, so I don't have to run kubectl and PromQL by hand and correlate them myself.
• Platform engineer: When my team wants an AI assistant for SRE work, I want a multi-agent setup where each agent sees only the tools it needs, so I'm not giving one model access to everything.
• SRE lead: When the agent can't finish an investigation, I want it to say so, so my team doesn't act on a confident wrong answer.
• Engineering leader: When I evaluate AI for operations, I want to start read-only on a real cluster, so I can judge the findings before discussing any write access.

Pain points it removes

• Manual correlation. Workload state lives in Kubernetes and health signals live in Prometheus. The coordinator queries both and joins the results.
• Blind spots in routine checks. A broad question surfaces problems nobody was looking for, like the crash-looping pod in the video on this page.
• Overconfident AI answers. Specialists are told not to fabricate, and the team reports incomplete analysis instead of filling gaps.
• Over-permissioned agents. Tool allowlists keep each specialist inside its job.
• Opaque reasoning. The plan, the worker assignments, and every tool call are observable and can be exported as OpenTelemetry traces.

What it replaces, and why it's better

The value it drives

• Earlier detection. Open-ended health checks catch issues that dashboards and alert rules weren't built for.
• Answers you can trust. Every finding traces to cluster or metric data, and gaps are labeled as gaps.
• Safe adoption path. Starting read-only gets AI into operations without any write-access decisions.
• Model flexibility. You can assign different models to the coordinator and the specialists to match each role's needs.

What to measure in a pilot: issues found that weren't already alerting, how often findings hold up on human review, and how often the team reports an incomplete analysis instead of guessing.

Why AURA

• Production-tested. AURA started as Mezmo's internal harness for operating our own SaaS. Our engineering and SRE teams still use it in production, and it powers thousands of agent sessions each month in Mezmo's hosted platform.
• Multi-agent orchestration built in. Coordinator planning, dependency-aware task execution, and per-worker tool and model assignment are config, not code.
• Runs inside your security boundary. Deploy in your own infrastructure, including air-gapped environments, with control over model, tool, storage, and telemetry destinations.
• Model-agnostic. OpenAI, Anthropic, Bedrock, Gemini, Ollama, or OpenRouter, per agent.
• Human approval when you want it. When you're ready to allow changes, you can put any tool that changes state behind webhook or in-chat approval. If approval is denied or times out, the change doesn't run.
• Traceable end to end. Every request, model turn, tool call, and orchestration step can be exported as OpenTelemetry traces.

Start from where you are

• Want a single agent first? kubernetes-sre.toml uses the same Kubernetes and Prometheus connections without orchestration.
• Need to keep everything local? The config includes a commented-out Ollama block. Swap it in to run on a local model.
• Running AURA in-cluster? The Kubernetes SRE quickstart deploys with Helm.
• Ready to allow changes with sign-off? hitl-conversational.toml requires approval before changes like pod deletion run.
• Want the agents to read your runbooks? rag-knowledge-base.toml adds a vector store of runbooks that a specialist can search.
• Just want to look? Read the config below. No install required.

Look inside the config

Routing is plain language in the coordinator's prompt:

[agent]
name = "Kubernetes SRE Agent (Orchestrated)"
system_prompt = """
You are an SRE assistant coordinator. You decompose Kubernetes observability
tasks into specialized sub-tasks and delegate to the appropriate worker.

ROUTING GUIDANCE:
- Workload discovery, health checks, namespace listing, service inspection: delegate to k8s-discovery
- Prometheus queries, target health, metrics, metric metadata: delegate to prometheus-analyst
- Simple factual questions about the cluster: answer directly if possible
"""

Each specialist gets its own model and an explicit tool allowlist

[orchestration.worker.prometheus-analyst]
turn_depth = 8
mcp_filter = [
  "health_check",
  "execute_query",
  "execute_range_query",
  "list_metrics",
  "get_metric_metadata",
  "get_targets",
]
preamble = """
You are a Prometheus Analyst. Always call the appropriate tool.
Do not fabricate metric values.
"""

Want to add a Slack-posting specialist or a runbook reader? Add a worker block, give it its tools, and route to it from the coordinator prompt.

Try it in four steps

1. Install AURA (Linux or macOS):

curl -fsSL https://raw.githubusercontent.com/mezmo/aura/main/scripts/install.sh | bash

2. Start the Kubernetes MCP server in read-only mode (uses your current kubectl context):

npx -y kubernetes-mcp-server@latest --port 8081 --read-only

3. Start the Prometheus MCP server:

export PROMETHEUS_URL="http://localhost:9090"
docker run -i --rm -e PROMETHEUS_URL -e PROMETHEUS_MCP_SERVER_TRANSPORT=http \
  -e PROMETHEUS_MCP_BIND_PORT=8082 -p 8082:8082 ghcr.io/pab1it0/prometheus-mcp-server:latest

4. Set your model key, download the example, and run it:

export OPENAI_API_KEY="sk-..."
curl -fsSLO https://raw.githubusercontent.com/mezmo/aura/main/examples/complete/kubernetes-sre-orchestrated-multi-model.toml
aura --config kubernetes-sre-orchestrated-multi-model.toml

More detail: AURA example configs.

Then ask the boring question: "Check the health of my cluster. Are all pods running?"

FAQ

Can this agent change anything in my cluster?

Not in this example. The Kubernetes MCP server runs with --read-only, and each specialist can only call the read-only tools on its allowlist. When you're ready to allow changes, give a specialist write tools and gate them behind human approval (see hitl-conversational.toml).

Why use multiple agents instead of one?

Scoping. Each specialist sees only the tools for its job, runs on a model you choose for that job, and has its own turn limit. The coordinator never calls worker tools directly, so every action traces back to a specific specialist.

What happens if a model provider fails partway through?

The team reports the analysis as incomplete instead of guessing. You can see that behavior in the video on this page.

Does my cluster data leave my environment?

AURA sends prompts and tool results only to the model providers and services you configure. To keep everything in-house, run AURA in your own infrastructure with a local model. See the security and data-handling model.

What does it cost?

AURA is free under the Apache 2.0 license. You pay your model provider for usage.

Next steps

• Engineers: Clone the example and run the health check against a dev cluster.
• Leads and managers: Share this page with your platform team and pick one cluster to pilot read-only.
• Everyone: Join the AURA Slack community to ask questions and see what other teams are building.