AI SRE Agent Audits Kubernetes Runbook Coverage: AURA

The runbook is three versions out of date

0:00 3 a.m. The pager fires. You open the runbook, and it describes a service that shipped three versions ago. This is 31 Days of AURA, today, mending the runbook gap.

Why runbooks drift faster than clusters

0:12 Runbooks don't drift because anyone's careless. Clusters just change faster than docs. Every deploy, every new service, every renamed alert widens the gap between what's running and what's written down, and you find out at the worst possible moment.

One prompt against both sides of the gap

0:27 AURA can see every side of that gap. Through its tools, it inspects what's actually running. Deployments, services, configs, alerts. It reads your runbook repo like an engineer. And it reads the incident channel, where the real work happens. So instead of hoping the docs are current, you can just ask.

0:47 One prompt. Investigate the apps in the cluster against the runbook repo. It works both sides. Enumerating what's live, reading what's documented.

The finding: 38 workloads, zero runbooks

0:55 And there's the gap. 38 workloads across 16 namespaces, and zero runbooks. A README and an empty template. Coverage, 0%. And the three services failing right now, all OOMKilled, have nothing.

AURA authors three runbooks and opens a PR

1:16 Now ask AURA to create a PR, adding the runbooks it suggested. AURA rechecks live state, authors three complete runbooks against the repo's own template. Cart, frontend, Grafana, and opens the pull request. 700 lines grounded in what it actually observed. It doesn't push to main, it proposes.

The alerting gap the pull request surfaced

1:41 But look what the PR itself flags, an alerting gap. Only three Grafana alert rules exist. None fire on OOM kills, memory versus limit, or restart rate. Every one of these outages was caught by a human, not an alert.

1:56 So, next question. How would we resolve the alerting gap? AURA already knows both sides of this system. The plan comes back precise. Three failure modes, three PromQL rules, all in the alerting module that Terraform already provisions. No new repo, no new exporter. The metrics were always there.

The second PR, gated on every mutating call

2:18 Then, ask AURA to open a new PR and watch what happens on the way. Every mutating call stops for approval. The branch, the file, the pull request itself. A human in the loop before anything changes. Two rules land in Terramate. OOM kills and memory pressure. Pull request proposed, not pushed.

2:41 Two pull requests side by side. One adds the runbooks, three files, 700 lines, every step grounded in observed cluster state. The other adds the alerts that would have caught these incidents in the first place. You review, you merge, like any other change.

Re-running the audit

3:00 Then close the loop. Ask AURA, is the gap covered now? Same investigation, fresh answer. Three merged runbooks, five live alert rules. And one of them is already tracking frontend's memory climb. That's not a doc sprint. That's one conversation end-to-end.

The other direction: incidents teaching runbooks

3:21 Runbooks go stale from the other direction, too. The next incident teaches you something the docs don't know, a command the on-call engineer ran that actually worked. AURA reads the incident channel. It sees what you did, spots what the runbook missed, and proposes the update. Every page makes the next one shorter.

The setup: two MCP servers, two workers

3:40 So, how do you run this yourself? Two MCP servers in your agent, Kubernetes, and GitHub. Then two workers. cluster_inspector takes the read-only K8s tools, github_analyst takes the repo tools, plus the three writes it needs to branch, commit, and open a pull request.

3:59 The gate itself is one list. Create, update, push, merge. Anything that changes state. Want the other two halves? Slack for the incident channel, Grafana for the alerting. That's the whole setup.

4:15 That's how you mend the runbook gap. The cluster keeps changing. And now the docs and the alerts keep up. To learn more, visit docs.mezmo.com/aura. See you tomorrow.