How to verify an AI SRE agent’s root-cause diagnosis

TL;DR

• Treat an AI agent’s root-cause diagnosis as a hypothesis. State what broke, when symptoms began, and how the proposed cause could produce them.
• Open the original records behind each claim. Check timestamps, affected services, and request traces rather than relying on the agent’s confidence statement.
• Test a competing explanation against the same incident evidence. A rollout that coincides with timeouts does not establish that the rollout caused them.
• Run a safe check that could distinguish the proposed cause from its strongest alternative before accepting the diagnosis. The hypothetical checkout API investigation below tests a timeout-setting change against a dependency slowdown.

Why an agent's confidence is not evidence

An agent’s confidence score cannot tell you what broke. If the agent blames a configuration change for API timeouts, you need evidence that the change reached affected requests and caused the step where they stalled. A rollout just before the first timeout establishes sequence, not causation. A dependency could have slowed at the same time.

Google SRE’s troubleshooting guidance treats a proposed cause as a hypothesis to test against observations and competing explanations. For example, a traffic surge could trigger both an autoscaling event and API timeouts. Blaming autoscaling because it appears beside the errors would mistake two effects of the surge for cause and effect. You would need to check whether the timed-out requests passed through the suspected component and whether requests outside its scope also failed.

An inspectable investigation trail lets you open the agent’s tool calls and original records, including query scope, timestamps, and findings that might contradict the diagnosis. A fluent summary, confidence score, or account of private model reasoning does not give you those checks. Even a visible trail can omit a dependency or misread a trace. Treat the diagnosis as a claim you can revise when the underlying records support another explanation.

The six-step verification method

1. State the diagnosis as a causal chain. Write down the affected service, the suspected trigger, the intermediate failure, and the observed impact. Include the proposed start time and a prediction you could disprove. For example, a configuration change blamed for timeouts should have affected requests that used the changed configuration.
2. Trace each claim to its source. Open the deployment record, request trace, or metric query behind the agent’s summary. Record the source, timestamp and time zone, service, and region. A record may establish that an event happened without establishing that it caused the failure. Treat an agent’s confidence statement as a claim, not a substitute for those records.
3. Check chronology and scope. Compare the suspected trigger with the first affected request, allowing for clock differences. Check whether failing requests reached the proposed dependency and whether unaffected versions or regions provide a comparison. If the agent cannot show that connection, mark it as missing evidence.
4. Test competing explanations. Name another cause that fits the symptoms and write down what you would expect to observe if it were true. A service-wide traffic spike might affect old and new versions alike, while a faulty rollout might affect only the new version. Google’s troubleshooting guidance recommends comparing such predictions with observations rather than treating events that occur together as proof of cause.
5. Run a safe discriminating check. Prefer a read-only query or an isolated test that produces different expected results for the competing explanations. Record a negative result too. If a check requires a production change, assess its risk and get authorization before proceeding.
6. Record a bounded conclusion. Use “supported leading hypothesis” when evidence favors one cause but a decisive check remains. Use “confirmed within scope” when a check distinguishes it from the alternatives you tested. Use “unresolved” when the available evidence cannot separate them, and name the next check. These labels describe what you verified in one incident, not a score for the agent.

Worked example: verifying a hypothetical timeout diagnosis

The following incident is hypothetical. It is an invented example for practicing diagnosis verification, not a reported customer outcome.

The agent’s hypothetical report says, “At 14:07 UTC, checkout-api in us-east-1 began returning 504 responses after a 14:05 UTC rollout shortened its payment-service timeout to 500 milliseconds. Payment calls then expired before they could complete.”

You can test each link in that claim. An illustrative deployment record would establish when the configuration changed. Request traces would show where affected requests spent time, while version-specific traces would help show whether failures followed the changed setting. The rollout’s timing alone would not establish the cause. A payment-service slowdown or a traffic spike could also explain the timeouts, so the investigation must check those explanations before accepting the agent’s diagnosis.

The agent's claim and the first-pass evidence

The claimed causal chain is that the 14:05 UTC rollout set a 500-millisecond payment-service timeout, payment calls expired, and checkout requests returned 504 responses. Open the original records linked in the agent’s report before accepting that sequence. The illustrative deployment record deploy-042 places a rollout in the affected region at 14:05, but it does not show which configuration value changed.

The illustrative request trace trace-17 shows a checkout request waiting on the payment service before returning a 504. Open the trace to check which service ended the call, how long the payment span lasted, and whether the request ran on the new configuration. A payment span ending at roughly 500 milliseconds would fit the proposed timeout mechanism, but the trace alone cannot show what timeout was configured or whether the service ended the call for another reason.

The illustrative metric query checkout_504_rate shows 504 responses rising in the affected region at 14:07. Check its time zone, aggregation window, and version filters before comparing it with the rollout record. The query establishes a symptom after the recorded deployment time, not a causal link. Google’s troubleshooting guidance warns that events occurring together can have another explanation. At this point, the missing configuration diff and incomplete version coverage leave the agent’s proposed cause unverified.

Testing the competing explanations

The rollout is one possible cause, but the investigation must also test traffic and payment-service latency. An upstream traffic spike could have overloaded the API. A downstream dependency could have slowed requests regardless of the configuration. Google’s troubleshooting method calls for comparing each explanation’s predictions with observations that could disprove it.

If a service-wide traffic increase overloaded checkout-api, its request-count query for the affected region should show a rise before the first failures. In this illustrative investigation, the query shows request volume holding steady across that window. The traffic-spike explanation loses support for this incident scope, though a missing route-level breakdown leaves room for a spike concentrated on one endpoint.

If the downstream dependency caused the timeouts, request traces should show latency accumulating at its call across both old and new API instances. The illustrative traces instead show 504s concentrated on new instances. Payment calls on old instances in the same region and time window remain within their earlier latency range. Compare traces from the same region and incident window so a healthy instance elsewhere does not provide a misleading contrast.

Those observations favor the agent’s configuration hypothesis, but they do not establish the causal step. The investigation still needs the configuration diff and a check of how the changed setting affects timed-out requests. Until then, the diagnosis remains a supported hypothesis rather than a confirmed cause.

The discriminating check and the bounded conclusion

For the hypothetical API timeout incident, use a read-only query to compare request traces from the old and new configuration versions during the same traffic window. Group the traces by version and compare payment-call durations and termination reasons for requests on each version. Keep the deployment record, trace IDs, and metric queries with the result so another engineer can inspect the observations.

Suppose the traces show that timeouts cluster on the new version while requests on the old version remain healthy. That observation supports the agent’s diagnosis, but it does not show which configuration setting caused the wait. A rollback that reduces timeouts would add evidence, not prove the cause, if traffic or dependency health changed at the same time. Google’s troubleshooting guidance recommends comparing a hypothesis against observations that could disprove it and cautions that active tests can have side effects.

Record the conclusion as supported leading hypothesis. The missing evidence is a link between the changed setting and the delay visible in the traces. Next, inspect the configuration diff to confirm the timeout value, then test the old and new values against comparable payment calls in an isolated environment. Check whether the new value produces the observed call termination before attributing the 504s to that setting.

What to do when the evidence trail falls short

An incomplete evidence trail does not establish that the agent’s diagnosis is wrong. If you cannot inspect the record behind a causal claim, mark the conclusion unresolved. Name the missing observation and the check that could change your assessment.

If the agent provides tool calls and query results, use their timestamps and scope to find the original records. When it cannot show those sources, rerun the key read-only queries yourself and leave the proposed cause unverified.

Missing instrumentation calls for a narrower conclusion. Use an independent signal, such as deployment history or a dependency metric, and identify the causal step you still cannot observe. If clocks disagree, check their offsets before relying on event order. If reproducing the fault could disrupt production, choose an isolated test or a read-only comparison. Google’s troubleshooting guidance recommends recording unknowns and accounting for the risks of active tests.

AURA is an SRE Agent for Production. Mezmo built this standalone open-source SRE execution harness to help SREs manage production environments. Before relying on an AURA investigation, check the AURA documentation and the records available in your setup to see which tool calls and source observations you can inspect. An execution trace can show what the agent did, but it cannot establish that its diagnosis is correct.

Where this fits in your evaluation process

The RCA tools guide covers product options, and the evaluation checklist helps you test whether an option exposes the records behind its diagnoses. This guide addresses the narrower task of checking one diagnosis against incident records and competing explanations.

Conclusion

The useful outcome is a bounded decision, not a declaration that the agent is accurate. Record the observations that support the proposed cause, the alternatives you tested, and the evidence still missing. If the causes remain indistinguishable, mark the diagnosis unresolved and name the next safe check.

FAQ

How long should verification take during an active incident? Check the evidence needed for the next decision, rather than waiting to prove every causal step. Open the agent’s source records, confirm the timing and affected service, and test the strongest competing explanation if you can do so safely. You can mitigate a live failure while the diagnosis remains unresolved.

What if the agent won’t show its sources? Treat its diagnosis as an unverified hypothesis. Check the relevant records directly, or ask the agent for the tool calls, queries, and timestamps behind each claim. A confidence statement cannot replace those observations.

Does a rollback that fixes symptoms prove the diagnosis? No. The rollback supports the diagnosis, but traffic or a dependency may have changed at the same time. Compare what happened before and after the rollback, and record any competing explanation you cannot rule out.

How does verification differ from a postmortem review? During an incident, you check whether one proposed cause has enough support to guide a decision. A postmortem reviews the wider incident record, contributing factors, and follow-up actions after the immediate response.