Can AI agents diagnose real distributed-system failures? A new benchmark finds an evidence gap
Across 120 historical failures, leading agents often named the right file while missing the precise code blocks that caused the incident. The benchmark is an unreviewed preprint and tests diagnosis, not safe autonomous repair.
The short version
- What happened
- DistDebug-Bench contains 120 real bugs from 11 distributed systems, five programming languages and ten failure patterns, with 218 expert-labelled root-cause code blocks.
- Who it affects
- On-call engineers may gain a faster shortlist, but still need to verify the exact causal path before acting.
- What remains unproven
- Unreviewed preprint based on historical open-source incidents rather than prospective live deployments.
Keep in mind
Unreviewed preprint based on historical open-source incidents rather than prospective live deployments.
Some hard subgroups are small, including only seven cross-component opaque incidents.
Reference fixes may not capture every plausible causal intervention point.
The benchmark tests diagnosis, not autonomous repair, recovery time, security or operational harm.
Source analysis · 2 linked sources. Primary analysed source date: 2 November 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The direct answer: agents can narrow the search, but exact diagnosis remains unreliable
A new benchmark suggests that coding agents can often point engineers towards the right part of a large distributed system, but still struggle to identify the precise code that caused a failure. DistDebug-Bench evaluates 120 historical bugs drawn from 11 open-source distributed systems written in five programming languages. The authors labelled 218 root-cause code blocks and asked agents to diagnose each incident from the report, available evidence and the pre-fix repository. That is closer to real incident work than benchmarks that hand a model a neat failing test and a small file.
The distinction between finding a file and explaining a cause is the main result. Leading configurations achieved high file coverage, yet block-level coverage and precision fell substantially. Mini-SWE with GPT-5.5, for example, produced block precision of 34.7% and named an average of 3.6 blocks per diagnosis. An engineer may therefore receive a plausible shortlist containing the real cause alongside several irrelevant locations. That can save search time, but it cannot be treated as a verified root-cause analysis without checking the execution path and incident evidence.[1]
Why these failures are harder than ordinary coding tasks
Distributed-system incidents rarely present one clean error at one obvious location. In this collection, 45% of bugs required changes or reasoning across multiple root-cause blocks, 23% spanned multiple files, 69% of reports did not name the gold method and 63% contained no usable internal stack trace. The benchmark covers ten failure patterns, including timing, concurrency, state-machine and cross-component behaviour. Its input resembles the incomplete evidence an on-call engineer might actually receive: an issue description, logs or traces when available, and a large repository containing many plausible paths.
The hardest cases were those without a useful stack trace or method name. On 42 opaque, no-stack incidents, the best reported block coverage was 65.0%. On seven cross-component opaque bugs it fell to 40.5%. By contrast, local or adjacent bugs whose report named the method reached 95.8%, and no-stack reports that still named the method reached 94.8%. The direction is credible—specific clues help—but the seven-case cross-component result is exploratory. It should not be converted into a general failure rate for every production system.[1]
The distinction that matters
Where diagnosis quality falls away
The benchmark separates broad localisation from the exact code evidence needed for a causal diagnosis.
Method named
Local or adjacent bug
95.8% best block coverage
No stack
Method still named
94.8% best block coverage
Opaque report
No usable stack or method
65.0% across 42 cases
Cross-component
Opaque report
40.5% across seven cases
How the benchmark was labelled and checked
The researchers traced the historical fix for each issue and labelled the source blocks whose behaviour had to change to resolve it. Three annotators independently reviewed a retained set of 51 annotation sheets. Agreement was high at broad levels: Krippendorff's alpha was 0.919 for files and 0.866 for functions. Block coverage agreement was 0.861 and block-level F1 was 0.848. Those figures make the reference labels more credible than a benchmark built solely from automated commit diffs, although they do not make causality perfectly objective.
The benchmark also tries to separate genuine repository reasoning from memorised issue knowledge. Future commits were removed from the repositories. A leakage screen using GPT-5.4 reduced flagged attempts from 102 of 360 to four of 360 after the authors' controls. In a 66-case closed-book probe, revealing the issue identifier did not improve function coverage: it was 17.3% with the identifier and 21.4% without it. Allowing repository access raised coverage to 45.5%. This supports the view that code inspection mattered, while not proving that pretraining exposure was absent.[1][2]
Stability is a practical safety problem
A diagnosis tool should return similar evidence when asked the same question twice. The leading solution was flaky on 17% of incidents and on 26% of opaque, no-stack reports. That instability matters because incident response happens under time pressure. If repeated runs identify different components, an engineer must spend additional time deciding which chain of reasoning to trust. A confident narrative is not a substitute for a reproducible link from observed symptoms to the relevant code and state transition.
For engineering teams, the responsible use is as a search assistant rather than an autonomous incident commander. An agent can collect candidate files, connect log messages to code and propose hypotheses. A human can then compare those hypotheses with telemetry, deployment history and system invariants. The benchmark does not test whether an agent can execute a rollback, edit production code, coordinate teams or recognise when evidence is too weak. It also does not measure the cost of a false diagnosis, which can be much higher than the cost of asking an engineer to inspect an extra file.[1]
What this means for developers and incident teams
The result supports a tiered workflow. Low-risk assistance can begin with repository navigation, retrieval of related incidents and a cited list of candidate blocks. Higher-risk actions should require explicit evidence: the triggering request, relevant logs, the code path, the state that violated an invariant and a test that reproduces the failure. Teams can record whether the agent's first hypothesis was correct, how often it omitted a necessary component and whether it delayed or accelerated recovery. Those measures are more useful than a generic claim that an agent 'solved' an incident.
Tool builders should expose uncertainty and provenance at block level. A diagnosis that says only 'the storage module is responsible' is hard to audit. A better system links each claim to code, trace evidence and alternative explanations. It should also preserve the original evidence so an engineer can reproduce the reasoning. The benchmark's large gap between file discovery and precise block identification shows why broad localisation scores can flatter a product. Buyers evaluating incident-response agents should ask for exact-location precision, repeatability and performance on reports without clean stack traces.[1]
What remains unproven—and what would change the assessment
DistDebug-Bench is an unreviewed arXiv manuscript using curated historical open-source incidents. Historical fixes offer a defensible reference, but a commit may contain several reasonable intervention points, and an alternative repair might prevent the same failure. Some difficult categories contain only a handful of cases. The evaluated systems and model versions will also change. The benchmark does not establish performance on proprietary code, newly emerging failures, security incidents, live operational constraints or repositories whose documentation and test coverage are poor.
Confidence would rise with independent replication, larger cross-component subsets and prospective trials in which teams record diagnoses before the real cause is known. Such trials should measure time to a verified cause, false leads, omitted components, recurrence after repair and harm from incorrect actions. Evaluation should include repeated runs and controlled access to logs, traces and deployment metadata. Until then, the evidence supports a limited claim: agents can help search a complex codebase, but exact, stable and operationally safe root-cause diagnosis remains an open problem.[1][2]