Why do multi-agent AI systems keep failing?
An unreviewed analysis of 22,848 closed issues across 21 prominent open-source projects identified 944 genuine multi-agent problems. Handoffs, execution control and memory dominated—but repository reports cannot establish failure rates in production systems.
The short version
- What happened
- The researchers collected 22,848 closed issues from 21 projects, used a local DeepSeek-R1 filter to flag 2,335 candidates, then manually retained 944 issues genuinely involving multi-agent behaviour.
- Who it affects
- Users can receive incomplete, repeated or inconsistent results when agent handoffs and shared state fail even if each model appears capable in isolation.
- What remains unproven
- This is an unreviewed preprint submitted to a journal, not a peer-reviewed version of record.
Keep in mind
This is an unreviewed preprint submitted to a journal, not a peer-reviewed version of record.
Closed GitHub issues are selected reports with no denominator of executions, users or installations, so category shares are not failure rates.
The 21-project sample excludes smaller, private and less-active systems, and all issues were collected before 16 November 2025.
Model-assisted filtering and researcher coding can introduce omissions and subjective classifications despite validation and consensus checks.
Source analysis · 3 linked sources. Primary analysed source date: 1 October 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The question behind the issue tracker
Multi-agent AI products divide work among specialised model-driven components: one agent may plan, another call a tool, another retrieve information and another review the result. The arrangement promises broader capability, but it also creates more handoffs, shared state and stopping conditions. The researchers asked a practical software-engineering question rather than a benchmark question: what problems do developers actually report when building and using open-source LLM multi-agent systems, what appears to cause them, and what fixes are documented?
The evidence is a new, unreviewed preprint from researchers at Wuhan University, Central China Normal University and the University of Oulu. It mines repository discussions rather than running controlled model trials. That makes the study useful for finding recurring engineering patterns in real projects, but unsuitable for saying that a given framework fails a particular percentage of the time. A GitHub issue is a selected report, not a denominator of all successful and unsuccessful executions.[1][2][3]
How 22,848 issues became 944 cases
Projects had to describe multi-agent LLM functionality as a core component, have at least 3,000 stars, ten contributors and 100 closed issues, and show an update within the preceding year. The final 21 included AutoGen, CrewAI, MetaGPT, AgentScope, CAMEL, OpenAI Agents Python, Smolagents, Flowise, Ragflow and other frameworks, toolkits and workflow products. The team collected every closed issue available before 16 November 2025, producing 22,848 records. The popularity threshold intentionally focuses the sample on established public projects and excludes smaller or private systems.
A locally deployed DeepSeek-R1 model first classified the issue text and reduced the pool to 2,335 multi-agent candidates. The first author then manually reviewed those candidates; disputes were discussed with another author. The final dataset contained 944 issues judged to describe genuine multi-agent problems. In a 500-issue validation sample, two coders agreed strongly, with Cohen's kappa of 0.82; 102 of those 500 cases were retained as real multi-agent issues. The automated stage therefore helped triage the workload, but human interpretation still shaped the dataset.[2][3]
Handoffs, control flow and memory led the taxonomy
The authors used open coding and constant comparison to group the 944 cases into 17 issue categories. Orchestration and execution was largest at 208 cases, or 22.0%. It included failed handoffs, wrong execution order, loops that never ended, premature termination, resource contention and concurrency faults. Memory and state management was next at 14.7%, followed by feature requests at 13.2%, implementation issues at 10.6%, tool issues at 9.2% and integration issues at 6.7%. Security-labelled issues were only 0.3% of the curated set.
Those proportions describe what survived the study's selection pipeline. They do not show that security matters less than reliability, because maintainers may discuss vulnerabilities privately, users may not recognise them, and closed public issues favour problems that can be reproduced and resolved. Nor does the taxonomy separate failures caused by the foundation model from defects in schemas, libraries, prompts, state stores or orchestration code. Its value is architectural: extra agents add coordination surfaces where ordinary software defects and probabilistic model behaviour meet.[2]
What the reported causes and fixes show
The paper identified 541 causes across the issue discussions. Workflow problems accounted for 159, or 29.4%, followed by tool-integration problems and memory problems. Among orchestration and execution cases with an identified cause, 57.0% were attributed to workflow problems; among memory and state-management cases with an identified cause, 51.2% were attributed to memory problems. The examples include stale context being reused, handoff messages arriving too late, schemas disagreeing about whether an agent should send text or structured data, and tools not being registered where the next agent can reach them.
Solutions were visible for only 427 issues, 45.2% of the 944-case set. Workflow optimisation led with 111 solutions, or 26.0%, followed by configuration changes at 15.0%, tool-integration changes at 11.9%, infrastructure changes at 11.0% and improved error handling at 10.5%. These are practitioner-documented remedies, not experimentally compared treatments. A closed ticket can reflect a workaround, changed expectation or version update rather than a durable causal fix, and the researchers did not re-run every solution under a shared test harness.[2]
What teams can act on now
For software teams, the immediate implication is to treat agent orchestration as a distributed system, not as a longer prompt. Handoffs need explicit schemas, ownership and acknowledgement. Loops need budgets and deterministic termination conditions. Shared state needs versioning, isolation and observable updates. Tool calls need validation, timeouts, retries and clear permission boundaries. A workflow should preserve enough trace information to show which agent saw which state and why an execution stopped, without logging sensitive user content indiscriminately.
The issue set also argues against assuming that adding another agent automatically improves quality. Every added component can create a new interface, context copy and failure path. Teams should compare the multi-agent design against a simpler baseline and measure task completion, cost, latency, recoverability and human review burden. For consequential uses, a useful test is not merely whether agents produce an answer, but whether the system fails closed, exposes uncertainty and allows a person to reconstruct the chain without trusting the agents' own narrative of what happened.[2][3]
Limits, funding and what would change the assessment
The study includes only popular open-source GitHub projects and only closed issues created before 16 November 2025. Private deployments, unresolved tickets, support channels and smaller repositories are absent. The first author performed much of the coding and extraction, so consensus checks cannot remove all subjectivity. The model-assisted filter could miss quietly worded cases, while popularity and activity thresholds may overweight mature frameworks with active reporting cultures. The findings are descriptive and cannot compare vendors, predict an individual deployment or measure user harm.
The authors report partial support from China's National Natural Science Foundation under grants 92582203 and 62402348. The manuscript does not present a competing-interests statement. Confidence would rise with peer review, an independently reproduced coding sample, sensitivity analysis of the automated filter, exposure-based denominators such as executions or active installations, and controlled tests that compare proposed fixes. Longitudinal work should also ask whether orchestration, memory and tool failures decline across releases—or simply move into private operational logs.[1][2][3]