Agentic AI's average business gain did not predict what a new deployment would deliver
A peer-reviewed synthesis of 19 retail and supply-chain studies found a large positive average effect, but near-total variation between studies meant a new setting could plausibly see harm as well as benefit.
The short version
- What happened
- The peer-reviewed meta-analysis combined 19 studies with 5,292 observations across retail, supply chain, e-commerce and inventory contexts.
- Who it affects
- Workers and customers bear the consequences when a promising average is treated as a guaranteed local result. Pilots should measure correction work, errors and service outcomes alongside speed or cost.
- What remains unproven
- The 19 studies differed profoundly, producing 99.5% heterogeneity and a prediction interval spanning negative and positive effects.
Keep in mind
The 19 studies differed profoundly, producing 99.5% heterogeneity and a prediction interval spanning negative and positive effects.
The evidence base included journal articles, conference papers and preprints, and mixed simulations with real-world studies.
Operational domains did not explain differences reliably under the random-effects model; the e-commerce subgroup contained only three studies.
A selective-reporting adjustment reduced the pooled effect and removed conventional statistical significance.
Source analysis · 1 linked source. Primary analysed source date: 24 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The average was positive; the variation was the decision-relevant result
Researchers at Tshwane University of Technology combined 19 studies containing 5,292 observations across retail, supply-chain, e-commerce and inventory work. They converted reported outcomes to Hedges' g, a standardised effect size, and used a random-effects model with restricted maximum-likelihood estimation and a Hartung-Knapp adjustment for confidence intervals. The pooled result was large and positive: 1.58, with a 95% confidence interval from 0.26 to 2.89 and p=.022. Because g expresses differences in standard-deviation units, it does not translate directly into pounds saved, orders fulfilled or hours returned to workers. Those outcomes still need to be measured in each organisation.
That average is not a forecast for a new installation. Differences between studies were near total: I² was 99.5%, with tau-squared of 7.18. The review's 95% prediction interval—the range intended to describe a comparable future setting—ran from -4.20 to 7.36. In plain language, the available literature is consistent with substantial harm in one context and exceptional benefit in another. Architecture, task, data, implementation and whether a study was simulated or field-based may all matter more than the pooled headline.[1]
Domain labels did not explain the uncertainty reliably
The authors tested whether retail, supply chain, e-commerce and inventory studies produced different effects. A common-effect calculation reported large differences, but the more appropriate random-effects comparison did not reach conventional statistical significance (Q=6.73, three degrees of freedom, p=.081). E-commerce contained only three studies, so its estimate was treated as descriptive rather than conclusive.
The search was conducted from March to June 2026 and covered Scopus, IEEE Xplore, Web of Science and ACM, plus Google Scholar, arXiv, SSRN, selected research repositories, four hand-searched journals and citation chains. Google Scholar screening was limited to the first 200 results for each query. The review allowed journal articles, conference papers and preprints from 2021 to 2026. That breadth helps find emerging work, but it also combines evidence with different review status and operational realism. The paper's study-level assessment explicitly distinguishes simulations from real-world evidence.[1]
The distinction that matters
Why the pooled gain is not a deployment forecast
Three statistics answer different questions, and the widest one matters most for a new setting.
Pooled average
Combine 19 reported effects
Large positive mean: g=1.58
Between-study variation
Test whether studies agree
Near-total heterogeneity: I²=99.5%
New setting
Estimate a future true effect
Prediction interval spans harm to benefit
A local pilot needs a failure measure, not only an upside target
For retailers, logistics teams and workers, the sensible implication is not that agentic AI usually delivers a very large gain. It is that published results do not yet support a portable expectation. A pilot should predefine the relevant denominator and comparator: orders, stock-outs, delivery time, labour hours, customer resolutions or safety failures under the existing process versus the proposed system. It should also record human intervention and downstream errors, not only tasks the agent completed.
Selective reporting is another warning. A limit meta-analysis reduced the estimated effect to 1.26 with p=.066, no longer conventionally significant. The review was funded by Tshwane University of Technology and the authors declared no conflict. Confidence would rise with preregistered, real-world field experiments, consistent outcomes, architecture-specific comparisons and publication of null or harmful results. Until then, the strongest evidence-backed claim is variability—not a general business case.[1]