One LLM run can make a prompt look better than it is: repeated tests found randomness dominated some results
A peer-reviewed experiment across three causal-map tasks found that model-to-model and run-to-run variation could overwhelm controllable prompt choices. The result is a warning for researchers and teams that benchmark an AI workflow only once.
The short version
- What happened
- The researchers compared three algorithms for combining causal maps, using six model versions from the DeepSeek, Gemini and GPT families and five repeated runs for each experimental condition.
- Who it affects
- Researchers, auditors and workplace teams can make a poor tool look dependable if they report only one run. Repetition makes unstable cases visible before a model shapes a study result or operational decision.
- What remains unproven
- Three specialised causal-map combination algorithms
Keep in mind
Three specialised causal-map combination algorithms
Six model versions from three provider families in the reported factorial comparison
Five completed replications per condition, with some high-variance cells needing more
No human decision, factuality, productivity or safety outcome
Proprietary API behaviour may change after the recorded access window
Source analysis · 1 linked source. Primary analysed source date: 27 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The same controlled setup did not always give the same answer
A single successful LLM test can exaggerate how much a prompt, model setting or workflow contributed to the result. In a peer-reviewed experiment published on 27 October, researchers used LLMs to combine causal maps—directed graphs that represent possible relationships between factors. They varied prompt components and decoding approaches, repeated conditions five times and measured how much of the result could be attributed to choices the researcher controlled rather than residual variation between runs. That matters for consequential decisions in research, public policy and operational planning.
The study covered three combination algorithms and six model versions from the DeepSeek, Gemini and GPT families. Its factorial design tested prompt elements together instead of changing one element at a time. Across the experiment, model family and version mattered, prompt effects depended on the metric, and no decoding strategy was consistently best. For the most sequential algorithm, run-to-run randomness accounted for about 84% to almost 96% of variance for the two DeepSeek versions and about 73% to more than 81% for the two Gemini versions.[1]
Five repetitions were enough to reveal the problem—not always to estimate it precisely
The team ran 4,800 tests for its open-weight-model decoding comparison and thousands more across the wider factorial designs. Yet a confidence-interval check found that a small subset of high-variance conditions would need many more repetitions for a precise estimate. In one Algorithm 3 recall analysis, the median estimated total among varying conditions reached 7,684 runs for DeepSeek Chat v3.1 and 5,283 for GPT-4o. Those large budgets are estimates derived from the observed variance, not a universal minimum for evaluating an LLM.
The practical lesson is to match the replication plan to the decision. A research paper comparing prompts should preserve the exact model version, access dates, decoding settings and normalization choices, then report the spread of results. A workplace team testing whether an assistant is reliable should repeat realistic tasks, retain failures and examine whether a favourable average hides unstable cases. Turning temperature down to zero is not, by itself, proof of reproducibility.[1]
The distinction that matters
What the experiment separated
The factorial design asked which source of variation actually moved each performance measure.
Prompt
Examples, explanations and representations
Effects changed by model and metric
Decoding
Greedy, beam and contrastive strategies
No universally best method
Repeated run
Same condition executed again
Up to almost 96% of variance in one workflow
The finding is important, but its scope is deliberately narrow
The tasks joined causal maps about obesity and suicide prevention; they did not measure writing quality, factuality, coding success or decisions made by people using the output. The three algorithms also differed in how many sequential LLM steps they required, which helps explain why errors could cascade in the most variable workflow. Results from proprietary models may change as providers update systems behind an API.
The authors released their data and scripts, reported no specific funding and declared no competing interests. Stronger follow-up would reproduce the design on other tasks, use independently hosted model snapshots where possible and connect output variability to human decisions or downstream harms. For now, the paper supports a concrete research standard: a benchmark should describe a distribution, not just its most convenient run.[1]