Why did some AI agents lose accuracy after receiving the same tool result?
In a 158-question geochemistry benchmark, changing long simulator reports from inline text to retrievable files reduced accuracy for two tested GPT versions. The preprint traces much of the loss to agents choosing the wrong evidence, not failing to compute the answer.
The short version
- What happened
- Researchers tested six models on 158 execution-verified multiple-choice questions using the deterministic PHREEQC geochemistry simulator and audited 5,773 complete trajectories plus 11,007 simulator calls.
- Who it affects
- Scientists and engineers using agents need logs that show which artefact supported the final answer, not only whether a tool ran successfully.
- What remains unproven
- This is a 19 October 2026 preprint and has not been peer reviewed.
Keep in mind
This is a 19 October 2026 preprint and has not been peer reviewed.
The 158 questions came from one geochemistry simulator benchmark; they do not represent all scientific or workplace tool use.
Only GPT-5.1 retained a statistically significant retrieval penalty after correction across headline model comparisons; several models showed no detectable net effect.
The accessible manuscript did not display a funding or conflict-of-interest statement.
Source analysis · 1 linked source. Primary analysed source date: 19 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The agents often computed the right value and still chose wrongly
A new preprint finds that an AI agent's surrounding software can change accuracy even when the scientific simulator produces the same result. Researchers gave agents access to PHREEQC, a deterministic US Geological Survey geochemistry program, and asked 158 four-option questions whose answers they had re-executed and verified. Long reports were either placed directly in the model's context, capped at 100,000 characters, or stored behind a section index that the agent could inspect with file-reading tools.
For GPT-5.1, indexed retrieval reduced pooled accuracy by 11.3 percentage points compared with inline reports; for GPT-5.2 the reduction was 7.9 points. Claude Opus 4.6, Claude Sonnet 4.6 and GPT-5.4 showed no detectable net change. After correcting tests across the five headline models, the GPT-5.1 difference remained statistically significant, while GPT-5.2 did not. Gemini 3 Flash Preview was described separately because many runs failed to submit an answer within the step budget.[1]
An audit located the failure in evidence use
The team inspected 5,773 complete agent trajectories and re-ran all 11,007 logged simulator calls. The GPT agents generated the keyed value about as often under both delivery methods. Under retrieval, however, they frequently skipped the authoritative report and read a shorter file they had created themselves. That file often contained extra decimal places, and the agents sometimes truncated rather than rounded to the precision used by the answer choices—a pattern the authors call evidence substitution.
A stronger paired test froze GPT-5.2's simulator output at 131 checkpoints and replayed each item three times under both delivery conditions. Indexed retrieval scored 61.8% across 393 continuations, versus 71.2% inline: a 9.4-point difference with a 95% bootstrap interval from 4.6 to 14.5 points lower. Because the computation was held fixed, this experiment more directly ties the gap to how the same evidence reached the agent.[1]
The distinction that matters
Where the answer can fail
The audit separates producing, seeing and using the correct value.
Build
Simulator produces the keyed value
Computation can be correct
Exposure
Harness decides what the agent reads
Authoritative report may be skipped
Closure
Agent selects and formats the answer
Derivative output can displace the source
The practical fix was specific—and the boundary is narrow
Adding an instruction to answer from the official simulator report raised accuracy by 6.3 to 7.9 points for the three GPT versions tested. Merely reminding agents to round correctly did not help. That distinction matters for laboratories and engineering teams: a traceable tool result is not enough if the agent can silently prefer a derivative file, summary or calculation that looks easier to use.
The result does not show that retrieval is generally harmful. It covers one simulator, one multiple-choice benchmark, fixed prompts and a limited set of model versions. Several models had no detectable net penalty, and GPT-5.4's better computations offset an estimated evidence-use loss. The questions also reward exact selection among four options, so they may magnify precision and rounding errors differently from an open scientific report. The manuscript is a preprint, not peer reviewed; it states no visible funding or conflict disclosure. Replication on open-ended scientific tasks, other tools and newer models would determine whether authoritative-source instructions are a robust safeguard or a benchmark-specific patch.[1]