Should an AI agent turn one failure into a rule? A registered experiment found chance can stick
Across 90,084 synthetic answers, a lesson written after a random failure sometimes pushed a fresh model session towards the less reliable tool. A companion preprint found that rewritten instructions also need to match the model expected to use them. Neither study establishes production harm or safe autonomous self-improvement.
The short version
- What happened
- A registered synthetic study tested 256 worlds, 11 model endpoints and 90,084 answers; after a chance failure, an AI-written lesson increased selection of the worse tool from 0.4% to 11.6% in the main comparison.
- Who it affects
- Teams using agents that revise prompts, procedures or memories should not let one failed run silently become policy. For work affecting patients, learners, employees, public services, money or access, retain the event record, test the proposed change and keep approval with an accountable person.
- What remains unproven
- Both studies are preprints and not peer reviewed
Keep in mind
Both studies are preprints and not peer reviewed
Main risk study uses synthetic worlds and a first-tool choice without execution
One stored lesson was tested, not accumulated memory
No independent replication or external human review reported for the luck study
Skill study's main testbed deliberately selected 103 sensitive pairs from 607 tasks
Five-round skill experiments were run once per round
No production incident rate or regional outcome measured
Funding and conflicts not stated in the accessible manuscripts
Multi-source analysis · 3 linked sources. Primary analysed source date: 27 October 2026. Latest linked supporting source: 28 October 2026.
Read the full report · 5 min The evidence, methods, implications and limitations
A random failure became advice for the next decision
A registered preprint has isolated a plausible failure mode in agents that write lessons from experience. The researchers created 256 synthetic worlds, each with two interchangeable tools whose reliability was known. A scripted earlier run used one tool and either succeeded or failed by chance. A fresh session of the same model then saw the record plus one stored memory item and chose which tool it would use first next time. Across 11 model endpoints, the study collected 90,084 answers.
When the better tool had failed by chance, a lesson written from that run made readers choose the worse tool in 11.6% of next decisions. The rate was 0.4% with irrelevant memory and 0.4% when the lesson followed success. The estimated increase against irrelevant memory was 11.2 percentage points, with a 95% interval from 9.7 to 12.6 points. That is evidence that a model-written summary can outweigh the underlying record in this artificial setting—not a measured failure rate for deployed agents.[1]
Guardrails helped, but the effect varied by model
The authors tried two remedies. Asking the lesson writer to focus on decision quality rather than the outcome reduced worse-tool choices to 7.5%; an outcome-neutral instruction reduced them to 3.1%. An explicit statement that each run was statistically independent also weakened the effect descriptively. None of these controls eliminated it across every endpoint, which matters because a single aggregate safeguard can conceal different model behaviour.
The experiment was deliberately narrow. The reader named a first tool but did not execute it; it received one lesson, not a growing memory; and the worlds were synthetic. It had an authoritative reliability record, while real systems may have stale or incomplete evidence. The paper was not independently replicated, reports no external human review and does not separate the influence of lesson writer from lesson reader. Funding and conflicts were not stated in the accessible manuscript.[1]
The distinction that matters
How a one-off outcome can become a standing instruction
The risk appears when an observation, an explanation and an operating rule are treated as if they carried the same evidence.
Observe
One run succeeds or fails, sometimes by chance
Retain the raw event and uncertainty
Write
The model turns that outcome into a lesson
Test whether the claimed cause is supported
Reuse
A fresh session reads the stored lesson
Approve scope, monitor results and preserve rollback
Better instructions are possible, but they are model-specific changes
A companion preprint tested a different side of the same operational problem: adapting written skills without changing model weights. Seven models faced three benchmarks containing 607 tasks. Researchers selected 103 task–instruction pairs where model capability appeared especially important. On that selected set, a fixed meta-skill plus a stronger rewriting model improved weaker models by 5.1 to 14.1 points; five rounds of self-evolution ended 2.0 to 5.1 points above a comparison loop without the extra guidance.
Those figures are not a universal gain. The subset was chosen for skill sensitivity, each round ran once, retrieval was out of scope and the authors say not to overinterpret small exact gaps. Together, the papers support a cautious workflow: keep the raw event separate from the proposed lesson, test old and new instructions on comparable cases, scope the change to a named model and task, and keep consequential changes reversible and reviewable. The assessment would strengthen with independent replication, real executed workflows, accumulated-memory tests, preregistered production trials and evidence that safeguards transfer across models.[2][3]