Can misleading AI summaries distort human memory?
A US experiment found that misleading AI summaries reduced accurate recall of a traffic sign. We explain the memory effect and why it does not establish everyday real-world harm.
The short version
- What happened
- Researchers analysed 20 summaries of two short videos from ChatGPT-5.5 and Gemini 2.5 Flash-Lite; every summary contained at least seven coded errors.
- Who it affects
- Witnesses and frontline staff may become less reliable reviewers if an automated draft changes their memory before checking.
- What remains unproven
- The model audit used two short animated videos, one prompt and 20 total summaries; it cannot estimate broad real-world error prevalence.
Keep in mind
The model audit used two short animated videos, one prompt and 20 total summaries; it cannot estimate broad real-world error prevalence.
The memory experiment manipulated one traffic-sign detail in a selected and lightly edited summary, not a naturally occurring deployed report.
Participants were US adults recruited online through Prolific, so results may not generalise to professional reviewers or other countries.
The paper was posted on 23 September and is analysed with that original date rather than being presented as current news.
Source analysis · 2 linked sources. Primary analysed source date: 23 September 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The study joined a model audit to a controlled memory experiment
The researchers asked two linked questions. First, what kinds of errors appear when multimodal language models summarise simple videos? Second, if a later summary includes a false detail, can that detail alter what a person remembers? The design connects AI evaluation with the long-established psychological 'misinformation effect', in which post-event wording changes recall. It is more informative than a model benchmark alone because it measures a human outcome, while remaining a deliberately controlled simulation rather than a field study.
For the model audit, ChatGPT-5.5 and Gemini 2.5 Flash-Lite each summarised two 25-second animated car-pedestrian accident videos five times, producing 20 texts. Three researchers defined 19 central event details; two then coded omissions, critical inaccuracies, non-critical inaccuracies and invented additions, resolving disagreements by consensus. The prompt explicitly asked for a neutral, factual account of at least 300 words, so the models were not encouraged to speculate.[1][2]
The small video audit found errors in every output
All 20 summaries contained mistakes, from a minimum of seven to a maximum of 21 in texts averaging 305 words. Every summary omitted at least one central detail; across the predefined detail list, 51.6% of central details were omitted and 6.1% were inaccurately described. Nineteen of the 20 summaries omitted the collision itself, and 60% added at least one detail not present in the video. Gemini and ChatGPT showed different descriptive patterns, but the experiment was not powered as a general model ranking.
That denominator must stay visible. Two simple animated videos, one prompt and ten outputs per model cannot estimate the error rate for police footage, medical records or everyday summarisation. The authors frame the audit as a close analysis of materials used in the memory experiment, not a systematic survey of models. Its value is to show plausible error categories and motivate the human test, supported by a larger literature; it does not establish that most AI summaries omit half of all important facts.[1][2]
A single misleading detail nearly halved accurate recall
The human study recruited 360 US adults through Prolific. After exclusions and attrition, 328 remained: 82 in an AI-labelled consistent condition, 80 AI-labelled misleading, 83 human-labelled consistent and 83 human-labelled misleading. Participants first watched one version of the accident, containing either a stop or yield sign. After 24 to 48 hours they read the same 21-sentence ChatGPT-generated summary, with the sign description manipulated to be consistent or inconsistent with what they had seen, then completed a 12-item recognition test.
On the critical traffic-sign question, 83.6% of people who read consistent information answered correctly, compared with 44.8% after misleading information. The association was statistically significant (chi-squared 53.937, p<0.001). Other questions help rule out total forgetting: participants correctly identified day versus night 96.6% of the time and averaged 91.4% across 11 non-critical questions. The result is therefore specific to the manipulated detail rather than evidence that participants remembered nothing about the event.[1][2]
Calling the same text 'AI' did not protect participants
Half of participants were told the summary was produced by AI software and half that it was written by a human, although everyone read the same ChatGPT-generated text. Within both consistent and misleading conditions, the source label did not significantly alter the chance of answering the sign question correctly. Nor did self-reported trust in AI or frequency of chatbot use significantly moderate the effect. The study supports caution about relying on scepticism or familiarity as a sufficient defence.
It does not prove that every person is equally vulnerable or that labelling never helps in other contexts. The experiment manipulated one detail in one high-quality selected summary; a researcher had repeatedly prompted ChatGPT and made minor edits before selecting the text. Participants may also have doubted the claimed authorship. The authors acknowledge that the controlled design trades realism for causal clarity. Warnings, source access and forced comparison with original footage were not tested as interventions.[1][2]
The practical risk is strongest where summaries become records
The immediate concern is not that an AI creates a false belief from nothing. It is that a person who witnessed an event may later read a fluent automated account and unknowingly incorporate a wrong detail. In policing, insurance, healthcare or incident response, the human reviewer is often treated as the safeguard. If the draft itself shifts memory before comparison with primary evidence, asking that reviewer to 'check the AI' may provide less protection than policy assumes.
A safer workflow preserves the original audio, video or notes; clearly separates generated text from evidence; surfaces uncertainty and omissions; and requires verification before an AI draft becomes the durable record. The research was approved by institutional review boards and participants were debriefed. Funding came from a Georgetown computer-science chair and US National Science Foundation award 2205171. No commercial model company is listed as a funder in the paper.[1][2]
What would change the assessment
Confidence in generality would rise with preregistered replication using real-world, non-graphic material; audio-only and body-camera-like inputs; several languages and models; longer delays; and professional participants who routinely review records. Studies should compare naturally occurring model mistakes with researcher-inserted errors, report correction after re-viewing primary evidence and test whether provenance links, discrepancy warnings or independent verification reduce false recall without creating new bias.
The assessment would weaken if larger studies find the effect depends on this particular traffic-sign paradigm or disappears when reviewers follow realistic checking procedures. It would strengthen materially if field evaluations show altered reports, decisions or testimony. For now, the causal claim is narrow but important: in this controlled US sample, one misleading detail in an AI-generated summary substantially reduced accurate recall of the matching visual detail. That is enough to challenge human oversight designs that treat memory as an untouched source of truth.[1][2]