Does more memory make AI agents better? Anthropic's tests found useful gains—and costly overgrowth
Persistent notes improved one 30-question benchmark, while a wiki built over 128 sessions helped with some multi-document questions. But stored knowledge could become stale, overwrite itself or grow larger than the source material, making selective memory and verification as important as recall.
The short version
- What happened
- On a 30-question CL-Bench database sequence, persistent memory raised reported mean reward from 0.655 to 0.838 for Opus 5.5 and from 0.644 to 0.833 for Sonnet 5.5.
- Who it affects
- Workers using long-running agents need to know which memories are raw records, model summaries or verified facts before relying on them.
- What remains unproven
- Anthropic selected the evaluations and tested its own models; the report is not independently peer reviewed.
Keep in mind
Anthropic selected the evaluations and tested its own models; the report is not independently peer reviewed.
Several examples use proprietary corpora or single game runs that cannot be independently reproduced from the public page.
The report does not consistently provide repeated-run uncertainty or matched-compute comparisons.
Some tasks benefited from generated wikis while others did not, and one wiki exceeded the source corpus in size.
Task scores do not measure long-term privacy, deletion compliance, hallucinated memory or production failure rates.
Multi-source analysis · 2 linked sources. Primary analysed source date: 6 November 2026. Earliest linked supporting source: 4 May 2026.
Read the full report · 9 min The evidence, methods, implications and limitations
The answer: memory can help, but accumulation is not the same as learning
Anthropic's engineering report describes agent memory as information a model gains from experience and can read in later sessions. On one Continual Learning Bench database task, every question began in a fresh session while a directory of saved files persisted. Opus 5.5's reported mean reward rose from 0.655 without memory to 0.838 with it; Sonnet 5.5 rose from 0.644 to 0.833. Opus 5 improved from 0.564 to 0.828. The chart covers a 30-question sequence, showing that stronger models could turn graded feedback into verified rules used later.
Those gains do not mean an agent improves whenever storage is added. A memory can be wrong when written, become stale or distract the model from its task. Anthropic's other examples show generated knowledge maps that did not beat direct search and a wiki larger than the documents it was meant to summarize. The useful capability is selective, testable retention: deciding what may matter later, linking it to evidence, correcting it when conditions change and retrieving it at the right time.[1][2]
What CL-Bench compares
Continual Learning Bench is designed around sequences rather than independent questions. Earlier instances expose facts, schemas or strategies that can help later instances. A stateful system is compared with the same system operating without accumulated state, and the benchmark's gain metric subtracts that stateless baseline. The database task asks an agent to answer natural-language questions about an unfamiliar SQLite database whose schema changes across instances. Feedback after a question can guide later exploration if the system records and reuses it.
Anthropic used Claude Managed Agents and a persistent memory directory, but the engineering post does not report repeated-run confidence intervals for its selected chart. The public CL-Bench leaderboard averages most submitted systems over five runs, yet Anthropic's configuration and result are presented in the company post rather than as an independently validated benchmark submission. The comparison is therefore informative about one harness and sequence, not a universal estimate of memory benefit across agent tasks.[1][2]
The distinction that matters
Four layers of agent memory
Anthropic separates durable evidence from summaries and from the small amount loaded into each new session.
Transcripts
Preserve searchable event history
Rich evidence, high retrieval cost
Notes
Write selected observations and rules
Fast reuse, possible stale errors
Representations
Build wikis and linked maps offline
Synthesis can help multi-document questions
Injected context
Load a small index every session
Immediate guidance without the whole archive
Stronger models learned what was worth saving
Older Sonnet 4.6 often wrote untested theories and failed to read its own notes. Its score moved only from 0.330 to 0.364 with a general memory directory. When prompted to read memory before answering and explicitly test its theories, the score rose to 0.641. More capable models did this without detailed rules: Sonnet 5 read memory before answering nine times out of ten, while Sonnet 4.6 did so less than half the time. Telling stronger models to record only failures reduced much of their memory gain rather than improving it.
That result suggests memory policy interacts with model capability. A rigid rule may filter out successful strategies, stable facts or useful plans, while an unconstrained weaker model may save everything. The post does not isolate whether improvements came from better salience judgement, stronger base reasoning or other model-generation changes. For product teams, the lesson is to evaluate the whole agent-and-memory system. A storage feature that helps one model can burden another, and a prompt that repairs an older model may constrain a newer one.[1]
One game example shows the cost of saving everything
Anthropic compares one Pokémon Red run from Sonnet 3.7 with one from Fable 5 at roughly 10,000 steps. Sonnet had written 413 files across 225 folders, totalling 977,930 characters, and earned one badge. Fable kept ten files in two folders plus a top-level dashboard, totalling 43,406 characters, and earned four badges. Sonnet stored volatile screen state and repeatedly reorganized master folders; during one 17-step stretch it created six files, deleted nine and took no action in the game.
Fable stored plans, tested rules and information not continuously visible on screen. That is a vivid example of selective memory, not a controlled model comparison. There was one run for each model, the models differ in many capabilities and the report does not hold their prompts, actions or randomness constant. It cannot establish that a smaller memory folder caused better game performance. It does show a real engineering failure mode: retrieval and maintenance work can consume the agent's budget until remembering replaces doing.[1]
Offline consolidation built a wiki over 420 million tokens
Anthropic calls its offline consolidation process 'dreaming': agents revisit transcripts or files between live tasks to correct notes, build representations such as wikis and promote selected information into future context. In one internal experiment, about 350,000 files—roughly 420 million tokens—from the acquired software company Stainless were used to build a wiki. Across 128 sessions, Opus 5 agents explored documents, wrote and corrected pages, merged duplicates and checked claims. The evaluation asked 25 questions about the corpus.
Wiki-only performance rose from 13% of graded criteria after early work to 48% as the wiki expanded. When the model used both search and the wiki, the score rose from 54% to 61%, compared with 53% for search alone, while reportedly saving about 20,000 tokens per question. That is a promising selected result on one proprietary corpus and 25 questions. The source does not provide a cost analysis for the 128 consolidation sessions or show that the saved query tokens outweighed the compute used to build and maintain the wiki.[1]
Memory maps helped most when questions required synthesis
On a separate set of 24 merger questions, more than half required information from several documents and many required judgement or arithmetic. Claude scored 87% when consulting a generated wiki, compared with 73% using search alone. On another legal-brief task, where most of 57 criteria could be found by searching one document, the wiki showed no clear benefit. In another case, a 203-kilobyte wiki grew to 1.77 times the 115-kilobyte source corpus, recreating detail rather than compressing it.
Instructions also changed what survived. Several agents initially wrote to one shared contract table and overwrote each other, leaving only 41 of 57 graded facts available. After an iterative prompt change assigned one page per contract and required exact terms, all 57 facts were represented. That is a targeted repair found with access to the benchmark answer criteria. Production systems may not know in advance which facts matter, and exact quotation can preserve source wording while also increasing storage and privacy obligations.[1]
Persistent mistakes create persistent risk
A wrong note can be more damaging than a transient answer because it may be loaded into every future session. Anthropic describes a Pokémon agent that incorrectly recorded a landmark on the wrong side of a mountain, repeatedly walked into the same trap and then rationalized the failure. An offline pass checked the note against the transcript and corrected it, changing the later route. The example shows why a memory needs provenance, freshness and a way to be challenged rather than being treated as trusted ground truth.
Organizations should separate raw event logs, model-written notes and verified knowledge. Sensitive memories need retention limits and access controls. Important claims should link back to evidence, and changes should preserve an audit trail. Retrieval quality, maintenance cost, privacy exposure and downstream error should be measured alongside task score. Loading more context can also reduce instruction adherence, so Anthropic keeps the automatically injected layer small and uses it as an index to deeper material rather than copying the entire memory store into every prompt.[1]
What the evidence supports—and what would change it
The report is first-party Anthropic engineering evidence, not a peer-reviewed study. It combines public tasks, proprietary corpora, selected model generations and illustrative single runs. Anthropic develops and sells the models and memory products being discussed. The post is unusually candid about failures, but the company still selected the evaluations, scoring and examples. Independent researchers cannot reproduce the proprietary Stainless corpus experiment from the public information alone.
Stronger evidence would compare memory designs across providers on preregistered public tasks, repeat runs, report confidence intervals and account for total compute, latency and storage. Long-term tests should introduce deliberate misinformation, changing facts and privacy deletion requests, then measure whether agents correct or forget appropriately. Evidence would weaken if gains vanish after matching compute or if stale notes cause more harm than retrieval prevents. The present result supports careful memory engineering—not the claim that an ever-growing archive makes an agent continuously intelligent.[1][2]