Better memory helped a personal agent find the right rules—and still produced unsupported health assumptions
On 349 synthetic health and daily-living tasks, a memory-linked retrieval system reached 86.25% accuracy versus 79.94% for a leading memory baseline. Its remaining failures show why retrieving the right document is not enough when an agent misstates what it knows about a person.
The short version
- What happened
- ColaBench contains 349 synthetic tasks across 10 fictional profiles and a frozen knowledge base of 13,033 real public health and daily-living documents.
- Who it affects
- A personal agent that remembers sensitive history could save someone from repeating context, but a misclassified medicine, outdated condition or lost uncertainty can make a sourced answer personally wrong.
- What remains unproven
- This is an unreviewed preprint using 10 synthetic profiles; it does not measure outcomes for patients, carers or other real users.
Keep in mind
This is an unreviewed preprint using 10 synthetic profiles; it does not measure outcomes for patients, carers or other real users.
The benchmark construction, answering and evaluation pipelines rely heavily on language models. Accuracy was not independently calibrated by clinicians or a representative human panel.
All methods shared one answering model and a frozen knowledge base. Results may not transfer to other models, live web sources or domains outside health and everyday information.
The 111-task component ablations cover three selected profiles, and exploratory subgroup intervals are wide.
Several authors list company affiliations, including DataArc Tech Ltd.; no separate funding or competing-interest statement is reported.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Read the full report · 5 min The evidence, methods, implications and limitations
The test joined conversation memory to documents the agent had not seen
A personal agent may need two kinds of evidence at once: something a user disclosed weeks earlier and a rule found today. The researchers built 10 fictional profiles whose public conversation histories contain personal conditions, preferences and earlier document access. They then created 349 health and daily-living tasks backed by a frozen collection of 13,033 documents from sources including MedlinePlus, openFDA and the US National Cancer Institute. The personal histories are synthetic; the documents are real public information.
Five systems received the same current request, knowledge index and DeepSeek v4.1 Flash answering model. The difference was how they stored and retrieved history. MemKB first assembled a user-state summary, linked remembered conditions to documents associated with earlier conversations, and fused those candidates with ordinary semantic and keyword search. A GLM-5.3 Flash evaluator checked each final answer against a reference, required conclusions and the evidence actually shown to the system.[1]
Linking memory to knowledge improved the benchmark result
MemKB passed 86.25% of the 349 tasks, compared with 79.94% for Mem0 plus the same hybrid document retrieval. The paired gain was 6.30 percentage points; a profile-cluster bootstrap placed its 95% interval between 3.19 and 9.09 points. MemKB also used 4.07 interaction rounds per task on average, against 4.44 for that baseline. Removing the state summary or the memory-to-document links reduced accuracy in the smaller 111-task ablation set.
The improvement was not simply a matter of reading more reference material. When both systems accessed the reference document, MemKB scored 90.44% and the Mem0 baseline 86.03%; when both actually read it, the figures were 90.74% and 87.96%. The authors interpret the remaining gap as evidence that organising known personal conditions helps the agent apply retrieved rules. Because a model graded the answers, human calibration is still needed before treating those percentages as dependable real-world accuracy.[1]
The distinction that matters
Where a memory-guided answer can succeed—or fail
The benchmark traces both the personal premise and the external rule behind each required conclusion.
Remember
Recover a disclosed condition from earlier conversations
Preserve who said it, when and what remains unknown
Retrieve
Find the document whose rule matches that condition
Expose the evidence needed for the current request
Apply
Compare the rule with the person's known circumstances
Keep unsupported premises conditional instead of turning them into facts
A remembered detail can become a false fact during reasoning
The paper's most useful warning appears in the 48 failed MemKB answers. Seventeen were primarily classed as unsupported personal assumptions and 12 as rule misapplication. In one synthetic case, the record said only that a person used prescription pain medicine; the system converted that into opioid use when applying naloxone guidance. In another, the system read heat-risk guidance but still recommended a foot bath. Better retrieval did not prevent the agent from changing an unknown condition or misusing a visible rule.
For people, that is the difference between an assistant recalling context and a system making a personal judgment. The benchmark does not test live web changes, real users, clinical safety or privacy, and several authors list company affiliations. Independent human grading, prospective tests with consented users and an audit trail that shows which disclosure supports each conclusion would change the assessment. Until then, the result supports traceable memory evaluation—not autonomous health advice.[1]