Can an AI agent safely summarise years of myeloma care?

The answer: the agent can organise long records, but its error severity still demands expert control

The study provides one of the more realistic tests of language-model assistance with longitudinal cancer records. Rather than asking isolated textbook questions, the researchers assembled questions about staging, first occurrences, treatment intervals, eligibility and choices across years of fragmented myeloma care. The full skill-guided agent agreed with adjudicated expert answers on 80.1% of the primary test pairs. That exceeded simple and iterative retrieval approaches, and the advantage grew on the longest records.

The safety result is more important than the headline concordance. Researchers classified 57.8% of the system's errors as clinically significant, compared with 18.8% of disagreements among experts. The overall system error rate, 12.2%, was numerically similar to the 13.6% expert-disagreement rate, but those rates are not interchangeable when the consequences differ. An assistant that is wrong less often can still be less safe if its mistakes are more likely to alter staging, eligibility or treatment interpretation.[1]

What records and questions were evaluated

The source cohort covered 811 patients treated over 25 years, from 2001 to 2026, with 44,962 documents and 1,334,677 laboratory values. The primary evaluation then sampled 100 patients from the Technical University of Munich, yielding 469 patient-question pairs across 48 templates. Those patients had a median age of 65.1 years; 63% were men. A typical record contained 50.5 documents, 1,414 laboratory values and 5.5 years of history, while the largest records ran to hundreds of thousands of characters.

Four oncologists independently created reference answers in a double-annotation process, with disputed cases adjudicated by a senior haematologist. The question mix was uneven: 326 pairs were single-choice tasks, 70 concerned treatment intervals, 45 staging, 20 first occurrences and only eight eligibility. That last denominator is too small for a stable claim about eligibility decisions. An exploratory test on 20 MIMIC-IV patients broadened the setting, but it is not a large external validation.[1]

How the agent and its clinical skills were compared

The authors compared five approaches: single-pass retrieval-augmented generation, iterative retrieval, skill-augmented iterative retrieval, feeding the available record as full context, and a skill-guided agent. The clinical skill library contained 42 modules encoding task instructions and domain procedures. The agent could plan, choose tools, retrieve evidence and revise its path, while the skill-augmented retrieval system used selected modules without the same multi-step autonomy. The primary open model, gpt-oss-120b, ran locally through vLLM, and the researchers repeated the evaluation ten times.

Iterative retrieval reached 75.7% concordance and the full-context approach 75.8%. Adding selected skills raised the result to 79.0%, while the full agent reached 80.1%. Removing the skill library dropped the agent to 76.6%. That ablation is revealing: much of the value comes from explicit, curated clinical procedures rather than agency alone. On the most complex questions, the agent was 5.4 points above skill-augmented retrieval, but that numerical difference was not statistically established.

The agent also cost more time. Its median latency was 20.9 seconds on one H200 accelerator, compared with 6.6 seconds for iterative retrieval, 5.0 seconds for full context and 2.3 seconds for simple retrieval. A hospital would need to judge whether the added delay and engineering complexity buy enough improvement over the 79.0% skill-retrieval result, especially for shorter records where their performance was similar.[1]

Long records are where the result is most useful

The advantage was clearest when no single retrieval step could easily capture the relevant sequence. On the most complex tasks, the abstract reports an 11.2-point advantage over a comparator. In the longest decile of records—more than 541,000 characters, covering 10 patients and 47 question pairs—the agent's advantage was 10.3 points and the reported p value was 0.017. Multi-step search can help when evidence is distributed across diagnoses, laboratory trends and successive treatment lines.

That does not mean every long record should be delegated to an agent. The longest-record analysis is a small subgroup, and longer charts may contain repeated or contradictory data that make reference answers themselves difficult. A useful product would show the evidence trail for each extracted fact, distinguish recorded absence from missing documentation and invite the clinician to resolve contradictions. Summarisation without provenance could make an old error easier to repeat.[1]

Practical value lies in preparation and handover, not autonomous judgement

For haematology teams, a carefully constrained tool could prepare a timeline before a tumour board, identify when a treatment began or stopped, assemble staging evidence and flag missing values for review. That could reduce clerical search during handovers and consultations. The study does not show that it saves clinician time, changes treatment or improves survival, quality of life or patient understanding. Those outcomes require a prospective workflow comparison.

The error-severity finding supports a human-in-the-loop design. Every answer should link to the source note or laboratory result, display uncertainty and avoid silently filling gaps. High-risk questions such as transplant or trial eligibility should require explicit review of current criteria. Institutions would also need access controls, audit logs, retention rules and monitoring for changes in record format and clinical practice. A locally run model reduces some data-transfer concerns but does not by itself make the workflow compliant or safe.

The authors tested several additional open-model backbones, with agent results ranging from 76.8% for Qwen3 to 81.6% for Gemma4-31B in their reported comparisons. That suggests the workflow is not tied to one model, but it also shows that model choice changes performance. Revalidation would be necessary after any model, prompt, skill library or retrieval update.[1]

Why the result cannot yet be generalised to routine care

The main evaluation is retrospective and centred on one German tertiary hospital. Documentation practices, treatment access and record structure may differ in community hospitals and other countries. The 20-patient MIMIC-IV exploration is too small to settle transfer. Multiple myeloma is also unusually longitudinal; results cannot be assumed for emergency care, primary care or cancers with different staging and treatment patterns.

The study reports public and charitable funding, including Bavarian and German research support, and says funders had no role. Several authors disclose oncology-industry honoraria, advisory work or travel support. Those disclosures do not invalidate the evaluation, but readers should keep them visible when considering a clinical technology claim. There was no company-funded prospective deployment in the evidence reported here.

A stronger next study would lock the system, preregister error categories and compare clinicians with and without assistance across several institutions. It should measure time, citation checking, corrected and introduced errors, decisions changed, patient-facing communication and privacy incidents. Performance should be stratified by record length and question type with adequate denominators. Until that evidence exists, the system is best understood as a promising chart-synthesis assistant whose outputs require expert verification.[1]