An AI agent sped cancer-toxicity review—but prospective performance was lower
A Boston system reduced staff review time by 2.71 minutes per note in a 17-person crossover study. Silent prospective detection F1 was 0.72–0.79, below the 0.92 retrospective result.
The short version
- What happened
- Researchers tested six immune-related adverse-event categories in 263 retrospective test notes, 884 prospectively processed notes and a randomized crossover study involving 17 clinical research staff.
- Who it affects
- A patient receiving immunotherapy can develop serious side effects whose details are scattered through notes. Faster review may help research and safety monitoring, but the system has not shown that it improves diagnosis, treatment or outcomes.
- What remains unproven
- Boston academic centres
Keep in mind
Boston academic centres
263 retrospective test notes
884 silent prospective notes
17-person crossover study
No change to patient care
No patient outcomes
External validation still needed
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
Faster review, with a smaller real-time score
An agentic language-model system helped clinical research staff review cancer-immunotherapy toxicity notes faster and more consistently, according to a peer-reviewed Nature Cancer study published on 29 October. In a randomized crossover study, 17 staff members each reviewed 20 notes with and without assistance. The assisted condition reduced annotation time by 2.71 minutes per note, increased the odds of an exact match with the reference label by 45% (95% CI 1.01–2.08), and raised agreement between reviewers.
The technical result weakened when the system moved from a retrospective test to silent real-time use. Current-event detection reached a macro F1 of 0.92 on the retrospective evaluation, while binary F1 across three months of prospective processing ranged from 0.72 to 0.79. F1 combines precision and recall; it does not say whether every clinically dangerous event was caught. Multi-class severity grading was harder still, with retrospective F1 of 0.66.[1]
Three phases tested different questions
The team assembled notes from adults treated with immune checkpoint inhibitors at Boston academic hospitals between 2015 and 2024. It focused on six adverse events—pneumonitis, hepatitis, myocarditis, dermatitis, thyroiditis and colitis—and asked the system to extract whether an event was current or past, its grade, clinician-attributed cause and certainty. Phase one used 289 notes from 100 encounters: 26 development notes and 263 test notes.
Phase two processed 884 notes prospectively in silent mode from May to July 2025. The first 102 were dual-annotated; the remainder received one human annotation. Phase three randomized the order of assisted and manual review for 17 coordinators and nurses. That design measures annotation work, not clinical diagnosis: no prediction changed treatment, triggered an alert or demonstrated fewer hospital admissions, missed toxicities or deaths.[1]
The distinction that matters
Each phase answered a different question
The study moved from technical testing to silent deployment and then to staff workflow.
Retrospective
263 held-out notes
Current-event F1 0.92
Silent prospective
884 notes over three months
Binary F1 0.72–0.79
Crossover
17 staff; 20 notes each
2.71 minutes saved per note
Human verification remains part of the system
The best configuration used repeated model runs and consensus, costing about US$0.021 per note. Removing that self-consistency step approximately halved cost but reduced detection F1 from 0.92 to 0.78. A 779-term rules baseline also performed reasonably for simple binary detection, a reminder that the more complex agent must justify its extra cost through reliable extraction of grade, attribution and supporting evidence—not merely by finding keywords.
The study was funded by NIH/NCI and other research funders; several authors disclosed healthcare, pharmaceutical or AI-company relationships. External hospitals, other record systems and outcome-focused trials remain necessary. The assessment would change if independent teams reproduce the prospective results, measure event-specific misses and show that human-verified alerts improve timely care without adding harmful false alarms or automation bias.[1]