Did an LLM help reviewers find cancer-treatment skin toxicity?
In a 160-patient retrospective study, LLM-assisted reviewers extracted skin toxicities from clinical notes more accurately and consistently than unassisted reviewers. It did not test real-time diagnosis, treatment decisions or patient outcomes.
The short version
- What happened
- The retrospective sample contained 160 immune-checkpoint-inhibitor recipients treated at Massachusetts General Hospital, Brigham and Women’s Hospital or Dana-Farber Cancer Institute.
- Who it affects
- Researchers and registry staff may spend less time searching long cancer records, but they still need authority and training to reject the model’s recommendation.
- What remains unproven
- The retrospective sample contained 160 patients from three hospitals in one regional academic system.
Keep in mind
The retrospective sample contained 160 patients from three hospitals in one regional academic system.
One oncodermatologist supplied the reference labels, so specialist disagreement was not measured.
The prompt was refined in ten iterations on the same 100-case benchmark used for model comparison.
Four non-dermatologist reviewers were divided between conditions rather than each testing both workflows.
No prospective alerting, diagnosis, treatment decision, cost or patient outcome was evaluated.
Source analysis · 1 linked source. Primary analysed source date: 15 October 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The answer: assistance improved a narrow retrospective review task
An LLM-supported workflow helped non-dermatologist reviewers identify skin toxicities linked to cancer immunotherapy more accurately and consistently in a small retrospective study. Compared with unassisted manual review, the assisted group achieved an F1 score of 0.88 rather than 0.77. Agreement between reviewers also rose from a Cohen’s kappa of 0.50 to 0.82, while average self-reported review time was about one-third shorter.
Those results matter because immune checkpoint inhibitors can trigger inflammatory adverse events, and evidence may be scattered across lengthy clinical notes. Researchers building cohorts or safety registries spend substantial time deciding whether a recorded rash or other skin problem was plausibly treatment related. A tool that retrieves the relevant passages and leaves a person to accept, modify or reject its recommendation could reduce that abstraction burden.
The study did not show that the system diagnoses patients, alerts clinicians in time to prevent harm or improves treatment. It reconstructed events from existing records for research review. The correct reading is therefore specific: in this sample and workflow, people given LLM-generated evidence performed a binary chart-classification task better than a separate unassisted pair. That is promising evidence for human-supervised data extraction, not a clinical deployment claim.[1]
What the researchers tested in 160 patient records
The researchers randomly selected 160 people from a clinical database of patients who received immune checkpoint inhibitors at Massachusetts General Hospital, Brigham and Women’s Hospital or Dana-Farber Cancer Institute. The notes covered the period from two weeks before treatment began through two years and 90 days after the first dose. All three institutions belong to the same Boston-area academic ecosystem, so the study is multi-hospital but not a geographically independent validation.
One board-certified supportive oncodermatologist reviewed the records and supplied the reference classification for whether each patient had a cutaneous immune-related adverse event. That expert adjudication gives the study a clinically informed target, but a single adjudicator cannot reveal how often specialists would disagree. It also means the reported accuracy is agreement with one expert-derived reference, not proof that every label represents an objectively certain event.
A 100-patient holdout set was first used to compare four language models in a zero-shot classification task. GPT-4o produced sensitivity of 0.82 and specificity of 0.88, ahead of Llama 3.1-70B, Gemma-2-27B and Qwen-2.5-72B on the balance reported. The researchers then iteratively refined prompts over ten rounds on those same 100 cases before building the final assisted workflow. That reuse is important: it can make the benchmark fit the observed sample better than a fully untouched test would.[1]
The system retrieved passages, proposed a label and invited correction
The final workflow combined dense and sparse retrieval, including BM25 keyword matching, to find potentially relevant text in a longitudinal record. A synthesiser agent assembled the evidence and a critic agent checked the proposed classification. Reviewers saw a binary recommendation together with supporting excerpts rather than an unexplained score. The model ran through GPT-4o on a HIPAA-compliant Azure environment inside Mass General Brigham’s secure infrastructure.
This design addresses two common problems in clinical language-model work. First, retrieval narrows attention to the notes most likely to contain an event, which can make a long record manageable. Second, showing the underlying excerpts gives a reviewer something concrete to inspect. It does not guarantee factuality: an irrelevant passage can still be retrieved, and a persuasive summary can still be wrong. The ability to reject or edit the output is therefore part of the intervention, not an optional safety decoration.
Four non-dermatologist reviewers were trained by the specialist and divided into two pairs. One pair reviewed records manually; the other used the LLM interface and could accept, modify or reject its recommendation. Because different people populated the two conditions, the comparison combines the effect of the tool with any differences between reviewers. A crossover design in which the same reviewers used both workflows on separate, randomised cases would have separated those effects more cleanly.[1]
Accuracy and agreement improved, but the denominator is modest
The assisted reviewers achieved sensitivity of 0.91, specificity of 0.92, precision of 0.86 and F1 of 0.88. The unassisted pair achieved sensitivity of 0.79, specificity of 0.87, precision of 0.76 and F1 of 0.77. Raw agreement within the assisted pair was 92.6%, compared with 76.5% for manual review; chance-adjusted agreement rose to kappa 0.82 from 0.50, with a reported p value below 0.001.
These measures capture different costs. Higher sensitivity means fewer specialist-labelled toxicities were missed; higher specificity means fewer records without the target event were wrongly marked positive. Precision tells researchers how many flagged cases were true according to the reference. F1 summarises the balance between recall and precision, while kappa asks how consistently the two reviewers reached the same answer beyond expected chance agreement.
The paper’s peer-reviewed abstract describes review time as approximately one-third shorter. That is a correction worth preserving because the earlier manuscript version described the saving as roughly half. Time was self-reported rather than recorded automatically, and a modest number of reviewers completed a specialised task after study training. The finding supports a prospective timed evaluation; it does not yet establish staffing savings in an operational cancer service.[1]
The most defensible use is research abstraction, not bedside diagnosis
For clinical researchers, the practical opportunity is faster construction of adverse-event datasets. Better extraction could support safety surveillance, retrospective studies and quality review when ordinary billing codes miss events or fail to capture why clinicians thought a symptom was treatment related. Human review remains valuable because the classification depends on chronology, alternative explanations and context that may not be stated in one sentence.
For clinicians and patients, the boundary must remain visible. The model was not evaluated as an alert during treatment, did not compare itself with dermatologists making live diagnoses and did not measure whether anyone received earlier care. A missed toxicity could delay assessment; a false positive could lead to anxiety, referrals or treatment disruption. Before any clinical use, a locked system would need prospective testing at the point where an alert changes work, with clear escalation rules and patient-important outcomes.
The study also used protected records and a commercial model in a secure institutional environment. Other hospitals may have different note styles, terminology, patient populations, documentation gaps and cloud arrangements. Reproducing the workflow requires not only prompts but compatible data access, privacy controls, model-version tracking, logging and a workforce able to review the evidence. Accuracy in one research pipeline does not make those organisational requirements disappear.[1]
Evidence limits and what would change the assessment
This is a peer-reviewed open-access Brief Communication with a clearly described clinical question, comparator and reference process. It reports both autonomous-model and human-assisted performance, and the assisted people outperformed GPT-4o acting alone. Funding came from research foundations and US public grants; the corresponding author disclosed advisory or consulting relationships and honoraria from several pharmaceutical and biotechnology companies, stated as financial competing interests.
The evidence remains preliminary. The total sample was 160 people; the prompt was refined on the 100-case benchmark; one specialist supplied the reference labels; and only four non-dermatologist reviewers took part. The hospitals belong to one health system and region. Review time was self-reported. There was no prospective workflow, no independent external cohort, no cost analysis and no test of whether the system changed diagnoses, treatment or outcomes.
Confidence would rise with a preregistered, locked-model study at independent hospitals, multiple specialist adjudicators, automatically recorded review time and a crossover or randomised reader design. Clinical use would need a prospective trial that measures missed and false alerts, time to specialist assessment, treatment changes, patient harm and performance across demographic and cancer subgroups. Until then, the result supports a carefully supervised research tool, not autonomous toxicity surveillance.[1]