Can AI reliably extract breast-cancer trial data from health records?
For many fields, but not all. In a peer-reviewed single-hospital study of 113 patients in 11 breast-cancer trials, an AI pipeline matched manually curated trial forms well on several variables, yet accuracy fell to 46% for treatment start dates and 23% for end dates. The system remains an assistive extraction tool requiring human verification.
The short version
- What happened
- MIRROR retrospectively compared AI-extracted data with routinely curated electronic case-report forms for 113 patients enrolled in 11 breast-cancer trials at one Seville hospital.
- Who it affects
- Research staff could spend less time copying well-defined fields and more time reviewing complex records and supporting participants.
- What remains unproven
- All 113 patients came from one Spanish hospital and 11 trials run through the same research network; there was no external hospital or EHR validation.
Keep in mind
All 113 patients came from one Spanish hospital and 11 trials run through the same research network; there was no external hospital or EHR validation.
The routinely curated eCRF was used as the operational reference standard without a parallel independent annotation set or formal inter-annotator agreement.
For selected fields, up to 10% of available data helped refine extraction pipelines; evaluation used the remaining 90%–100%, but no fully independent external dataset.
Accuracy varied sharply by field, falling to 46% for study-treatment start dates and 23% for end dates.
The study did not measure staff time, cost, regulatory quality, patient outcomes or whether automated extraction changed a trial decision.
The proprietary system was not directly integrated into the hospital’s main EHR, creating an additional interoperability constraint.
Source analysis · 1 linked source. Primary analysed source date: 7 October 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The direct answer: useful for defined fields, unreliable as a complete replacement
The MIRROR study shows that an AI pipeline can reproduce many data fields used in breast-cancer research with high agreement against manually curated trial forms. Birth year and administrative sex matched at 100% in this dataset; menopausal status reached 91% accuracy, drug-use status 96%, current cancer 90%, receptor information 88% and study treatment 99%. The system also attached references back to the source text, giving a reviewer a way to inspect why each value was extracted.
Reliability was not uniform. Accuracy was 83% for smoking and previous cancer, 81% for previous non-cancer disease and 87% for laboratory tests. Most importantly, extracting the start date of study treatment achieved 46% accuracy and the end date 23%. Those are not marginal fields in a clinical trial. The authors therefore position the system as an assistive tool that can accelerate well-defined extraction while leaving complex and temporally sensitive variables under human review.[1]
What the study compared
MIRROR was a retrospective longitudinal study at Virgen del Rocío University Hospital in Seville. It included 113 adults with early or advanced breast cancer who had participated in 11 clinical trials between January 2012 and December 2021. The hospital was one recruiting centre for each trial. The comparison used electronic case-report forms, or eCRFs, that clinical-research staff had populated during the original trials as the operational reference standard.
The AI system extracted the same kinds of information directly from the hospital’s electronic records. The evaluation covered 18 domains: birth date, sex, menopausal status, drug use, alcohol use, smoking, previous cancer, previous non-cancer disease, family history, current cancer, receptor status, study treatment, concomitant medication, laboratory tests, radiological response, overall survival, and study-treatment start and end dates. Family history could not be scored because the eCRFs did not contain it, even though 11 patients had family-history information in their records.[1]
The pipeline combined several techniques rather than one model
The paper does not describe a single all-purpose language model reading every chart. Its CapTrial pipeline applied different methods to different data types. Structured hospital tables were ingested and normalized through extract-transform-load routines. Unstructured documents were selected and preprocessed, then analysed with regular expressions, medical named-entity recognition, a masked RoBERTa model and, for fields requiring context across notes, a GPT-4-based aggregation step. Extracted entities were normalized into predefined formats before patient-level comparison with the eCRF.
No task-specific model was trained on the complete MIRROR cohort. For selected tasks, a limited development subset of at most 10% of available data was used to refine extraction pipelines. Reported performance was then assessed on the remaining 90%–100%, depending on the field. That separation reduces direct reuse of evaluation examples, although the paper does not provide external testing from another hospital, record system or language environment.[1]
The denominator was substantial, but it still represents one local system
Across the 113 patients, the system processed 138,454 structured records and approximately 1.85 million words of unstructured text. Those volumes are more informative than quoting patient count alone because each person’s longitudinal record can contain many laboratory results, medication entries, pathology reports and clinical notes. The system’s ability to retain a reference to the original text is especially valuable when thousands of candidate entries must be checked.
Volume does not create geographic or technical generalisability. All records came from one hospital, and the proprietary extraction software was not integrated directly with the hospital’s main EHR. The researchers had to work with an intermediate data system, introducing interoperability and data-loss constraints. The results therefore show what this pipeline could retrieve from this local record environment, not what any hospital could expect from a plug-and-play product.[1]
Where performance was strongest—and where it broke down
The paper reports 100% accuracy, precision, recall and F1 for birth year and sex. Menopausal status achieved 91% accuracy and 90% F1; drug-use status 96% accuracy and 94% F1; alcohol use 87% accuracy and 93% F1; smoking 83% accuracy and 81% F1; previous cancer 83% accuracy and 88% F1; previous non-cancer disease 81% accuracy and 90% F1; receptor status 88% across the four reported measures; and radiological response 83% accuracy with a 91% F1. Some fields were evaluated as direct matches rather than binary classifications, so precision and recall were not always applicable.
High headline accuracy sometimes depended on restricting the comparison to dates or entries that could be aligned between systems. Concomitant medication and overall survival were reported at 100% accuracy under the study’s matching rules, while underlying record counts differed between the eCRF and EHR-derived data. By contrast, study-treatment start and end dates performed poorly because the reference date and the most recent relevant clinical report did not consistently correspond. This is exactly the kind of temporal ambiguity that makes automated clinical abstraction difficult.[1]
Why the manual trial form is an imperfect reference standard
MIRROR treated the trial eCRF as operational ground truth because trained research staff had completed it during regulated studies. That is pragmatic, but it does not prove the manual value was always correct. The researchers did not generate a second independent human annotation set and did not calculate inter-annotator agreement. Manual transcription errors, incomplete forms and discrepancies between the trial snapshot and later health-record information can all look like AI errors when concordance is the metric.
The source-linking feature partly addresses this problem. When the AI and eCRF disagreed, reviewers could inspect the exact EHR fragment and determine whether the pipeline misread the record, the information was missing, or the reference form differed. The paper says this helped identify discrepancies, but it does not report a separate adjudicated accuracy estimate after resolving every disagreement. The safest interpretation is therefore agreement with routine manual capture, not absolute clinical truth.[1]
What this could change for trial teams and patients
Clinical-trial teams spend substantial time transferring information from health records into structured research forms. Automating dependable fields could reduce repetitive work and let staff focus on ambiguous cases, data quality and patient-facing activity. Source references can also make review faster than a system that returns an answer without showing where it came from. More complete extraction might support real-world evidence studies and reduce duplicate entry across research systems.
Those benefits were not measured here. The study did not compare abstraction time, staff workload, cost, query rates, regulatory inspection findings or downstream trial decisions. It did not alter patient care or test whether better data capture improves recruitment, safety reporting or treatment outcomes. Patient records also raise privacy and governance issues: any deployment would need strict access control, audit trails, local validation and a defined person accountable for approving extracted data.[1]
Funding, commercial interests and what would change the assessment
The authors report no specific grant funding for MIRROR. Several authors were full-time employees of MEDSIR, which sponsored the included trials, or Science4Tech Solutions, the company that developed CapTrial; other authors disclosed research funding, advisory work, travel support, patents or ownership interests involving healthcare companies. Those relationships do not invalidate the results, but they make independent replication especially important.
Confidence would rise with prospective testing across hospitals, EHR vendors, languages and independently adjudicated records. A stronger study would lock the pipeline before evaluation, use a second human abstraction team, report agreement and adjudication by field, and measure time, cost and correction workload. It should also test whether date extraction and temporally complex medication histories improve without losing traceability. Until then, MIRROR supports targeted, human-supervised automation—not autonomous population of clinical-trial databases.[1]