Can AI turn rare-disease notes into reliable treatment evidence?
A 108-patient Rett-syndrome study extracted treatment-related features from 26,763 notes, but only one of 18 targets had a full ground truth—and documentation is not clinical improvement.
The short version
- What happened
- The single-centre study analysed 26,763 notes from 108 patients with sustained trofinetide exposure over 30 months.
- Who it affects
- Families and specialist teams could gain a faster way to find treatment-related information scattered through years of notes.
- What remains unproven
- The study is an unreviewed, single-centre retrospective analysis. Only severe diarrhoea had a full expert reference set, and the pipeline measured documented presence rather than direction of change, severity or causation.
Keep in mind
The study is an unreviewed, single-centre retrospective analysis. Only severe diarrhoea had a full expert reference set, and the pipeline measured documented presence rather than direction of change, severity or causation.
Source analysis · 1 linked source. Primary analysed source date: 15 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The system organised notes; it did not prove a drug worked
Researchers used two large language models in agreement to turn 26,763 clinical notes from 108 people with Rett syndrome into a timeline of 18 treatment-related features. The pipeline could help rare-disease teams search narrative records that are too labour-intensive to review repeatedly. Its output, however, is whether a feature was documented at a visit—not whether the patient improved, worsened or changed because of trofinetide.
The unreviewed study came from Texas Children's Hospital and followed patients with confirmed sustained exposure for up to 30 months. It tracked four adverse effects, 13 features of Rett syndrome and medication refusal. Notes first passed a keyword filter; two model calls then classified the same material, and the pipeline required consensus before producing a patient-by-timepoint matrix.[1]
Only severe diarrhoea had a full reference set
Prompt design was developed on severe diarrhoea, the treatment's best-known adverse effect. The reported comparison used 183 expert-labelled cells from 56 patients and included 47 positives. The selected prompt scored 84.2% accuracy and F1 0.674, 47 percentage points above regular-expression matching. But it was selected and scored on the same cells, so the authors call that accuracy an in-sample upper bound rather than independent validation.
For the other 17 targets, experts reviewed only six calls per phenotype—three positive and three negative—and agreed with 92 of 102. That 90.2% spot-check agreement cannot establish sensitivity or prevalence. Later follow-up also thinned sharply: just 13 evaluable cells remained at 24 months and six at 30 months. Clinical notes missed information carried by phone or patient portals, and time-window placement explained 12 of the 29 diarrhoea discrepancies. The four-person clinical team also resolved labels together rather than annotating independently, so the reliability of the reference standard itself could not be measured.[1]
The distinction that matters
What the pipeline can and cannot conclude
The same note moves through filtering and two model calls before becoming a timeline entry.
Find
Keyword-filter 26,763 notes
Candidate passages for 18 features
Classify
Require two LLMs to agree
Documented presence at each timepoint
Clinical meaning
Compare with severity and external evidence
Still needed to infer improvement or harm
Why clinicians still need a separate evidence step
The tool could make chart review more scalable, especially when a small specialist service needs to locate side effects or follow-up discussions. It cannot attribute trajectories to the medicine rather than concurrent care or the underlying condition. Thirteen core features were recorded as present in 92–100% of patients at every visit, illustrating why presence alone cannot reveal treatment response.
The work had public and charitable funding. Two authors disclose consulting relationships, including with trofinetide maker Acadia Pharmaceuticals; the paper does not say those funders or companies controlled the analysis. The underlying notes cannot be shared because of privacy rules, although prompts, keyword lists and aggregate results are available. A credible next step needs independent annotations for every phenotype, external hospitals, severity or direction-of-change measures and prospective tests of whether clinicians find problems faster without missing them. Until then, this is a record-structuring study, not evidence of clinical benefit.[1]