Can AI catch specialist referrals that primary care forgot to send? Singapore's system found more—but missed half
Across seven public clinics, a weekly AI safety check found 225 clinician-verified referral failures over 20 weeks. Its 48.8% sensitivity means it cannot be the only safeguard.
The short version
- What happened
- The MR-AI system scanned roughly 25,000 consultation notes each week across seven Singapore primary-care clinics for clear referral intent without a matching electronic order.
- Who it affects
- A missed referral can leave a patient waiting for an appointment that was never ordered. This system gave clinics another chance to find and repair some of those failures, but it missed about half of the reference cases and did not demonstrate faster treatment or better health outcomes.
- What remains unproven
- Prospective quasi-experimental study
Keep in mind
Prospective quasi-experimental study
Seven clinics in one Singapore health system
20-week intervention
About 25,000 notes scanned weekly
428 flags and 225 clinician-verified cases
Single-week 20,369-note accuracy sample
43 reference missed referrals
48.8% sensitivity
99.9% specificity
Human reference standard
No patient outcome or cost-effectiveness result
True incidence unknown
Proprietary system and prompts
Source analysis · 1 linked source. Primary analysed source date: 30 October 2026.
Read the full report · 5 min The evidence, methods, implications and limitations
A weekly scan found referral failures before patients had to chase them
An AI-supported safety check substantially increased detection of specialist referrals that clinicians intended but did not order. Across seven public primary-care clinics in western Singapore, MR-AI scanned about 25,000 consultation notes each week. Over a 20-week intervention it raised 428 possible failures; senior clinicians verified 225. Patients separately reported 31, taking the total detected during the period to 256.
The peer-reviewed study compared that period with two matched 20-week baselines, each of which recorded 18 patient-reported missed referrals. An interrupted time-series model estimated an immediate 4.7 percentage-point increase in the weekly proportion of clinic visits with a detected missed referral (95% confidence interval 2.4 to 6.9 points; p<0.001), about seven times the pre-implementation proportion. This measures detection, not proof that the underlying failure rate rose or that patients had better health outcomes.[1]
High specificity kept the review list short, but sensitivity was moderate
MR-AI first used rules to filter notes, then a locally deployed Llama 3.1 8B model classified whether the text expressed definite referral intent without a corresponding order. In a blinded accuracy exercise covering 20,369 notes from one week, seven reviewers established 43 missed referrals. The AI found 21 and missed 22: sensitivity was 48.8% (95% CI 33.3% to 64.5%). It correctly left almost all negative notes unflagged, with 99.9% specificity, and 21 of its 34 flags were confirmed, a positive predictive value of 61.8%.
That operating point was deliberately conservative to limit alert fatigue. Manual review of the same week took about 41 more staff-hours than AI processing plus verification, according to recorded spreadsheet activity and system logs. Across the full intervention, 225 of 428 AI flags were confirmed, a 52.6% verified-flag rate. Clinicians still had to inspect every flag, contact patients, correct the referral and record the incident; the model did not complete those safety tasks.[1]
The distinction that matters
What the blinded accuracy week showed
Seven reviewers assessed 20,369 notes without recorded referral orders. Their adjudicated decisions formed the study's reference standard.
Reference missed referrals
43 cases
21 found by MR-AI; 22 missed
Sensitivity
48.8%
The safety net detected about half of reference cases
Specificity
99.9%
Only 13 of 20,326 reference negatives were flagged
AI flags
34 cases
21 confirmed; PPV 61.8%
The system is a backstop, not evidence that referral care improved
For patients, an earlier warning could shorten the period in which an intended specialist appointment simply does not exist. For clinic teams, scanning can focus scarce review time on a short list rather than every note. But the study did not measure waiting time, diagnosis, treatment, harm avoided or cost-effectiveness. Only 123 of the 225 clinician-verified cases were entered in the operational incident system, and the clinical urgency of many referrals could not be determined retrospectively.
This was an observational evaluation in one Singapore health system. Patient reporting and retrospective screening were both incomplete, the true incidence is unknown, and the reference standard depended on human interpretation. The one-week accuracy sample had a 0.21% prevalence, which makes overall accuracy and negative predictive value look especially high. Seasonal or behavioural changes could also affect the time-series result. Larger prospective studies should test patient outcomes, costs and performance across clinical settings. Until then, MR-AI is evidence for an additional safety net—not for removing ordering controls, documentation standards or human oversight. The authors reported no specific funding or competing interests, although the institutional system and prompts are proprietary.[1]