Can a delirium-risk model transfer without retraining? Three in four alerts were false positives
A model deployed at a second Mount Sinai hospital kept an AUROC of 0.91 and was associated with more detected delirium. Yet its positive predictive value was only 26%, and most admissions were labelled from billing codes rather than a uniform bedside assessment.
The short version
- What happened
- The model scored 9,020 admissions at Mount Sinai Morningside and achieved AUROC 0.91, sensitivity 92% and specificity 84% at its operational threshold without local retraining.
- Who it affects
- Delirium teams may use the score to prioritise assessments, but roughly three in four high-risk classifications did not match the study's delirium label.
- What remains unproven
- Both hospitals belonged to one health system and shared EHR infrastructure, workflows and data definitions.
Keep in mind
Both hospitals belonged to one health system and shared EHR infrastructure, workflows and data definitions.
Only 2,649 of 9,020 external-site admissions received CAM; the remainder used diagnosis codes as the reference label.
The model helped select patients for CAM, creating verification and workflow-incorporation bias.
The pre-model and post-model cohorts came from different years and were not randomised.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The direct finding: discrimination travelled, but precision did not become clinically comfortable
A JAMA Network Open study reports that a delirium-risk model trained and developed at Mount Sinai Hospital retained an area under the receiver operating characteristic curve of 0.91 after deployment at Mount Sinai Morningside, without retraining on the second hospital's data. At the external site's prespecified threshold of 0.50, sensitivity was 92% and specificity 84%. The result matters because clinical models commonly deteriorate when moved into a different population or workflow, and local rebuilding requires data engineering, specialist staff and continuing maintenance that many hospitals cannot supply.
The operational weakness appears in the positive predictive value: 25.9%, reported as 26% in the paper. Put plainly, about one in four admissions labelled high risk met the study's delirium outcome, while about three in four did not. The negative predictive value was 99%, but that figure is aided by the low measured prevalence of 5.7% at Morningside. AUROC describes ranking across thresholds; it does not say how much work a chosen threshold creates. A service deciding whether to deploy this model must therefore budget for false alerts and trained assessment capacity, not stop at the 0.91 headline.[1]
The denominator: 33,095 admissions across two hospitals, with different labels inside the external cohort
The quality-improvement analysis covered 33,095 non-intensive-care admissions involving people aged 60 or older. During the contemporaneous deployment period from 1 December 2024 to 31 December 2025, the model scored 9,020 admissions at Morningside and 21,186 at the development hospital. The patient groups differed materially: Morningside had an older median age, more medical rather than surgical admissions, a higher recorded dementia prevalence and different racial and ethnic composition. Those differences make the transfer test more informative than reusing a held-out sample from one hospital, even though both sites remain inside one health system.
The reference standard was not uniform. At Morningside, 2,649 of 9,020 admissions had a Confusion Assessment Method assessment, while the remaining 6,371 were classified using delirium-related ICD-10-CM codes. CAM is a structured bedside assessment; a diagnosis code is created through a different clinical and administrative process and can miss undocumented cases. The model itself prioritised high-risk patients for CAM assessment, so the people receiving the stronger reference test were selected partly by the prediction under evaluation. This differential verification can make performance look better or worse in ways that a simple confusion matrix does not resolve.[1]
The distinction that matters
How the result changes from ranking to real work
Good discrimination did not remove the need for bedside confirmation or service-capacity choices.
External site
Score 9,020 admissions
AUROC 0.91 without retraining
High-risk threshold
Prioritise CAM assessments
92% sensitivity and 84% specificity
Alert burden
Compare alerts with delirium labels
26% positive predictive value
Workflow yield
Compare monthly CAM detection
9.23% before and 19.59% after
Evidence boundary
Inspect design and reference standard
Historical comparison and selective testing
Detection increased after deployment, but the before-and-after comparison is not a causal trial
The mean monthly rate of CAM-positive delirium among assessed patients increased from 9.23% before the model to 19.59% after deployment at Morningside. An adjusted logistic model estimated 1.63 times the odds of delirium detection after deployment, with a 95% confidence interval from 1.37 to 1.94. That is an important workflow result: a small delirium service used the score to focus daily assessments, and a larger share of those assessments found delirium. It indicates that risk ranking may help scarce specialist attention reach patients more likely to need it.
It does not establish that the model doubled true hospital-wide detection. The pre-model cohort covered October 2018 to September 2019, whereas the post-model cohort covered December 2024 to December 2025. Patients, documentation, staffing and clinical practice could have changed across that long interval. The denominator also changed: the earlier group included 2,889 admissions that all received CAM, while only 2,649 of 9,020 post-deployment admissions received CAM. The outcome was yield among selected assessments, not a uniformly measured delirium incidence across every admission. A randomised or stepped-wedge deployment would offer stronger causal evidence.[1]
This is within-system transportability, not independent health-system validation
The two hospitals serve different communities, but they share Mount Sinai's organisation, Epic infrastructure, data definitions, documentation practices and custom workflow tools. The development team also operated the model and evaluated its transfer. Those common elements reduce the technical shocks that often break a model elsewhere. The paper appropriately describes factors that may have preserved performance, including the heterogeneous medical and surgical development population and the shared electronic record. The evidence supports transport within this system; it cannot yet support a claim that another hospital can install the model unchanged and expect the same results.
Even the operating thresholds were site-specific: 0.50 at Morningside and 0.55 at Mount Sinai Hospital. At the development site, AUROC was 0.94, sensitivity 84%, specificity 89% and positive predictive value 19%. Different thresholds reflected local staffing and screening capacity, demonstrating that deployment is partly an operations decision rather than a property of the model alone. An organisation with fewer assessors might need a higher threshold and miss more delirium; one prioritising sensitivity might accept more false alerts. Decision-curve analysis, calibration and explicit capacity modelling would help leaders choose that trade-off more transparently.[1]
What it means for nurses, doctors, patients and hospital leaders
For the delirium team, the score is best understood as a queueing tool: it identifies whom to assess first, not who definitively has delirium. Nurses and doctors still need to recognise an acute change, complete an appropriate assessment, search for causes and act on the result. A false positive can consume scarce time and contribute to alert fatigue; a false negative can delay recognition in an older patient whose confusion may otherwise be misattributed. The visible score should therefore be paired with clear escalation routes and should never prevent assessment when clinical concern conflicts with the model.
For patients and families, earlier recognition can matter because delirium is distressing and often signals an acute medical problem. Yet this study did not randomise care or report patient-centred outcomes from the external deployment, such as duration of delirium, falls, discharge destination, mortality or family experience. It evaluated discrimination and detection yield. Hospital boards should insist on local monitoring of both benefit and burden: assessment volume, time to review, false-alert distribution, missed cases, medication use and outcomes, stratified across demographic and clinical groups. The authors reported no conflicts of interest, and the institutional review board granted a consent waiver for minimal-risk record use.[1]
What evidence would change this assessment
Confidence would rise with prospective validation in geographically and organisationally independent health systems using a frozen model, predefined thresholds and a uniform reference assessment applied to a representative sample regardless of risk score. Calibration plots, decision curves and subgroup error rates should accompany AUROC. A stepped-wedge or cluster-randomised implementation could test whether the prioritisation workflow improves timely recognition and patient outcomes without overwhelming staff. Reporting the number needed to assess per additional confirmed case would make the operational value easier for clinical services to judge.
The assessment would weaken if performance falls outside the shared Epic environment, if false positives concentrate in particular groups, or if detection gains disappear when every patient receives the same reference test. It would strengthen if independent teams reproduce the result and show improved outcomes at sustainable workload. For now, the study is consequential peer-reviewed evidence of a live model travelling between hospitals in one system, with unusually clear numbers on its false-positive burden. It is not Breaking: it reports a completed quality-improvement evaluation, not an urgent safety alert, regulatory action or newly unfolding clinical incident.[1]