An Alzheimer’s MRI model scored 94.5%. On new data, it fell to 61.6%

The harder test is the one that matters

An AI model designed to classify Alzheimer’s disease stages from brain scans proved much less accurate when tested on images from another collection. For clinicians considering such systems, the study offers a warning: the first headline score may say little about performance beyond the data a model already knows.

The peer-reviewed engineering study reports 94.5% accuracy for a four-stage classifier on its internal test set. Applied without retraining to 292 images from an independent collection, the same model reached 61.6%, with macro-F1 falling to 0.565. The roughly 33-percentage-point gap matters more than the impressive first score: the result did not carry over reliably to the new collection.

The researchers deserve credit for reporting the external test and for stating that the model is not ready for clinical use. The work shows partial transfer, especially for visually distinctive moderate dementia, but performance on the clinically difficult intermediate categories deteriorated sharply. It is a methods study using public image collections, not a prospective diagnostic trial, and it provides no evidence that using the model improves diagnosis, treatment or patient outcomes.[1]

How the model and data were assembled

The system combines an ImageNet-pretrained ConvNeXt-Tiny image backbone with a Kolmogorov–Arnold Network, or KAN, classification head. The head maps a 768-dimensional feature vector to four labels: non-demented, very mild, mild and moderate dementia. Training used mild image transformations, MixUp, CutMix, label smoothing, AdamW optimisation and test-time augmentation.

The publicly distributed OASIS slice collection contained 86,437 axial images: 67,222 labelled non-demented, 13,725 very mild, 5,002 mild and 488 moderate. The researchers capped each class at 1,500 slices, held out balanced validation and test sets before oversampling and replicated only the remaining training images. That avoids the obvious error of duplicating a test image into training and gives each class equal weight in the 292-slice internal test set.[1]

Patient-level leakage remains unresolved

The more important dependence problem could not be eliminated. One three-dimensional MRI scan produces many neighbouring two-dimensional slices, but the public slice collection does not retain per-slice patient identifiers. The study could check that files were not bit-for-bit duplicates across partitions, yet it could not stop slices from the same person appearing in both training and test sets. Similar anatomy across adjacent slices can make internal performance look better than generalisation to a new patient.

Calling the internal protocol leakage-controlled therefore needs a precise qualification: it controls duplication caused by oversampling, not shared-patient leakage. The authors acknowledge this and name a patient-level split as the most important next step. They also used one fixed split rather than repeated hold-outs or cross-validation, so the stability of the 94.5% estimate across alternative partitions is unknown.[1]

The external errors concentrate where staging matters

The independent dataset used the same four labels and contributed 73 images per class. Moderate dementia remained perfectly recalled at 73 of 73 images, and non-demented recall was 79.45%, close to the internal 82.19%. Mild-dementia recall, however, collapsed from 98.63% internally to 10.96% externally. Very-mild recall fell from 97.26% to 56.16%.

Many mild images shifted into the neighbouring very-mild or moderate categories. That pattern is consistent with acquisition differences, contrast, slice selection and label construction moving the decision boundary. It is not enough to say the external set is harder: a model proposed for staging needs trustworthy discrimination at the earlier and intermediate stages, where care decisions are sensitive and visible structural differences are subtle.[1]

The novel head was not the source of a clear gain

In a matched shortened training protocol, the KAN and conventional multilayer-perceptron heads were statistically indistinguishable on accuracy (McNemar p=0.52). The MLP led on several threshold-dependent metrics, while KAN led on ROC-AUC. The KAN head had about nine times as many head parameters, though it increased the full model by only 2.84% and added no material runtime in the reported GPU test.

The paper's ablation instead identifies pretrained features and expanded data use as the principal contributors. Its training-partition resampling also outperformed the tested focal-loss, class-balanced-loss and feature-space SMOTE approaches. Those results are useful engineering evidence, but they weaken any simple claim that the new KAN component produced the headline performance.[1]

What clinicians and health systems should take from it

Clinicians should not use this model to stage a patient. The labels came from public image repositories, the unit of evaluation was an image slice rather than a patient pathway, and the study did not compare predictions with a blinded multi-reader clinical reference. Saliency maps concentrated on plausible brain regions, but no quantitative test established that those maps were faithful explanations or that clinicians could act on them safely.

For health systems, the external decline is a practical demonstration of why local validation must precede deployment. Testing should preserve patient separation, use scans from every intended scanner and population, report calibration and clinically defined errors, and evaluate the entire workflow—including radiologist review, uncertainty handling and the consequences of a wrong stage—not only per-image accuracy.[1]

What would change the assessment

The next evidence should use subject-linked volumetric MRI with patient-level partitions, repeated or cross-validated estimates and multi-site cohorts such as ADNI under an appropriate data-use agreement. A locked model should be evaluated prospectively against clinical diagnoses and independent readers, with age, sex, scanner, comorbidity and disease-spectrum breakdowns. Domain adaptation must be trained separately from the external test it is intended to pass.

Until then, the most important contribution is not a deployable Alzheimer’s classifier. It is a transparent example of how internal success can shrink under distribution shift and how a novel architectural component may add less than disciplined data handling. The external test makes the paper more useful—and the clinical claim much narrower—than the 94.5% headline alone would suggest.[1]