Why did a thyroid AI benchmark look almost perfect?
A peer-reviewed audit found that the benchmark's documentation misdescribed its three classes and placed 89 diagnosed hyperthyroid records inside the majority class. A 92.6% accuracy baseline could simply predict that majority class.
The short version
- What happened
- The researcher matched 3,597 benchmark records to diagnosis codes in the same UCI archive with no discordance, finding three hypothyroid-spectrum strata rather than the documented hyperfunction class.
- Who it affects
- Clinicians and patients should not read a near-perfect benchmark score as evidence that a model is ready to diagnose thyroid disease; the study did not test prospective care, external hospitals or patient outcomes.
- What remains unproven
- This is a controlled audit of one public benchmark, not a validation study in a clinic or a representative sample of current patients.
Keep in mind
This is a controlled audit of one public benchmark, not a validation study in a clinic or a representative sample of current patients.
The diagnosis-code match clarifies the archive's class structure, but it does not establish that the historic records or hormone cutoffs meet today's clinical standards.
The journal is sharing an accepted peer-reviewed version before final copy-editing; the article may receive editorial changes when the Version of Record replaces it.
The study has one author, reports no external funding and declares no competing interests; independent replication would still strengthen the conclusion.
Source analysis · 1 linked source. Primary analysed source date: 20 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The label audit changed what the benchmark was measuring
The ann-thyroid dataset is widely used to compare machine-learning classifiers, often with near-perfect-looking scores. Osman Eray at Akdeniz University audited the public benchmark before asking which algorithm won. The UCI files do not name their three classes, while the documentation describes a hyperfunction class. Matching 3,597 records to diagnosis codes distributed in the same archive produced one-to-one agreement and instead identified overt hypothyroid, subclinical hypothyroid and euthyroid groups. The audit also located 89 diagnosed hyperthyroid records inside the majority class.
That imbalance makes overall accuracy a poor guide. Always predicting the majority class already scores 92.6% accuracy, while its macro-F1 is just 0.320 because the rarer classes are largely missed. The finding does not show that thyroid diagnosis is easy or that the labels are clinically definitive. It shows that a benchmark can reward a plausible-looking score even when its class meaning and minority performance have not been checked.[1]
Algorithmic refinements added less than the reporting suggested
The peer-reviewed study tested ten classifiers across ten stratified splits in a leakage-free pipeline. A factorial ablation separated feature selection from class balancing; nine selection methods and seven balancing strategies were compared with statistical tests and equivalence margins set in advance. Class balancing raised macro-specificity from 0.905 to 0.987, but feature-selection methods occupied a ROC-AUC range of only 0.0018 and balancing strategies a macro-F1 range of 0.0098. The detected orderings were reproducible, yet every pairwise difference remained inside the prespecified equivalence margin.
Metric choice also changed the story. In five of ten classifiers, the pipeline significantly reduced precision-recall area for minority classes while ROC-AUC moved by about 0.002. A transparent rule using two cutoffs on thyroid-stimulating hormone and total thyroxine reached 0.885 macro-F1 in 0.2 seconds, comparable with tuned logistic regression. For researchers and procurement teams, the practical lesson is to audit labels, report per-class metrics and compare against simple clinical rules before crediting a more elaborate model.[1]
The distinction that matters
How a high score can survive a weak benchmark
Three checks separate genuine model value from performance inherited from the data.
Labels
Match records to diagnosis codes
The documented class meaning changed
Baseline
Always predict the 92.6% majority
Accuracy looked strong while macro-F1 fell to 0.320
Comparator
Fit two hormone cutoffs
A simple rule rivalled tuned logistic regression