Did data leakage make a Ghana kidney-screening AI look stronger?
A 155-patient preprint found 83.9% accuracy when synthetic cases were created before the train-test split, but 61.5% on 39 genuinely unseen patients. The live tool uses models fitted under the optimistic protocol.
The short version
- What happened
- The retrospective study used records from 155 patients at one Ghanaian hospital: 43 had a recorded CKD level and 112 did not.
- Who it affects
- A low-cost blood-count screen could be valuable where confirmatory testing is harder to access. But an inflated score can falsely reassure patients, create avoidable referrals or persuade clinicians that a model has travelled further than the evidence supports.
- What remains unproven
- Unreviewed retrospective preprint from one Ghanaian hospital.
Keep in mind
Unreviewed retrospective preprint from one Ghanaian hospital.
Only 155 patients and 43 CKD cases; the cleaner holdout contained 39 people and 11 cases.
The outcome was a recorded hospital CKD category, without paired eGFR, creatinine or urine-albumin verification and without confirmation of chronicity.
The deployed models were fitted under the leaky preprocessing protocol; the paper does not report prospective clinical outcomes from website use.
No separate funding or competing-interest statement was identified in the paper.
Source analysis · 1 linked source. Primary analysed source date: 31 October 2026.
Read the full report · 5 min The evidence, methods, implications and limitations
One preprocessing choice moved the result by 22.4 points
A Ghanaian research team has measured how much a common evaluation mistake can flatter a clinical AI score. Its unreviewed study used 155 de-identified records from Ho Technical University Hospital: 43 people had a recorded chronic-kidney-disease level and 112 did not. Ten predictors—age, cardiovascular disease and eight full-blood-count measurements—fed 11 algorithms, six resampling strategies and 13 ensembles.
When the team used SMOTE to create synthetic CKD records before dividing the data, random forest and extra trees each reached 83.9% accuracy on a 56-row test partition. Their ROC AUCs were 0.925 and 0.939. But synthetic examples can be partial copies of people who later appear in the test set. In the cleaner protocol, oversampling occurred only inside training folds. A final ensemble then achieved 61.5% accuracy and ROC AUC 0.604 on 39 real, unseen patients. Its accuracy was also below the 71.8% obtained by always predicting the majority class, although its class-balanced accuracy was 0.622.[1]
The live screening tool uses the optimistic models
The paper is unusually explicit about the consequence: the two models serving NephroScan GH were fitted after SMOTE had been applied to all 155 patients. That created 69 synthetic CKD rows before the split. Among the resulting 56 test rows, accuracy was 93.3% for 15 synthetic rows and 80.5% for 41 real rows—but even the real rows could have synthetic neighbours derived from them in training. The authors therefore call the evaluation optimistic by construction.
For a person using the website, this distinction matters more than the headline AUC. In the leakage-controlled holdout, the model found seven of 11 CKD cases and missed four; it also flagged 11 of 28 people without CKD. The site reportedly averages two forests, explains their contributions and directs users to Ghanaian laboratories and renal centres. It also tells users to obtain estimated glomerular filtration rate and urine albumin-to-creatinine ratio tests. Those safeguards reduce the chance that a score is mistaken for a diagnosis, but they do not validate the deployed model.[1]
The distinction that matters
Same records, different test boundary
The model inputs stayed similar; what changed was whether synthetic cases could carry information across the train-test boundary.
Optimistic protocol
SMOTE before the split; 224 rows including 69 synthetic CKD rows
83.9% accuracy; ROC AUC 0.925–0.939
Cleaner protocol
SMOTE only inside training folds; 39 real people held out
61.5% accuracy; ROC AUC 0.604
Clinical confirmation
eGFR and urine albumin-to-creatinine testing
Still required; the model does not diagnose CKD
A larger prospective study must test the clinical question
The conservative result is itself highly uncertain. The 39-person holdout contained only 11 CKD cases; its ROC AUC confidence interval ran from 0.371 to 0.833, spanning values compatible with poor or useful discrimination. The records came from one hospital, and the outcome was its recorded CKD level rather than a protocol-verified diagnosis. The researchers lacked paired creatinine, eGFR or urine-albumin results and could not confirm the three-month chronicity criterion. Age was the only variable that separated groups after correction for multiple comparisons.
This is therefore stronger evidence about evaluation design than about kidney screening. It shows, on a concrete African dataset, why synthetic oversampling must stay inside the training process and why a live demonstration is not clinical validation. What would change the assessment is prospective, multicentre testing with prespecified thresholds, verified CKD labels, calibration, subgroup results and patient follow-up. Until then, the practical use is to prompt standard testing—not to rule kidney disease in or out.[1]