Did AI find cultural health differences—or just data leakage and sample-size noise?
A peer-reviewed reanalysis of 482 Indian and 70 Japanese students found a perfect obesity classifier collapsed to chance after BMI was removed. The study is a methods warning, not a national comparison.
The short version
- What happened
- The final convenience samples contained 482 students from one Indian university and 70 from one Japanese university, after eligibility and exclusion rules.
- Who it affects
- Students and patients can be misrepresented when unstable model explanations are labelled as cultural traits; careful validation helps prevent weak evidence from shaping screening or policy.
- What remains unproven
- Self-reported data, one university per country, n=70 in Japan, inconsistent instruments and no national sampling or formal measurement-invariance test constrain the conclusions.
Keep in mind
Self-reported data, one university per country, n=70 in Japan, inconsistent instruments and no national sampling or formal measurement-invariance test constrain the conclusions.
Source analysis · 1 linked source. Primary analysed source date: 16 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The perfect obesity model had been given the answer
A machine-learning model appeared to classify student obesity perfectly until researchers removed the variable used to define obesity. In a peer-reviewed reanalysis of university cohorts in India and Japan, discrimination fell from an area under the curve of 1.00 to 0.50–0.57 after body mass index was excluded as a predictor. That is essentially chance performance and a textbook example of target leakage.
The dataset came from surveys conducted in June–August 2024. Of 1,238 initial responses, age and health exclusions left 482 students at one Indian university and 70 at one Japanese university. The teams tested penalised logistic regression, random forests and gradient-boosted trees with held-out evaluation, permutation importance and bootstrap intervals. The paper is best read as an audit of methods, not a map of two national cultures.[1]
Unequal samples can manufacture a cultural story
The researchers also revisited claims that different lifestyle features mattered in the two countries. When they repeatedly reduced the Indian cohort to match the Japanese sample of 70, every feature-importance value observed in Japan fell within the range produced by sample-size variation alone. That does not prove the cohorts are identical; it means this dataset cannot distinguish a cultural signal from instability created by the eightfold size difference.
The only remaining modelled outcome was elevated self-rated stress. Held-out AUC ranged from 0.49 to 0.71, from chance to modest discrimination. The stress measure was a single 0–10 item rather than the validated ten-item scale described in the original plan. Diabetes and hypertension could not support modelling after the stated exclusions were correctly applied: eligible positive cases disappeared, and hypertension was not asked in Japan.
Those details show why an attractive cross-country narrative can survive several layers of weak evidence. A model may fit circular labels, an importance ranking may move with sample size, and inconsistent survey instruments may be mistaken for population difference. None of those problems is repaired by choosing a more complex algorithm.[1]
The distinction that matters
Three checks changed the interpretation
Each check asks whether the model learned a health signal or a property of the dataset.
Leakage
Remove BMI from the obesity predictors
AUC falls from 1.00 to 0.50–0.57
Validation
Score models on held-out data
Stress prediction is modest at best
Sample size
Shrink India to n=70 repeatedly
The apparent country gap fits sampling noise
A defensible comparison starts before model training
For digital-health teams, the practical checklist is concrete: list every variable used to construct the outcome, remove those variables from the predictors, reserve a held-out test set, report uncertainty and repeat the comparison with matched sample sizes. Cross-cultural work should also test whether an instrument means the same thing in each setting before comparing model explanations.
The evidence check cuts both ways. This reanalysis is transparent and includes reproducible code, but it still relies on self-reported measures and convenience samples from one institution per country. The two samples are not representative of India or Japan, and the study performed no formal measurement-invariance test. Confidence in a cultural difference would require harmonised, validated measures; larger multi-site probability samples; preregistered outcomes; external validation; and stable effects after sample-size matching. The study’s contribution is therefore corrective: it shows how much apparent insight can disappear when the leakage is removed.[1]