Can a skin-lesion AI survive the move from dermoscopes to smartphone photos?
A four-class model’s macro-F1 fell from 74.27% internally to 54.92% on 1,376 unseen smartphone images. Its safety gate retained only 11.5% of external cases, exposing the cost of reliable abstention.
The short version
- What happened
- The researchers developed the model on 11,259 eligible dermoscopy images and kept 1,376 smartphone-acquired PAD-UFES-20 images completely outside training and threshold setting.
- Who it affects
- People photographing a suspicious lesion need a tool that recognises when its answer is unreliable; frequent abstention also means services must have capacity for timely specialist review.
- What remains unproven
- Retrospective public datasets, four harmonised diagnoses, compound acquisition and prevalence shift, incomplete subgroup evidence and no prospective clinician or patient outcomes.
Keep in mind
Retrospective public datasets, four harmonised diagnoses, compound acquisition and prevalence shift, incomplete subgroup evidence and no prospective clinician or patient outcomes.
Source analysis · 1 linked source. Primary analysed source date: 16 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
External images exposed a large reliability gap
A skin-lesion classifier that looked reasonably strong on dermoscopy images lost substantial performance when moved to unseen smartphone photographs. In a peer-reviewed four-class study, mean macro-F1 fell from 74.27% on the source-domain test sets to 54.92% on 1,376 PAD-UFES-20 images. The result matters because a model used outside specialist imaging must survive changes in cameras, lighting, backgrounds and patient mix—not just repeat a laboratory benchmark.
Researchers developed the system on 11,259 eligible images from HAM10000 and Derm7pt, covering basal cell carcinoma, melanoma, melanocytic naevus and a harmonised benign-keratosis category. They separated data by lesion or clinical case and reserved distinct partitions for training, model selection, calibration, reliability thresholds and internal testing. The smartphone dataset and a separate Stanford DDI audit did not influence training or threshold choice.[1]
The average concealed a sharp melanoma failure
Performance did not fall evenly. Melanoma F1 decreased from 58.21% internally to 27.38% externally, while naevus F1 fell from 89.65% to 60.80%. Basal-cell-carcinoma F1 was broadly preserved, but its sensitivity dropped from 78.88% to 64.66%. Macro-AUROC declined much less—from 0.926 to 0.880—showing why a ranking metric alone can look reassuring while thresholded clinical classifications deteriorate.
The external shift was not simply ‘dermoscope versus phone’. Nearly 70% of source images were naevi, whereas 61.4% of the smartphone cohort were basal cell carcinomas. Devices, clinical setting, dataset construction and diagnosis prevalence all changed together. The authors correctly describe a compound transfer test, not an isolated measurement of camera effects.
Their selective-decision gate adds another useful warning. It retained 32.37% of internal cases, with 97.86% accepted-case accuracy, but only 11.50% externally, with 92.97% accuracy. Refusing uncertain cases protected performance, yet almost nine in ten external images were not retained for an unrestricted prediction. A system that is safer because it abstains frequently may still create a large specialist-review workload.[1]
The distinction that matters
What changed when the model left its source domain
The external test changed acquisition and diagnosis mix while keeping the trained model and thresholds fixed.
Source test
Dermoscopy-derived internal evaluation
Macro-F1 74.27%; 32.37% retained
External test
1,376 unseen smartphone images
Macro-F1 54.92%; 11.50% retained
Clinical gap
Defer uncertain cases to specialists
Workload and patient outcomes were not tested
A heatmap is not a clinical explanation
For dermatology teams, the practical lesson is to validate the whole decision pathway: who captures the image, which lesions enter the system, when it abstains and how quickly a clinician can review the remainder. The paper’s Grad-CAM and Integrated Gradients audit describes explanation stability, but it does not show that highlighted pixels correspond to clinically causal features or improve a clinician’s decision.
The evidence check remains retrospective and dataset-based. No clinician used the tool prospectively, no patient outcome changed, and performance reflects four harmonised categories rather than the full range of skin disease. The model also relied on clinical metadata, whose availability and quality may differ in practice. The paper reports performance across five random seeds, but that repeated training does not substitute for testing a locked system in a real clinical pathway. Confidence would rise with prospective multi-device studies, prespecified skin-tone subgroup recruitment, comparison against clinicians, measured referral workload and patient-level outcomes. Until then, the study supports specialist-supervised triage and explicit abstention—not autonomous smartphone diagnosis.[1]