A melanoma model reached 95.7% sensitivity—but its test split was not patient-independent
The hybrid model missed 29 of 679 melanoma images in a locked test set. Related images may still cross partitions, and no independent cohort was tested.
The short version
- What happened
- The study used 25,331 ISIC 2019 images in a fixed 70:15:15 image-level split; the locked test set contained 3,801 images, including 679 melanoma images.
- Who it affects
- A person with a suspicious lesion needs a pathway that performs reliably across clinics, devices and skin tones. This study measured images from one public collection and did not test referral decisions or patient outcomes.
- What remains unproven
- 25,331 images from ISIC 2019
Keep in mind
25,331 images from ISIC 2019
3,801-image test set
Image-level split
Patient and lesion independence unverified
No external cohort
No calibration or subgroup analysis
No clinical outcomes
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
A strong image result with a clinically important miss count
A hybrid image model correctly identified 650 of 679 melanoma images in its held-out test set, according to a peer-reviewed Frontiers in Artificial Intelligence paper published on 29 October. That corresponds to 95.73% sensitivity, while 29 melanoma images were missed. The model also produced 26 false positives among 3,122 non-melanoma images, with reported specificity of 99.17%, balanced accuracy of 97.45% and ROC-AUC of 0.98.
Those figures describe classification of dermoscopic images, not diagnosis of people. The negative class pooled seven other ISIC categories, including basal-cell carcinoma, squamous-cell carcinoma and actinic keratosis. It therefore means non-melanoma, not benign. A false negative could delay further assessment, while a false positive could prompt avoidable referral or biopsy; the paper did not test either consequence in clinical care.[1]
The test set was locked, but independence is unresolved
Researchers used all 25,331 images in the ISIC 2019 training collection and made a fixed stratified image-level split: 17,731 for training, 3,799 for validation and 3,801 for final testing. Training-only augmentation and validation-only model selection reduce obvious test leakage. The system combined fixed U-Net lesion localization, EfficientNet-B0 for local boundary features, a Swin Transformer for global context and a feature-optimization block before binary classification.
The key qualification is the unit of splitting. The retained experiment did not provide a verifiable patient or lesion identifier, and the authors could not complete a duplicate or near-duplicate audit. Images of the same lesion or patient may therefore have crossed partitions, making the test task easier than a genuinely independent clinical evaluation. HAM10000 was not used as external evidence because it overlaps with ISIC 2019.[1]
The distinction that matters
What the locked test set actually contained
The confusion matrix exposes both the headline performance and the errors hidden by accuracy.
Melanoma images
679 in the test set
650 found; 29 missed
Other images
3,122 in the test set
3,096 correct; 26 false alarms
Independence
Image-level split
Patient/lesion separation unverified
What would make the result clinically persuasive
The paper provides no independent external validation, source-diagnosis subgroup errors, probability calibration, precision–recall curve, patient-level confidence intervals or formal case-level comparison with another model. Grad-CAM++ visualizations were qualitative and were not assessed by dermatologists. The fixed U-Net preprocessing component also lacked retained training provenance and independent segmentation benchmarking. The authors reported no funding and no commercial or financial conflicts.
A stronger assessment would test a frozen model on a non-overlapping cohort from different clinics, cameras and skin-tone groups, using patient-grouped partitions and clinician-confirmed outcomes. Calibration and decision-curve analysis should show whether scores support a safe referral threshold. Until then, the model is a promising benchmark pipeline, not evidence that it can screen patients or reduce missed melanoma in practice.[1]