Do genome AI models predict harmful variants this well? Mixing variant types can inflate the score
A benchmark of 242,132 ClinVar variants found that aggregate accuracy can look exceptional even when a model is much weaker within clinically relevant variant categories. No single model led across every context.
The short version
- What happened
- Researchers tested 15 DNA and protein models, plus a conservation baseline, on 242,132 ClinVar single-nucleotide variants with definitive pathogenic or benign labels.
- Who it affects
- For patients and families, a very high benchmark score can sound like near-certain interpretation of a genetic result. This study shows why laboratories need performance for the exact variant class under review, followed by clinical evidence, before treating a pooled score as diagnostic capability.
- What remains unproven
- Retrospective benchmark
Keep in mind
Retrospective benchmark
242,132 ClinVar single-nucleotide variants
15 AI models plus PhyloP
13 variant categories
No prospective patients
No diagnostic or health outcome
No indels or structural variants
Some very small subgroups
Possible limited AlphaMissense tuning overlap
Model-score extraction choices can affect results
Source analysis · 1 linked source. Primary analysed source date: 30 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
A high overall score can partly measure the mix of variants
The central finding is a warning about how genome AI is evaluated. When pathogenic and benign examples come from variant categories with very different disease-label rates, a model can achieve an impressive aggregate score by recognising the category rather than separating harmful from harmless variants within that category. In a new peer-reviewed benchmark, Evo2-7B reached an AUROC of 0.979 across all noncoding variants, yet scored 0.892 for non-splice introns and 0.697 for canonical splice sites considered separately.
The authors made the problem visible by combining two categories. Among 10,174 splice variants, 99.3% were labelled pathogenic and Evo2-7B achieved an AUROC of 0.714. Among 2,386 variants in 5′ untranslated regions, only 1.3% were pathogenic and the score was 0.808. Merging the groups lifted AUROC to 0.977 even though performance within neither group approached that figure. A trivial rule based only on each variant type's pathogenic prevalence also performed strongly on the mixed dataset.[1]
The benchmark found specialists, not one universal winner
The study used 242,132 single-nucleotide variants from the February 2026 ClinVar release, restricted to definitive pathogenic or benign classifications. It compared nine DNA models, six protein models and the PhyloP evolutionary-conservation baseline across 13 functional categories ranging from 62 to 115,832 variants. No model led everywhere. AlphaMissense was strongest for missense variants at 0.966 AUROC, while AlphaGenome reached 0.968 for non-splice introns and 0.936 for synonymous variants. For splice variants, GPN-Star reached 0.818 while AlphaGenome recorded 0.618.
That context dependence matters for clinical laboratory scientists and people waiting for an explanation of a genetic finding. A tool selected from a pooled leaderboard may not be the best tool for the specific variant under review. The result does not show that these systems lack value; it shows why the denominator and subgroup must match the intended decision. The authors released the benchmark data and code so model developers can report the same disaggregated comparisons.[1]
The distinction that matters
How mixing two variant types changed the same model's score
Evo2-7B's AUROC rose sharply when splice and 5′ UTR variants were pooled, although discrimination within each category was much lower.
Splice variants
AUROC 0.714
10,174 variants; 99.3% pathogenic
5′ UTR variants
AUROC 0.808
2,386 variants; 1.3% pathogenic
Both groups merged
AUROC 0.977
12,560 variants with very different label rates
This was a benchmark, not evidence of better diagnoses
The team repeated analyses after balancing classes within genomic regions, limiting ClinVar records by review status, using precision-recall curves and comparing rankings with COSMIC cancer variants. The broad pattern held. But ClinVar labels are not a prospective clinical population, and some categories were tiny: the stop-loss group contained just 62 variants. The analysis covered single-nucleotide substitutions, not insertions, deletions or structural variants, and it could not test categories that lack enough definitive ClinVar labels.
Score extraction and model design also matter. AlphaGenome showed instability in some settings, and the authors noted possible limited tuning overlap for AlphaMissense, although its performance was similar on variants submitted after 2023. No patient management, diagnostic yield or health outcome was measured. Independent prospective tests on the exact variant classes used in clinical services—and evidence that predictions improve expert interpretation without widening ancestry or laboratory biases—would change the assessment. The study reported no external funding and no competing interests.[1]