Why did ‘uncertain’ AI choices fail to beat random sampling?
A peer-reviewed materials study found that several uncertainty-driven active-learning strategies often matched or trailed random selection on deliberately unfamiliar chemistry. Calibration changed confidence estimates, but usually did not reveal the information the models needed.
The short version
- What happened
- The experiments used 1,128 measured-solubility compounds and JARVIS-22 materials data, withholding fluorine- or iron-containing chemistries to create deliberately unfamiliar tests.
- Who it affects
- Materials scientists and drug-discovery teams use active learning to decide which costly experiment comes next. A poorly chosen uncertainty signal can spend that budget without buying the expected improvement.
- What remains unproven
- The paper evaluates three regression tasks and constructed distribution shifts; other scientific domains and acquisition objectives may behave differently.
Keep in mind
The paper evaluates three regression tasks and constructed distribution shifts; other scientific domains and acquisition objectives may behave differently.
Several comparisons have overlapping error bars, so the evidence supports ‘often no better’ rather than a universal ranking of methods.
The study is computational and retrospective. It does not measure time, cost or discovery success in a live laboratory.
The authors report Schmidt AI in Science and Canada First Research Excellence Fund support and declare no competing interests; one author lists an affiliation with Artificial, Inc.
Multi-source analysis · 2 linked sources. Primary analysed source date: 24 October 2026. Earliest linked supporting source: 21 November 2025.
Read the full report · 3 min The evidence, methods, implications and limitations
Calibration fixed the scale, not the search problem
Active learning is meant to save experiments: a model chooses the next sample expected to teach it most. In this peer-reviewed study, researchers compared uncertainty estimates from neural-network ensembles, loss-landscape sampling, random forests and XGBoost. They tested aqueous solubility on 1,128 ESOL compounds and two JARVIS-22 materials properties—bandgap and formation energy—while withholding fluorine- or iron-containing materials to mimic unfamiliar chemistry.
For the large materials partitions, more than 56,000 in-distribution samples trained the models. The out-of-distribution test sets ranged from 163 to 1,087 materials depending on task and split. Calibration could make confidence intervals look statistically better aligned, yet uncertainty-maximising acquisition often matched or trailed random sampling. In the fluorine formation-energy task, loss-landscape uncertainty did not beat random selection until roughly half of the held-out training pool had been added; error bars overlapped in several comparisons.[1][2]
A warning for expensive discovery loops
A laboratory could waste scarce synthesis or assay capacity if a model’s widest error bars do not point to the samples that close its real knowledge gaps. The paper’s practical recommendation is to study uncertainty in the feature-input space and consider tools such as counterfactual explanations when data are genuinely out of distribution.
The result is bounded: it uses three prediction tasks and synthetic hold-outs based on chemistry membership, not prospective autonomous laboratories. Confidence would rise with preregistered, repeated campaigns in which real experiments—not database lookups—test whether alternative acquisition rules reach a target performance with fewer samples.[1][2]
The distinction that matters
What the study actually compared
The same loop was tested with different signals for choosing the next unfamiliar sample.
Train
Fit models without fluorine- or iron-containing materials
Creates a deliberate distribution shift
Choose
Add samples by highest estimated uncertainty or at random
Tests whether confidence spread finds useful missing information
Measure
Track prediction error on held-out chemistry
Random selection often matched or beat uncertainty maximisation