Did training AI for diversity produce better science ideas?
An 8B model scored well on an automated idea-portfolio metric and matched a frontier model in a four-expert pilot. That is not evidence that the ideas became successful research.
The short version
- What happened
- The researchers trained an 8B model on 7,172 document-grounded instances and evaluated 1,740 contamination-controlled cases across 16 domains.
- Who it affects
- Scientists could receive a broader shortlist of possible directions, but may spend more time filtering plausible-looking ideas and checking source grounding.
- What remains unproven
- The study is an unreviewed, commercially affiliated preprint. Its primary metric relies on model judges, the expert pilot had four NLP researchers, and proof-of-concept outcomes came from one coding agent’s self-reports.
Keep in mind
The study is an unreviewed, commercially affiliated preprint. Its primary metric relies on model judges, the expert pilot had four NLP researchers, and proof-of-concept outcomes came from one coding agent’s self-reports.
Source analysis · 1 linked source. Primary analysed source date: 15 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The training target rewarded a portfolio, not one polished proposal
A Salesforce-led team trained an 8-billion-parameter model to produce three scientific ideas that were individually promising and deliberately different from one another. On the authors’ automated ‘creative utility’ measure, its strongest version scored 0.64, ahead of 0.38 for Claude Opus 4.8 and 0.28 for GPT-6 Astra. That comparison shows the model learned the study’s portfolio objective; it does not show that its ideas became successful science.
The system received summaries of related papers and wrote structured idea cards covering a gap, claim, method and inspirations. During reinforcement learning, an idea earned less credit when it resembled an earlier idea in the same set. The model was trained on 7,172 instances and tested on 1,740 contamination-controlled cases drawn from 2026 papers across 16 domains.[1]
Automated scores and expert judgement told a more cautious story
Creative utility combined an automated judgement of quality against a known published ‘gold’ idea with model-judged distance between proposals. That makes the evaluation reproducible at scale, but it also rewards resemblance to what research later proved publishable and depends on other models’ preferences. A separate IdeaSeed check found every model’s population of methods less diverse than human proposals. The strongest no-thinking version scored 0.65 in normalised method entropy, against 0.87 for people.
The human pilot included four NLP experts, each with a PhD and at least four top-tier papers. They blind-rated 60 portfolios of three ideas across the systems. The trained model averaged 2.17 promising ideas and 2.83 unique ideas per portfolio; Claude averaged 2.20 and 2.72. That is encouraging, but four experts in one field cannot establish broad scientific value. The same team also handed 132 ideas to one coding agent for 90-minute, US$30 proof-of-concept runs. At least one idea received a self-reported supporting result in 33% of three-idea sets for the trained model versus 27% for Claude; most sets therefore produced no such support.[1]
The distinction that matters
What the diversity reward changed
Each new idea received credit for quality and for how far it moved from ideas already in the set.
Ground
Read summaries of related papers
Generate an idea tied to named sources
Compare
Judge quality against a published reference idea
Estimate whether the direction looks worthwhile
Diversify
Discount ideas similar to earlier ones
Reward a broader three-idea portfolio
Researchers still have to choose what deserves time
For research teams, generating a deliberately varied shortlist could reduce the tendency to repeat one fashionable direction. It could also create more low-cost proposals to inspect, shifting effort toward screening assumptions, novelty and feasibility. None of the tests measured whether an idea survived peer review, reproduced, helped another scientist or produced long-term impact.
Commercial interest is also material: four of six authors are affiliated with Salesforce AI Research and another completed the work there. The preprint is under review and reports no separate funding or competing-interests statement. A convincing next step would preregister idea selection, recruit independent experts across fields, fund comparable human and model proposals, and track blinded review, replication and research value over time. Until then, the study supports a narrower conclusion: set-level training can make model-generated idea portfolios look more varied without an obvious loss on small expert and coding-agent checks, not that it can automate scientific judgement.[1]