Do AI research agents turn mixed evidence into a cleaner story than the sources support?
Across seven systems and four research benchmarks, at least 22% of scored claims in every system-dataset combination were judged materially over-smoothed. The new metric is itself partly LLM-based and human agreement was imperfect.
The short version
- What happened
- Researchers scored reports from seven systems across four datasets and found materially over-smoothed claims in every system-dataset combination.
- Who it affects
- A clinician, civil servant, researcher or manager may receive a polished cited answer that conceals disagreement or narrow study conditions, making source inspection essential for consequential decisions.
- What remains unproven
- Unreviewed preprint
Keep in mind
Unreviewed preprint
Partly LLM-generated evidence extraction and judging
Small human-validation subset
Subjective five-point rubric
Curated research benchmarks rather than observed workplace decisions
Commercial model versions can change
Source analysis · 1 linked source. Primary analysed source date: 26 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
A citation can be relevant while the conclusion is still too strong
A new unreviewed NYU study finds that AI research agents often write decisive conclusions from evidence that is narrower, conditional or internally mixed. The authors call this claim smoothing: cited documents may support pieces of a statement, while the synthesis drops qualifications, expands its scope or implies that separately tested ideas work together. For a reader, the danger is subtler than a fabricated citation because every link can look plausible.
The team evaluated seven systems—including proprietary web-search agents, Gemini Deep Research and two open research systems—on four report-generation datasets. SmoothScore extracts each report’s main conclusions, searches the cited documents for supporting and sceptical evidence, groups that evidence by aspect and assigns a one-to-five score. Across every system-dataset combination, at least 22% of claims scored three or higher, the study’s threshold for material smoothing. Rates and severity varied substantially by system.[1]
Reading the underlying documents helped only when the prompt demanded calibration
Trace data exposed a practical mechanism. On one benchmark, one agent opened 95 of 496 cited pages and another opened 34 of 1,458, often relying on short search snippets. The researchers regenerated reports for 61 questions with the full cited documents, an instruction to preserve scope and uncertainty, or both. Full documents and calibration instructions together reduced materially smoothed claims from 29% to 12% for one system and from 61% to 25% for the other. Either intervention alone helped less.
That result does not show how often people are misled in ordinary use. The experiments use curated research tasks, changing commercial systems and an evaluator designed by the same research team. Report quality was not assessed by domain experts for every claim. Funding included public and foundation grants, an Amazon grant and NVIDIA compute support; the paper does not identify these organizations as controlling the analysis.[1]
The distinction that matters
How a supported claim can still overreach
SmoothScore asks whether the scope and strength of a synthesis match the whole cited evidence set.
Support
Some sources back parts of the claim
Citation check may pass
Counter-evidence
Other sources narrow or qualify it
Scope should shrink
Smoothing
Report drops those boundaries
Conclusion sounds stronger
The measurement is promising, but still subjective
Two people independently scored 73 claims. They agreed exactly 58% of the time and within one point 96% of the time. The automated judge agreed exactly with one annotator on 56% and within one point on 93%; among claims the judge flagged as materially smoothed, that annotator agreed on 62%. This supports using the score as a screening tool, not as an objective truth about every sentence or a definitive ranking of providers.
For researchers, journalists and public-service teams, the immediate lesson is procedural: require the agent to open central sources, actively retrieve counter-evidence and state where results stop generalising. A robust next test would use blinded domain experts, live professional tasks and preregistered scoring across repeated model versions. Until then, fluency, citation density and a confident summary should not substitute for checking the underlying evidence.[1]