Did AI editing help readers understand Chinese texts? Two experiments found no comprehension gain
Across 992 participants, target-guided rewrites did not improve author-keyed comprehension. One small alignment result was exploratory and tied to the authors' own scoring targets.
The short version
- What happened
- Study 1 randomly allocated 572 Chinese readers across original and GPT-4, Claude 3 Opus or Gemini Advanced versions; Study 2 allocated 420 readers across original, GPT-4 and Claude versions.
- Who it affects
- Writers, editors and readers should not assume that a polished AI rewrite is easier to understand or more faithful simply because it sounds smoother.
- What remains unproven
- Fixed Chinese texts and predominantly female, highly educated platform samples
Keep in mind
Fixed Chinese texts and predominantly female, highly educated platform samples
Study 1 sample extension followed inspection of interim results
Targets and answer keys guided both the rewrites and outcome scoring
No human-editor comparison
Author-written comprehension items were not independently validated
Product labels lacked model snapshots and sampling settings
Source analysis · 1 linked source. Primary analysed source date: 26 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
Target-guided rewrites changed one narrow signal, not comprehension
A peer-reviewed study tested whether AI-refined Chinese writing reached readers differently from the original text. Study 1 randomly assigned 572 participants—143 per condition—to original writing or versions refined by GPT-4, Claude 3 Opus or Gemini Advanced. Study 2 assigned 420 participants—140 per condition—to original academic excerpts or GPT-4 and Claude rewrites. Readers answered the same questions, and the models had been given the authors' intended ratings or answer-key information while revising.
GPT-4 produced a small improvement in exact-match emotional alignment in Study 1 (p = .002; signed r = .18). Gemini's unadjusted decline did not meet the corrected threshold, and Claude did not differ from the originals. More importantly, none of the systems produced a detectable comprehension difference. In Study 2, neither GPT-4 nor Claude changed author-keyed comprehension or the six designated scholarly attributes.[1]
The experiment does not settle whether ordinary AI copyediting helps
The design was more directed than everyday proofreading: models received the same target ratings or answer keys later used to judge readers. That can make a rewrite unusually sensitive to the outcome and creates a risk of target leakage. There was no human-editor comparison, so any difference cannot be attributed uniquely to AI. The comprehension questions were author-written rather than independently validated, and the participants were general readers rather than subject experts or peer reviewers.
Study 1 recruitment was expanded after interim results were inspected, so its final p-values are exploratory rather than confirmatory. The products were recorded only as GPT-4, Claude 3 Opus and Gemini Advanced, without model snapshots or sampling settings, limiting reproducibility. For writers and editors, the practical conclusion is modest: a target-guided rewrite did not reliably improve comprehension of these fixed texts. Independent, preregistered tests should compare AI and human editing, preserve version settings and assess meaning beyond answer-key matches.[1]
The distinction that matters
What changed across the two reader experiments
The experiments separated a narrow alignment score from whether readers recovered selected information.
Study 1 · 572 readers
Original plus three AI-refined conditions
One exploratory GPT-4 alignment gain; no comprehension difference
Study 2 · 420 readers
Original plus two AI-refined conditions
No detected alignment or comprehension difference
Human editing
No comparison condition
AI-specific effect unresolved