AI lifted proof homework scores—but the advantage disappeared in the exam
In a 153-student course, earlier access to two AI tools improved supported homework by 2.56 points but produced no detectable gain on the next AI-free exam. Later chatbot use was observational, so it cannot establish that the chatbot caused lower scores.
The short version
- What happened
- Researchers randomly assigned 153 students in one undergraduate proof course to receive an AI chatbot and proof-feedback tool earlier or later.
- Who it affects
- A student can submit stronger AI-assisted work without becoming better prepared to solve a new problem alone. Teachers need separate evidence of independent understanding before treating higher homework marks as learning.
- What remains unproven
- The study is a preprint and has not been peer reviewed.
Keep in mind
The study is a preprint and has not been peer reviewed.
It covers one proof-oriented university course; 153 of 308 enrolled students consented, and the two AI interfaces were introduced together.
Later tool use was self-selected, so the association between message volume and exam scores is not a causal estimate.
The study ended before the final exam because of budget limits and did not measure cognitive effort directly.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
Better homework did not become a measurable exam gain
Students were split into early- and later-access groups in an ordinary Carnegie Mellon proof course. The tools arrived together: an open-ended GPT-4o chatbot and an interface that required a written proof before giving feedback. Across two homework assignments, early access produced a small adjusted gain. On the next midterm, completed without AI, the groups did not differ significantly.[1]
The later usage result needs a causal warning
Once all students could use both tools, every additional 100 chatbot messages was associated with 3.02 fewer points on the following exam after adjustment for prior performance and self-efficacy. Students with lower self-efficacy also used the tools more. Because students chose how much help to request, unmeasured struggle, outside assistance or study time could explain part of the pattern; cognitive offloading was not measured directly.[1]
The distinction that matters
What the study can and cannot attribute
Randomisation applied to access timing, not to each tool or to later usage volume.
Randomised
Earlier versus later access to both tools
+2.56 homework points; no detected exam gain
Observed
Self-selected chatbot message count
Associated with lower later scores
Unproven
Cognitive offloading mechanism
Plausible interpretation, not a measured causal effect
Teachers need to test what students can do alone
The practical distinction is between help that critiques a learner's attempt and help that supplies missing reasoning. Schools evaluating an AI tutor should compare supported work with a later task completed independently, and should not use chat counts as a disciplinary signal. A stronger test would randomise students separately to answer-giving chat, hint-only chat and feedback on an attempted solution.[1]