AI lecture summaries raised quiz scores by 0.28 points—did students learn more?
In an unreviewed year-long study, 120 US medical students were randomized to receive AI-generated lecture summaries or take the same quizzes without them. Scores leaned towards the summary group, but the individual-level difference was small and statistically uncertain.
The short version
- What happened
- The single-school study enrolled 137 third-year medical students; 120 submitted at least one analysed quiz, producing 317 responses across 11 neurology clerkship cohorts.
- Who it affects
- Medical students may value a concise, checked summary when clinical rotations leave little study time. This study cannot tell a school that summaries improve durable learning, and it gives no evidence that an unreviewed AI output should replace faculty teaching or source material.
- What remains unproven
- This is an unreviewed preprint from one US medical school and one neurology clerkship.
Keep in mind
This is an unreviewed preprint from one US medical school and one neurology clerkship.
The principal student-level effect was small and statistically uncertain; the cohort-direction test was one-sided and its two-sided value was just above 0.05.
Repeated quiz responses were clustered within students, the quizzes had a strong ceiling effect and no delayed-retention outcome was measured.
Survey results were self-reported, used varying denominators and offered no neutral response option.
Two authors founded a medical-education product company, although the paper states it did not fund or participate in the study.
Source analysis · 1 linked source. Primary analysed source date: 30 October 2026.
Read the full report · 5 min The evidence, methods, implications and limitations
The clearest result is a small, uncertain score difference
A year-long study at one US medical school found that third-year neurology students offered AI-generated lecture summaries scored, on average, 8.14 out of 10 on weekly quizzes. Students sent the same quiz without the summary averaged 7.86. That 0.28-point difference favoured the summary group, but its 95% confidence interval ran from 0.23 points lower to 0.80 higher, and the individual-level comparison was not statistically significant (P = 0.28). The study therefore does not establish that the summaries improved learning, despite a consistent-looking pattern across rotations.
The researchers enrolled 137 students across 11 sequential clerkship cohorts from January 2025 to January 2026. Seventeen submitted no quiz; 120 students contributed 317 analysed responses, with 62 assigned to receive summaries and 58 to receive no summary. Assignment happened within each cohort before the first quiz. Both groups answered the same ten questions, while the intervention group received a one-page summary generated from that week’s recorded Neurology Grand Rounds lecture. Participation was voluntary, scores did not count towards grades, and completing a quiz counted towards attendance.[1]
A favourable cohort pattern does not resolve how much students learned
The summary arm scored higher in nine of 11 cohorts. A pre-specified one-sided randomization test on that direction produced P = 0.026; the two-sided value was 0.053. But counting direction gives a small cohort the same weight as a large one. When the analysis weighted differences by response count, the advantage was 0.22 points and did not reach significance (one-sided P = 0.21). Repeated quizzes also clustered within students: the authors estimated that 317 responses provided precision closer to 195 independent observations. Their mixed-effects estimate remained 0.27 points, with a confidence interval crossing zero.
The quiz design made a difference harder to interpret. Nearly two-thirds of responses were already eight or above, 17% were perfect scores, and most questions tested recall rather than clinical application. Sixty-three students answered the end-of-cohort survey, with changing denominators by item and no neutral response option. Among those reporting receipt of a summary, 26 of 33 said it made them more likely to review the lecture and 14 of 17 said it reduced the time needed to understand it. Those perceptions may matter to busy trainees, but they are self-reports rather than evidence of durable knowledge or better patient care.[1]
The distinction that matters
What the study’s three signals actually show
All figures come from the same unreviewed, single-school intervention; they answer different questions and should not be combined as if they were one result.
Student averages
62 summary vs 58 no-summary students
+0.28 of 10; 95% CI -0.23 to +0.80; P = 0.28
Cohort direction
11 rotations compared by which arm scored higher
9 of 11 favoured summaries; two-sided P = 0.053
Student reports
63 survey respondents, with item-specific denominators
26/33 reviewed more; 14/17 reported faster understanding
Calibration was careful, but the evidence remains preliminary
Before the classroom phase, the team compared six prompts for summaries and six for questions across two models, generating 2,692 rubric annotations. Claude 3.5 Sonnet with one prompt produced the best summaries, while GPT-4 with another produced the best questions; no model led both tasks. Outputs were checked against lecture transcripts and reviewed for relevance before release. This is a useful design lesson: institutions should test prompts and content types rather than assume a preferred model works equally well for every educational task.
The paper is a medRxiv preprint and has not been peer reviewed. It comes from one school, one specialty and short quizzes delivered near the lectures; it did not test delayed recall, clinical reasoning, examination results or patient outcomes. Detecting the observed student-level effect with conventional power would have required roughly 394 students per arm, far more than the 62 and 58 analysed. Two authors founded Dendro Education, which develops medical-education products; the paper says the company did not fund or participate in the study, and the study received no external funding. A stronger assessment would need an independently run, preregistered, multi-school trial using harder application questions and delayed testing, with a fixed primary outcome and enough students to estimate both benefit and harm precisely.[1]