Do learning agents improve results? A 32-person study cannot isolate the agent effect
An unreviewed China–Singapore study found larger short-term assessment gains with an agent-organised learning programme than a comparison tutor. Different materials, practice and assessment designs mean the result does not show that multi-agent coordination caused the difference.
The short version
- What happened
- Thirty-two learners aged 18–30 completed a roughly 54-hour comparison: 16 used Agentic University and 16 used the GenMentor system.
- Who it affects
- Students may value a coherent path and visible feedback, but schools should not infer that more agents automatically improve learning. Teachers need inspectable materials, correction routes and evidence that gains persist without assistance.
- What remains unproven
- Unreviewed preprint
Keep in mind
Unreviewed preprint
32 learners
16 per group
Short roughly 54-hour study
One assigned career goal
Online recruitment
Five allocation schemes considered
Researcher-built agent-generated assessment
Different content and practice formats
No component ablation
No long-term retention
Model-judged synthetic studies
No funding or competing-interests statement surfaced
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
Learners gained more in one short, assigned course
The Agentic University system lets specialised agents and a learner maintain a revisable ‘Capability Program’ linking goals, materials, assessments and feedback. Forty people registered online; after non-participation, ineligibility and two pre-specified high self-rating exclusions, 32 learners aged 18–30 remained. Sixteen used Agentic University and 16 used GenMentor for an assigned goal: becoming an AI video-generation product manager. Everyone completed a roughly 54-hour window. Mean assessment scores rose from 21.88 to 34.06 with Agentic University and from 24.73 to 30.33 with GenMentor.[1]
The difference is encouraging but fragile
The mean gain difference was 6.58 points on a 50-point scale, with a Mann–Whitney comparison of U=184.5, raw p=0.035 and rank-biserial correlation 0.44. Only perceived cognitive support remained significant after correction among four experience measures: 78.13 versus 65.63. Self-efficacy and metacognition gains were numerically higher but not statistically distinguishable. Allocation was not a simple preregistered random draw: the team generated five schemes and chose one after considering balance across five baseline indicators. The test questions and answer options were also generated by an agent from online interview scenarios.[1]
The distinction that matters
What changed between the two learning packages
The study compared complete systems, so several possible explanations move together.
Participants
16 per group
All completed
Agentic University
Videos, projects, simulation, replanning
+12.19 points
GenMentor
Text, quizzes, Replan button
+5.61 points
Causal component
No matched ablation
Not isolated
The comparison changed several things at once
Agentic University supplied videos, quizzes, written microprojects, a dialogue simulation and evidence-guided replanning; GenMentor used text materials, quizzes and a Replan button. That complete-system comparison may help decide what package to study next, but it cannot show that the multi-agent organisation itself improved learning. The sample was small, mostly university-focused and followed one researcher-assigned career goal for less than three days. The paper is also an unreviewed preprint, and its first two evaluation studies relied heavily on model judges. A stronger test would preregister allocation and outcomes, match learning time and content, use independently validated assessments, compare components separately and measure unaided retention weeks or months later.[1]