Do agent-style AI tutors improve learning? Two trials suggest gains—but attrition clouds the result
A Tsinghua preprint found a structured legal tutor beat plain chat and a coding workbench improved expert-rated mastery. The programming analysis included only 54 completers from 140 recruits, and the preregistrations are not linked.
The short version
- What happened
- In a three-arm legal-learning trial, 111 analysed students using the same base model gained 22.8 points with AI Cosmos, 18.5 with a teacher prompt and 15.7 with plain chat; the planned AI Cosmos-versus-chat contrast was seven points.
- Who it affects
- Students may learn differently from systems that require them to explain, build and debug instead of delivering an answer. These trials are too small and selective to justify replacing teachers or buying an agentic tutoring platform on their own.
- What remains unproven
- Unreviewed preprint
Keep in mind
Unreviewed preprint
Preregistrations not linked
Programming analysis only 54 completers from 140 recruits
No intent-to-treat analysis
Per-arm exclusions not reported
Small programming arms
Underlying model version not named
Delayed legal comparison underpowered
No ordinary-classroom deployment
No funding or competing-interests statement
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
Structured interaction improved one immediate legal outcome
Tsinghua University's AI Cosmos team reports two three-arm randomized studies that kept the underlying language model constant while changing how students interacted with it. In the legal study, 131 students were assigned to plain chat, the same model prompted as a teacher, or a structured scenario with role-based dialogue. Minimum-engagement exclusions left 111 participants—37 per arm—for analysis. They completed matched civil- and criminal-law pre-tests, learning tasks and immediate post-tests scored on a 100-point scale.
Average gains rose from 15.7 points with plain chat to 18.5 with the teacher prompt and 22.8 with AI Cosmos. The prespecified Cosmos-versus-chat contrast was seven points, with Cohen's d of 0.56 and p=0.018; an analysis adjusting for baseline gave a 4.5-point difference with the same p-value. The overall test was only marginal, and gains on open responses did not differ. At four weeks, 18 people per arm retained a directionally larger Cosmos gain, but the between-group comparison was inconclusive.[1]
The programming result distinguished test scores from mastery
The second study compared plain chat, a teacher-style assistant and AI Cosmos with an executable coding workbench. Participants practised newly written algorithm problems, then independently attempted two unseen problems one week later. Two problem authors, blind to condition, rescored code using a five-level mastery rubric. Among submissions receiving positive automated-judge credit, 68% in Cosmos met the mastery threshold, against 45% with plain chat and 32% with the teacher baseline. The Cosmos-versus-teacher comparison had p=0.009; the difference from plain chat did not reach significance.
Mean post-test scores were 68.0, 61.5 and 49.8 respectively, but the overall comparison was not statistically significant (p=0.20). The expert review matters because an automated judge can award points to incomplete or brute-force code. Yet it also creates a narrower claim: the study supports a difference in the rubric-rated quality of completers' code more clearly than a general improvement in programming performance.[1]
The distinction that matters
What the two trials can—and cannot—show
Random assignment strengthens the comparisons, but exclusions and attrition mean the reported analyses do not represent every person initially assigned or recruited.
Legal trial
131 assigned
111 analysed
Programming trial
140 recruited
54 completed
Programming score
Three-arm comparison
Not significant
Code mastery
Cosmos vs teacher
Higher among completers
Attrition prevents a confident causal claim about all recruits
The main limitation is participant flow. Of 140 programming recruits, 100 completed the diagnostic and only 54 completed the multi-session protocol; the analysed arms contained 17–19 people. Advancement required full-score submissions on three training problems, a gate the intervention itself could influence. The paper does not report per-arm dropout at each stage or an intent-to-treat analysis, so the groups of completers may differ in ways random assignment no longer protects against. The legal study likewise omits per-arm counts for its 20 engagement exclusions.
The authors describe both studies as preregistered but do not link a registry, and the paper does not name the base-model version or provide funding and competing-interest statements. For teachers and institutions, the useful signal is that interaction design and independent assessment may matter as much as model choice. Confidence would rise with linked protocols, complete participant-flow tables, intent-to-treat estimates, a named model, peer review and replication in ordinary classrooms over longer periods.[1]