Did a Socratic AI raise student engagement?
Across eight course sections at four Chinese universities, students offered EduBot reported larger within-semester engagement gains. Section-level allocation, one course per university and no follow-up beyond the semester limit causal and general claims.
The short version
- What happened
- The study enrolled 583 students in eight sections of four introductory courses; 555 complete cases comprised 278 intervention and 277 control students.
- Who it affects
- Students may gain a private, flexible place to practise questions, but current evidence measures engagement rather than durable learning or grades.
- What remains unproven
- Only eight course sections were allocated, limiting cluster-level causal inference
Keep in mind
Only eight course sections were allocated, limiting cluster-level causal inference
University and discipline were perfectly confounded
Engagement outcomes were self-reported rather than direct learning measures
No follow-up tested whether differences persisted beyond the semester
The intervention bundled several features, preventing component attribution
Source analysis · 1 linked source. Primary analysed source date: 15 October 2026.
Read the full report · 6 min The evidence, methods, implications and limitations
The answer: engagement improved, but learning was not proved
Students in four course sections given a Socratic conversational agent reported stronger engagement trajectories over one semester than students in four parallel sections taught as usual. The largest standardised post-test difference was behavioural engagement at 0.71, followed by agentic engagement at 0.57, cognitive engagement at 0.55 and emotional engagement at 0.34. Group-by-time interactions remained statistically significant after adjustment for the four tested dimensions.
That is encouraging evidence for a structured learning aid, not proof that a chatbot improves achievement. Engagement was measured with student questionnaires, and the study did not establish higher exam marks, deeper knowledge retention or effects after the semester. Only eight sections were allocated. A difference attached to a section may reflect class dynamics or other unmeasured factors, and each university contributed a different subject, making university and discipline impossible to separate.[1]
How the eight-section comparison worked
Four Chinese research universities contributed one introductory course each: psychology at Zhejiang, computer science at Nanjing, business at Wuhan and biology at Fudan. Every course ran in two parallel sections with the same instructor, syllabus, materials, assessments and teaching-assistant support. Students had enrolled through normal timetabling before the study announcement. A course coordinator used a coin toss to assign one section to EduBot and one to the control condition.
The researchers obtained consent from 583 students and analysed complete data for 555: 278 in intervention sections and 277 in controls. Scores were collected at weeks 1, 7 and 14, with EduBot available for 12 weeks. The analysis used mixed-effects models to account for repeated observations and nesting in course sections, plus mixed ANOVA and sensitivity analyses. Qualitative evidence came from interviews and focus groups and was used to generate explanations, not to prove statistical mediation.[1]
What changed on the seven-point scales
Behavioural engagement in the intervention group rose from 4.82 to 5.47, while the control group moved from 4.79 to 4.76. Cognitive engagement rose from 4.91 to 5.36 with EduBot and changed from 4.88 to 4.85 in control sections. Agentic engagement—students contributing questions and shaping their learning—rose from 4.45 to 5.11 versus 4.41 to 4.52. Emotional engagement showed the smallest contrast: 5.04 to 5.19 versus 5.01 to 4.82.
Usage was uneven. Intervention students logged a mean of 31.42 sessions and a median of 28, but 18 students, 6.5%, never used the tool. Within the intervention group, more sessions were associated with higher post-test engagement after adjustment for baseline and covariates. That dose-response pattern is compatible with benefit, yet motivated students may also choose to use the tool more. It cannot by itself show that extra sessions caused the higher scores.[1]
Why the Socratic design may matter
EduBot was designed to ask questions and scaffold reasoning instead of immediately supplying answers. The thematic analysis identified three proposed mechanisms: students described a low-judgment place to repeat questions, prompts that made them persist through a problem and access outside office hours. Teachers and assistants reported that routine questions could be handled before students sought human help. These accounts are plausible and useful for design, but they were not tested as measured causal pathways.
The intervention was a package. It changed availability, conversational format, personalisation and questioning style at once. The comparison therefore cannot tell a university whether Socratic prompts, round-the-clock access or simply extra contact produced the difference. Nor can it show whether the same design would help in a seminar, a vocational course or a less digitally prepared institution. Component tests and replicated courses are needed before choosing one feature as the active ingredient.[1]
Practical implications for teachers and universities
Teachers considering a similar tool should begin with a bounded purpose: concept clarification and guided practice, not unsupervised grading or replacement of contact. They should inspect how the system responds to misconceptions, require citations where appropriate, protect student data and keep a route to a human teacher. Engagement should be tracked alongside learning outcomes. A busier discussion board or higher self-report score is not enough if misconceptions persist or students become dependent on prompts.
Universities should pay particular attention to non-users. The 18 students who never opened EduBot may represent access, trust, disability, motivation or digital-literacy barriers. Exploratory results suggested larger agentic gains for first-year students and larger cognitive gains for students with lower prior GPA, but only two of eight moderation tests survived correction and they were not pre-registered. Institutions should not present those subgroup findings as settled evidence of equity benefits.[1]
Evidence limits and what would change the assessment
The strongest limitation is the small number of clusters. Students were not individually randomised, only eight sections participated and unmeasured section effects remain possible despite balanced measured characteristics. University and course were completely confounded, so a subject effect could look like an institutional effect. Measures were self-reported and the last survey occurred in the week after access ended, leaving durability unknown. The integrated package also prevents attribution to any single design feature.
The study was funded by China's National Social Science Fund through a project on university teachers' digital literacy; the authors declared no commercial conflict. Confidence would rise with a preregistered cluster-randomised trial across many replicated courses, direct knowledge tests and grades, longer follow-up, documented model behaviour and analysis of who declines or stops use. For now, the result supports supervised pilots of Socratic tutoring. It does not justify replacing teachers or claiming that AI raised attainment.[1]