Does AI help trainee teachers reflect better after metacognitive training?
In a randomized 60-person experiment, a configured Socratic AI improved written reflection only when paired with a 20-minute metacognitive lesson. The immediate, single-university result does not show that generic chatbot access improves teaching practice.
The short version
- What happened
- Sixty third-year trainee teachers at one Chinese university were randomized equally across metacognitive or placebo training and AI Socratic or conventional digital-tool conditions.
- Who it affects
- Trainee teachers may need explicit self-questioning and evidence-evaluation routines before an AI dialogue supports rather than substitutes for reflection.
- What remains unproven
- Only 60 participants completed the experiment, with 15 in each factorial cell.
Keep in mind
Only 60 participants completed the experiment, with 15 in each factorial cell.
The single-university volunteer sample limits institutional and cultural generalisability.
The AI condition bundled generation, adaptive questioning, dialogue and a Socratic design.
Outcomes were immediate written reflections, not retained skill, classroom practice or pupil benefit.
Prompt patterns were consistent with a mechanism but were not tested as statistical mediation.
Source analysis · 1 linked source. Primary analysed source date: 13 October 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The answer: preparation mattered more than access alone
A configured AI system improved trainee teachers' written reflection only when users first received a short lesson in planning, self-questioning, monitoring and evaluation. Among 60 participants randomized across four groups, the adjusted AI-versus-traditional-tool difference was 0.56 points when metacognitive training was present. Without that training, the difference was minus 0.03 points and statistically indistinguishable from zero. The interaction between training and tool condition was significant, with a partial eta-squared of 0.165. The useful finding is conditional: how learners approach the tool may be as important as giving them access.
This is not evidence that any chatbot makes teachers more reflective. The AI condition used a DeepSeek V3 web interface deliberately configured to avoid ready-made answers and ask progressively probing questions. It bundled generative output, dialogue, adaptive follow-up and Socratic instructional design. The comparison group used a text editor and a university digital repository. Because those features were not separated, the experiment cannot tell whether generation, questioning, novelty or another component caused the difference. Nor did it observe a classroom, a pupil or a teaching decision beyond the study task.[1]
How the experiment worked
Sixty third-year pre-service teachers at Changzhou Institute of Technology completed the full procedure. Stratified random allocation placed 15 in each cell of a two-by-two design: metacognitive training or a time-matched placebo lesson, combined with the configured AI or conventional digital tools. Before any intervention, everyone had 30 minutes to analyse a teaching vignette and submit a written reflection. One week later, participants received a 20-minute lesson—either metacognitive strategies or PowerPoint design—and completed a matched post-test case with their assigned tool.
Trained raters scored the written reflections on a rubric covering evidence, attribution, perspectives, strategy and clarity. The researchers used an analysis of covariance to compare post-test performance while controlling for the pretest. Baseline scores did not differ significantly across groups, and reported checks supported the model assumptions. Sensitivity analyses excluding influential observations, using robust standard errors and restricting the sample to 53 participants who reported following their assigned condition retained the interaction. Those checks make the internal result more credible, though they cannot compensate for the small cells or single setting.[1]
What changed in the written reflections and prompts
The adjusted post-test mean was 4.13 for metacognitive training plus AI, compared with 3.57 for metacognitive training plus conventional tools, 3.33 for placebo plus AI and 3.36 for placebo plus conventional tools. The interaction was strongest for depth and attribution, theory and perspective taking, and strategy generation and transfer. It was not significant for basic description and evidence or for structure and clarity. That pattern is compatible with the idea that preparation helped participants use dialogue for higher-order analysis rather than simply polishing a response.
The researchers also coded 487 prompts from the two AI groups. Among participants who received metacognitive training, 44.1% of prompts were classified as higher-order and 21.3% as lower-order. In the placebo group, 12.2% were higher-order and 54.5% lower-order. Two doctoral coders achieved a Cohen's kappa of 0.86 on an initial 20% sample. This behavioural difference supports the proposed mechanism, but it does not prove mediation: the log analysis did not statistically connect each participant's prompt pattern with the treatment and final reflection score through an indirect-effect model.[1]
What teachers and course designers can use now
Teacher educators considering AI-supported reflection should teach a method for using it, not just demonstrate prompts. Learners can be required to state an initial interpretation, identify evidence, test an alternative explanation, consider another perspective and revisit their conclusion after the system responds. The AI can be configured to withhold complete answers until those steps occur. This creates productive friction and gives instructors observable process evidence. It also reduces the risk that a reflective exercise becomes a request for a fluent improvement plan written by the system.
The experiment does not establish that the combined method is better than an equally structured non-AI Socratic scaffold. Its traditional condition offered a repository and text editor, not a rule-based questioning tool or trained human tutor. Institutions should therefore pilot the smallest intervention that meets the learning goal and compare it with credible alternatives. They should also keep assessment focused on the learner's reasoning and ability to justify action. A well-formed prompt is not evidence that a trainee can adapt safely in a classroom or recognise when an AI suggestion conflicts with a pupil's needs.[1]
Why the result remains narrow
Each experimental cell contained only 15 people, making estimates vulnerable to sampling variation even though the reported effect survived several checks. Participants were volunteers with basic information-technology skills at one Chinese university. Local norms around teacher authority, criticism and technology could affect both how reflection is written and how comfortably a student challenges an AI system. The adapted reflection questionnaire was examined in the same small sample, so its psychometric structure is exploratory rather than a definitive cross-cultural validation.
The intervention was brief and the post-test immediate. The study cannot tell whether participants retained metacognitive habits, used them during teaching practice or improved outcomes for pupils. It also cannot separate the effect of DeepSeek V3 from the system prompt and interaction design. The authors report no external funding and no commercial conflicts. That independence is useful context, but reproducible configuration details, preregistration and independent replication would be more important for judging whether the result survives across models and institutions.[1]
What would change the assessment
A larger preregistered trial across universities should retain random allocation, use blinded raters and compare four credible supports: configured AI, a non-AI Socratic scaffold, human-guided reflection and ordinary digital tools. A dismantling design could separate adaptive questioning from generative content and dialogue. Researchers should publish the configuration and test more than one model so that a finding about pedagogy is not mistaken for a claim about a particular product.
Follow-up during teaching placements would provide the most important evidence. Researchers could examine lesson revisions, responses to classroom dilemmas, use of evidence, willingness to challenge automated suggestions and effects on pupils or mentors. Benefits should persist when the novelty has faded and when participants face messy, emotionally charged cases rather than matched written vignettes. Until then, the careful conclusion is that metacognitive preparation improved short-term use of one Socratic AI environment; generic access alone did not.[1]