Did AI-assisted teaching improve computational thinking?
A 14-week pilot at one Chinese medical college found larger gains in an AI-assisted class than in a conventional class. But one intact class received each condition, so the 120-student study cannot separate the teaching package from class, teacher, time-on-task or other group differences.
The short version
- What happened
- Two intact university classes totalling 120 students completed a 14-week comparison; the AI-assisted class scored higher on the final composite measure, with Cohen's d of 0.97.
- Who it affects
- Teachers may gain a clearer structure for adaptive practice, but need protected time and authority to review recommendations and correct system errors.
- What remains unproven
- Only two intact classes at one Chinese medical college participated, with one class per condition; class and treatment effects cannot be separated.
Keep in mind
Only two intact classes at one Chinese medical college participated, with one class per condition; class and treatment effects cannot be separated.
The intervention bundled resources, adaptive sequencing, multimodal support and evaluation, so the contribution of any one AI component is unknown.
Week-five and week-nine abbreviated assessments were not formally equated to the full instrument and support only descriptive trajectory claims.
The study did not establish long-term retention, transfer to unfamiliar tasks, independent project performance or reduced dependence on AI support.
Costs, teacher workload, accessibility, system errors and infrastructure requirements were not central outcomes in the reported comparison.
Source analysis · 1 linked source. Primary analysed source date: 9 October 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The answer: scores improved, but the design cannot isolate AI
Students in the AI-assisted class made larger measured gains in computational thinking than students in the comparison class during a 14-week pilot at Puyang Medical College in Henan, China. The final composite difference produced t(118)=5.36, p<0.001 and Cohen's d=0.97. The largest reported differences were in abstraction, with d=1.03, and evaluation with generalisation, with d=0.98. An analysis adjusting for pre-test scores still found a group difference.
Those are sizeable effects, but they do not establish that the adaptive engine caused them. The study assigned one intact class to the full AI-assisted package and one intact class to conventional instruction. Class membership and treatment were therefore the same variable. Any difference in teachers, peer dynamics, timetable, motivation, resource exposure or time spent online could move the result, and no statistical adjustment can remove a factor that exists only at class level when there is one class per condition.
The most accurate interpretation is feasibility: the researchers implemented a coherent four-layer programme for a full term, students completed it, and the intervention class improved on the chosen measure. The study generates a testable model for larger trials. It does not justify a claim that AI-assisted teaching will produce the same improvement in other institutions, subjects or student populations.[1]
What the four-layer package actually included
The first layer managed learning resources. Materials were organised so students could receive content suited to the topic and their current position in the course. The second layer used deep knowledge tracing—a long short-term memory model intended to estimate changing mastery from prior interactions—to recommend an adaptive learning path. The third added multimodal interaction support, while the fourth evaluated computational thinking across several dimensions and competency levels using a fuzzy comprehensive evaluation framework.
This matters because the intervention was not simply access to a chatbot. It combined new resources, sequencing, feedback, interaction and assessment. A higher score could reflect the package as a whole, increased structure, additional practice or closer monitoring. It cannot be attributed cleanly to generative AI, deep knowledge tracing or any one interface. Schools considering adoption should treat the programme as a complex instructional redesign rather than a software switch.
The outcome framework covered decomposition, abstraction, algorithmic thinking, debugging and evaluation with generalisation across three competency tiers. Mid-course checks at weeks five and nine used shortened parallel forms. The authors explicitly note that those abbreviated forms were not formally equated to the full assessment, so the interim trajectory should be read descriptively rather than as a precise growth curve.[1]
The 120-student comparison and its denominator
The quasi-experiment included 120 students in two existing classes at a single medical college. Each class received one condition in its entirety. The paper reports pre- and post-intervention computational-thinking measures and the 14-week delivery period. Written consent was obtained, students could withdraw without academic penalty and could ask for behavioural logs to be excluded. The college's research ethics committee approved the protocol.
The comparison group received conventional instruction, while the experimental group received the integrated four-layer package. The published abstract describes an ANCOVA that adjusted the final comparison for baseline scores. That helps address individual pre-test differences, but it does not create independent treatment units. The effective unit of assignment is the class, leaving one unit in each arm and no way to estimate variation between classes.
The researchers also analysed behavioural correlates. Task-completion volume and debugging efficiency were the strongest predictors of learning gains among the collected measures. Medium-experience learners appeared to benefit most from adaptive sequencing, while low-experience learners gained mainly on foundational dimensions. These subgroup findings are exploratory: dividing a 120-student sample reduces precision, and behaviour may reflect engagement or prior ability rather than cause improvement.[1]
Why the effect size should not be treated as a product claim
Cohen's d of 0.97 describes the standardised difference observed between these two classes. It is not a forecast that another school will gain nearly one standard deviation. With only one class in each condition, the estimate includes every unmeasured feature that separated those groups. Novelty, teacher attention, platform availability and different amounts of practice could all contribute.
The outcome itself was the study's composite computational-thinking measure, not an external examination, long-term retention test or independent project completed after the intervention. The paper reports no evidence here about whether students transferred skills to a new subject, retained them months later or became less dependent on automated support. Measurement tied closely to an intervention can capture genuine learning while still overestimating broader capability.
Nor does the study isolate costs. Building and maintaining adaptive resources, reviewing model output, supporting students and monitoring data can create substantial teacher and technical work. A scalable programme needs evidence about preparation time, accessibility, device and connectivity requirements, error correction and the students who disengage—not only average scores among those who complete the course.[1]
Practical implications for teachers and institutions
For teachers, the useful takeaway is the instructional architecture. Resources, adaptation, interaction and evaluation were designed together around named thinking skills. That is more defensible than giving students an unrestricted tool and hoping usage becomes learning. A local pilot could preserve this structure while simplifying the technology and measuring which component changes practice.
For students, adaptive sequencing may help match practice to current mastery, but it can also narrow what a system thinks they are ready to attempt. Learners should be able to see why material was recommended, ask for a different route and reach a teacher when the inferred level is wrong. Debugging and explanation should remain assessed human capabilities rather than being silently performed by the system.
For institutions, the safest next step is a limited, supported trial with independent assessment and a pre-agreed exit plan. Procurement should cover data minimisation, retention, accessibility, bias, teacher override and audit logs. The study's large within-site difference is a reason to investigate the model carefully, not a reason to replace conventional teaching across a curriculum.[1]
What would change the assessment
A stronger test would randomise many classes or students across several institutions, stratify by prior experience and use the same teachers or balanced teaching assignments across conditions. An active control should match resources, time-on-task and feedback while withholding the adaptive component. A dismantling design could then test whether resource curation, sequencing, multimodal interaction or the evaluation framework produces the gain.
Outcomes should include blinded performance on unfamiliar problems, retention months later, independent project quality, error detection, workload and accessibility. Researchers should publish attrition, missing data, system errors and results for students with different prior attainment and support needs. Formal equating of interim and final instruments would make learning trajectories interpretable.
Until then, the finding remains encouraging but local: one AI-assisted class outscored one conventional class after a term-long package. The work was funded by a project on a campus intelligent learning assistant based on generative AI, and the authors declared no competing interests. Replication that separates treatment from class is necessary before the effect can guide wide adoption.[1]