Better learning with AI? Look closely at what the studies actually tested

A promising result, with a catch for teachers

A new review reports strong learning gains from teaching supported by technology. It offers a less certain answer to a question many teachers face: does adding AI help pupils learn? Only a few of the studies it combined explicitly tested an AI intervention.

The peer-reviewed meta-analysis, published in Frontiers in Education, combined 11 studies involving 477 learners. It reports a large pooled advantage for blended and technology-enhanced instruction over traditional teaching: Hedges' g=0.89, with a reported 95% confidence interval from 0.76 to 1.10. This standardised effect describes the combined studies; it does not isolate the benefit of an AI system.

The paper itself supplies the decisive qualification: only a few included studies explicitly incorporated AI-driven interventions. Others used learning-management systems, mobile applications, podcasts, vocabulary tools or broader blended-learning designs. The pooled outcomes also ranged from academic achievement and literacy to vocabulary, digital literacy and motivation. The result therefore describes a mixed family of technology-supported teaching approaches rather than a single AI intervention or a direct improvement in writing proficiency.[1]

What the researchers searched and combined

The authors searched ERIC, Scopus, Web of Science and Google Scholar and manually checked journals and reference lists. They report finding 510 records, removing 112 duplicates, screening 398 titles and abstracts and assessing 78 full texts. Eleven studies met their final criteria. Two coders screened the literature, with reported agreement of Cohen's kappa=0.87, and extracted means, standard deviations or test statistics used to calculate Hedges' g.

Seven of the 11 studies used quasi-experimental pretest–posttest control-group designs; the remainder included other experimental or single-group pre–post designs allowed by the protocol. Individual samples ranged from 23 to 94 learners, with a mean of about 43.4. The studies covered primary, secondary and tertiary settings in Indonesia, Thailand, Malaysia, China, Australia and the United States, with most coming from Southeast Asia.[1]

The pooled number hides major variation

The reported study effects ranged from 0.50 to 1.81, and heterogeneity was high: Cochran's Q was 68.42 on 10 degrees of freedom, p<0.001, and I² was 86.8%. A random-effects model is appropriate when true effects differ, but it does not make the interventions or outcomes interchangeable. High heterogeneity means the average should not be read as the result a teacher can expect from a particular product, class or group of students.

Exploratory subgroup estimates were larger in primary education than higher education, for four-to-eight-week interventions than shorter or longer programmes, and for interactive tools than static materials. Yet those comparisons split only 11 studies into still smaller groups. The authors correctly label them exploratory and warn that estimates may be unstable and underpowered. They cannot establish an optimum programme length or prove that interactivity itself caused the differences.[1]

Two eligibility tensions need resolution

The methods say the search covered studies published from 2020 through 2023, but the included-study table lists Xie et al. as a 2024 study. The eligibility section also says meta-analyses were excluded, while another included citation—Rohmawati, Holisin and Kristanti—has a title identifying it as a blended-learning meta-analysis. These may be reporting or citation errors rather than flaws in the underlying calculation, but the published article does not reconcile them.

That matters because a meta-analysis depends on a transparent, reproducible boundary around the studies being pooled. Including a study outside the declared period or combining a prior meta-analysis as though it were a primary experiment could change weighting and double-count evidence. A correction, supplementary extraction table and accessible analysis file would allow readers to determine whether the issue is editorial wording, a miscitation or a substantive eligibility problem.[1]

What teachers and school leaders can use now

The defensible lesson is that purposeful digital and blended instruction can outperform a comparison condition in small studies, especially when learners receive interaction and feedback. That supports careful local trials of tools tied to a defined learning objective. It does not justify buying an AI writing system on the strength of g=0.89, because the pooled estimate is neither AI-specific nor writing-specific.

Teachers should ask what students actually did, who provided feedback, how much class time changed and whether the assessment measured durable independent learning. A programme may succeed because of structured practice, extra attention or novelty rather than automation. Schools also need to inspect privacy, accessibility, unequal access and the quality of generated feedback—questions a learning-outcome average cannot answer.[1]

What would change the assessment

Confidence in an AI-specific benefit would require a preregistered review that separates generative AI, automated feedback, adaptive tutoring, ordinary learning platforms and non-AI digital tools. Each outcome should be analysed within a comparable construct, with duplicate samples removed and study risk of bias graded. Larger trials should compare the same curriculum and teacher support with and without the AI component, then follow independent performance rather than only work produced with assistance.

For now, the new paper is useful as evidence that technology-supported teaching deserves continued study and as a warning against headline compression. Its large pooled effect belongs to a heterogeneous set of small digital-learning studies. It does not demonstrate that AI itself improves writing, that any named product is effective or that schools should replace professional judgement with a broad average.[1]