Did ChatGPT improve writing scores in a 16-week Taiwan course?
Twelve university students used ChatGPT throughout a five-stage inquiry-writing course. Their closed-book writing scores rose by 2.79 points, but the change was not statistically significant and the study had no comparison group.
The short version
- What happened
- The study followed 12 English-major students in one 16-week composition course at a national university in Taiwan; there was no comparison group or random assignment.
- Who it affects
- Students should not be told that the study proves ChatGPT improves writing; its measured score change was uncertain.
- What remains unproven
- The sample contained only 12 students from one English-major course at one university in Taiwan.
Keep in mind
The sample contained only 12 students from one English-major course at one university in Taiwan.
There was no control group or random assignment, so the study cannot attribute score changes to ChatGPT or the 5E framework.
The pre- and post-tests used different prompts, and the timed essay did not directly measure the extended inquiry-writing skills taught in the course.
The overall 2.79-point gain was not statistically significant at p = .13.
Submitted ChatGPT logs were incomplete, and reflective journals were the sole evidence for overall student perceptions.
The sole author designed and taught the intervention as well as leading data collection and analysis.
Source analysis · 1 linked source. Primary analysed source date: 5 October 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The clearest measured result is a non-result
A new peer-reviewed study from Taiwan offers a useful counterweight to claims that simply adding a chatbot improves student writing. Twelve university students completed a 16-week English-as-a-foreign-language composition course in which ChatGPT was threaded through a five-stage inquiry cycle: Engage, Explore, Explain, Elaborate and Evaluate. The students' average closed-book argumentative-writing score increased, but the change did not meet the study's threshold for statistical significance.
The mean rose from 78.13 before the course to 80.92 afterwards, an increase of 2.79 points. The paired-samples test produced t(11) = -1.63 and p = .13, with Cohen's d = 0.47. A moderate effect-size estimate in a group this small is not proof of benefit: sampling uncertainty is wide, there was no control group, and the two tests used different topics. The responsible reading is that this class showed a modest upward pattern that could reflect the intervention, ordinary teaching, practice, topic difficulty or chance.[1]
What the course actually asked students to do
The research took place in an intact undergraduate English-major course at a national university in Taiwan during autumn 2024. ChatGPT was not presented as a free-standing answer generator. Students used it during a sequence that moved from framing a topic and designing a questionnaire to interpreting findings, drafting sections and evaluating a completed research report. The instructor also trained students to specify roles, constraints and output formats in their prompts and to compare AI suggestions with peer and teacher feedback.
That design matters because the intervention was a package: an inquiry framework, normal instruction, peer discussion, reflective work and ChatGPT use. The study cannot isolate which element produced any change. It also cannot tell whether the same students would have developed similar habits in a well-taught inquiry course without the chatbot. The paper explicitly describes the research as exploratory rather than causal, a boundary that schools should preserve when considering adoption.[1]
The assessment measured a related but different kind of writing
At the beginning and end of the semester, students wrote an IELTS-style Task 2 essay for 40 minutes without ChatGPT or online resources. Two experienced EFL teachers scored task response, coherence and cohesion, lexical resource, and grammatical range and accuracy. Inter-rater reliability for the overall score was reported as ICC(2,2) = 0.80, with a 95% bootstrap confidence interval from 0.70 to 0.87.
The pre-test asked whether social media is more harmful or helpful, while the post-test concerned whether computer translation makes foreign-language learning a waste of time. Those prompts are not identical, and neither directly measures the extended research-writing tasks central to the course. Task response and grammatical range showed the largest mean gains, with p values of .09 and .08, while coherence and lexical-resource changes were smaller. None of those comparisons establishes that ChatGPT caused improvement.[1]
The qualitative record points to a teaching practice, not an efficacy verdict
The study also analysed 30 peer-discussion recordings, staged writing artefacts, submitted ChatGPT interaction screenshots or links, and end-of-course reflective journals. The authors report that students moved from broad requests toward more phase-specific prompting: asking for neutral questionnaire wording during exploration, candidate themes during analysis, or a whole-text clarity check during evaluation. Students sometimes revised or rejected suggestions rather than accepting them wholesale.
Those observations are useful because they make the learning process visible. They suggest that structured tasks can turn AI use into something teachers can discuss and critique. But the interaction logs were incomplete and used as supplementary evidence, so the record does not capture deleted, informal or unsubmitted conversations. Reflective journals were the only source for students' overall perceptions. Themes about agency or critical filtering therefore describe this small class; they do not measure independent learning gains across a wider student population.[1]
For teachers, the practical lesson is about scaffolding
The most defensible application is not a claim that ChatGPT raises scores. It is a design hypothesis: tie AI use to a defined stage of inquiry, require students to record and evaluate suggestions, and keep peers and instructors in the feedback loop. A teacher could ask students to explain why they accepted or rejected an AI-proposed survey question, identify evidence that supports an interpretation, and preserve versions showing how a draft changed. Those activities make judgment and authorship assessable.
The design also surfaces risks. If a course grades only polished final prose, students may optimise for fluent output instead of learning. If prompting becomes a substitute for subject knowledge, they may lack the basis to reject a plausible error. And if access to capable tools, devices or language support differs, an AI requirement can widen rather than narrow opportunity. Schools need an accessibility plan, transparent rules, protected routes for opting out and assessments that still reveal what students can do without the system.[1]
Funding and researcher roles deserve context
Taiwan's National Science and Technology Council supported the work under grant NSTC 113-2410-H-194-095. The paper declares no competing interests. The sole author designed the study and course intervention, collected the data, analysed the quantitative and qualitative material, and wrote the manuscript. Two EFL teachers independently rated the writing assessments, and subsets of qualitative material were double-coded, with initial agreement reported at 84% for peer-discussion segments and 88% for reflective journals.
That level of disclosure helps readers understand the study, but it cannot remove expectancy effects when the researcher is also closely involved with the intervention. Students knew they were participating, and a small intact class can be unusually responsive to a teacher's emphasis. Independent replication by instructors who did not design the framework would make the evidence more persuasive.[1]
What would change the assessment
A stronger test would compare multiple classes across institutions, ideally allocating similar students to the same writing curriculum with and without the structured ChatGPT component. It should pre-register primary outcomes; use equivalent or counterbalanced writing prompts; score work blind to group and timepoint; and report confidence intervals, attrition and tool usage. Direct ratings of research reports would better match the intervention than relying mainly on a timed argumentative essay.
Longer follow-up would answer whether students retain prompting and critical-evaluation skills after support is withdrawn. Process logs could show which AI interactions precede productive revision, while interviews and classroom observation could test whether reported agency matches behaviour. Until then, the paper supports careful experimentation with structured AI pedagogy. It does not show that ChatGPT improved writing scores, that the five-stage method is superior to other teaching, or that the approach will transfer beyond one 12-person Taiwanese class.[1]