Can GPT-4 write a valid chemistry test? The items mostly fitted—but their intended difficulty levels did not
Researchers generated 27 German electrochemistry questions and tested them with 95 upper-secondary students. Most items met many writing criteria and largely fitted a Rasch model, but language, distractors and competence-level alignment exposed why teachers still need expert and psychometric checks.
The short version
- What happened
- GPT-4 generated 27 electrochemistry items—nine for each of three intended competence levels—and researchers administered the unchanged test to 95 upper-secondary students.
- Who it affects
- Teachers can use generated questions as drafts but retain responsibility for content and score meaning.
- What remains unproven
- One 27-item German electrochemistry test, one GPT-4 process and 95 students.
Keep in mind
One 27-item German electrochemistry test, one GPT-4 process and 95 students.
No human-written comparison instrument.
Person reliability was 0.66 and level alignment unstable.
Researchers manually checked prompt compliance and textbook correctness.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The direct answer: GPT-4 produced a workable draft, not a ready-to-trust assessment
A peer-reviewed Frontiers in Education study published on 4 November reports a mixed result for AI-authored school assessment. GPT-4 generated 27 German-language multiple-choice questions on electrochemistry, divided evenly across three curriculum-defined competence levels. Ninety-five upper-secondary students completed the unchanged test after an electrochemistry unit. Many questions met established writing criteria, and most behaved acceptably under Rasch-model checks. Yet the claimed progression from recall to explanation and complex transfer did not reliably match empirical difficulty. Several distractors attracted almost nobody, and measurement precision for individuals was limited.
That combination matters more than a simple pass rate. A question can be grammatically plausible and fit a statistical model while failing to measure the intended competence. Teachers may use AI to reduce blank-page work, but a consequential test needs evidence that scores mean what the test claims. This single-topic study supports supervised drafting followed by expert review, student piloting and psychometric analysis. It does not support copying generated questions into an examination or assuming that a prompt label such as 'level III' creates a genuinely higher-order task.[1]
How the test was generated
Researchers pasted definitions and competence expectations from the North Rhine-Westphalia curriculum into an iteratively refined prompt. GPT-4 first produced nine items, then generation continued until there were nine at each of three levels: recall, explanation in a new context and complex transfer. Each item had four options with one correct answer. The prompt requested relevant background in the stem and language suited to the target group. Researchers manually checked format and topic and checked correctness against a chemistry textbook.
The questions were not edited before the study. That makes the evaluation more informative about generated output, but it is not how a careful teacher would normally deploy a test. The prompt was refined over several iterations until outputs met the requested structure, so this was neither one-shot generation nor an automatic pipeline. Human choices shaped the curriculum material, instructions, stopping point, correctness check and final set. The study evaluates an AI-assisted development process, not isolated model capability independent of expert scaffolding.[1]
The distinction that matters
A generated question must pass more than one test
The study evaluated wording, student-response behaviour and curriculum-level claims separately.
Generation
27 electrochemistry items
Nine requested at each competence level
Writing review
22 criteria, two coders
87.71% average with language weaknesses
Student test
95 analysed pupils
Most items fitted; person reliability 0.66
Distractors
Wrong-answer patterns
Twelve distractors in ten items drew under 5%
Level claims
Labels versus empirical groups
Only seven items were bootstrap-stable
Writing quality was broadly strong, with recurring weaknesses
Two authors independently coded all 27 items against 22 binary criteria, with Cohen's kappa of 0.93 before disagreements were resolved. The average pass rate was 87.71%, but that headline hides large differences. All items met criteria for contextualisation, curriculum content and specific competencies, avoidance of bias and several option-format rules. Only 33.33% reduced specialised language sufficiently, and 48.15% balanced answer-option length. Three items had spelling errors and six had syntax errors. Plausible distractors were present in 77.78% of items.
Aggregate quality scores can therefore mislead. Technical chemistry necessarily uses specialist terms, but unnecessary jargon can turn a subject test into a reading barrier. Unequal option length can cue test-wise students. Syntax errors may alter meaning, and an implausible distractor makes an item easier without measuring intended knowledge. A teacher should use a structured checklist rather than ask whether a question merely looks polished. Independent review is important because fluent wording can hide a cue, ambiguity or content assumption that the generator and first reviewer both overlook.[1]
The Rasch results were promising, not a certificate of validity
The test went to 98 students; three were excluded for missing responses, leaving 95. One item underfit and one overfit the Rasch model; remaining fit statistics were 0.79 to 1.28 under the authors' criteria. Item reliability was 0.91, indicating a stable relative hierarchy in this sample and model. Person reliability was 0.66, limiting precision for distinguishing individual students. Measurement information was concentrated around the middle of the observed ability range rather than equally strong at the extremes.
Rasch fit is not proof that every question is correct, fair or educationally useful. It asks whether response patterns conform reasonably to model expectations. It does not validate content by itself, establish that scores predict learning or show equivalence to an established test. The sample was modest and from one target context, and no age or gender variables were collected. The identical item order means order effects were not tested. Replication across schools, cohorts and item orders is needed before treating parameters as stable beyond this administration.[1]
Weak distractors and unstable level labels are the practical warning
Twelve distractors across ten questions were chosen by fewer than 5% of students, suggesting those wrong answers contributed little discrimination. Of 77 estimable distractors, 61—79.22%—had the expected negative relationship with ability, while 16 did not. A distractor that attracts almost nobody may need replacement; one that behaves unexpectedly can reveal ambiguity, misconception or scoring trouble. Teachers cannot judge this from wording alone. It becomes visible when representative students answer and response patterns are analysed.
The central weakness concerned competence levels. Intended curriculum labels showed weak agreement with empirical groups and were sensitive to anchoring rules. Only seven of 27 items retained their classification in at least 70% of bootstrap samples. Median adjusted agreement was 43.48%, with weighted kappa 0.314. Difficulty and cognitive demand are not identical: a complex task can be easy for a well-taught class, while a recall item can be hard because of obscure wording. A model's level label is a hypothesis for review, not metadata a school should trust automatically.[1]
What changes for teachers—and what would change the assessment
Teachers can use a model to propose curriculum-anchored drafts, then check content, language, cues, accessibility and distractor plausibility before piloting. AI may stimulate variants, but the final test needs a named professional owner. Generated items also need version records covering the prompt, model, curriculum input, human edits, source check and pilot statistics. Pupils can be harmed by ambiguous wording and hidden reading load when scores influence feedback, placement or confidence. High-stakes use demands stronger evidence than this study provides.
There was no matched human-written test, so the study cannot say whether GPT-4 was better, faster or cheaper than expert authorship. Confidence would rise with a preregistered multi-school comparison at matched curriculum coverage, using blinded expert review and repeated samples. Outcomes should include errors, accessibility, teacher time, cost, reliability and subgroup functioning. It would weaken if generated items repeatedly misclassified demand or disadvantaged particular pupils. This is consequential research but not Breaking: it offers assessment evidence, not an urgent school instruction or active safety event.[1]