Can AI write medical exam questions that are ready to use?
Almost never in this epilepsy benchmark. Seven language models produced 420 questions, but two blinded clinicians judged only two—0.48%—directly usable without revision. Answer accuracy ranged from 61.67% to 98.33%, and weak distractors were the recurring problem.
The short version
- What happened
- Seven models each completed the same 60 prespecified epilepsy tasks, producing 420 single-best-answer questions under a common one-shot prompt and without web search, tools or model-specific tuning.
- Who it affects
- Medical educators can treat model output as a draft queue, but every keyed answer, explanation and distractor still needs qualified review.
- What remains unproven
- Only one generation was collected for each model-task pair, preventing estimates of stochastic variability and repeatability.
Keep in mind
Only one generation was collected for each model-task pair, preventing estimates of stochastic variability and repeatability.
Two expert raters judged usability; no learners completed the items and no psychometric validity was measured.
The single-turn, no-tools protocol may understate the performance of carefully grounded or iterative workflows.
Model versions and API configurations change quickly, so the ranking may not persist.
The study did not measure authoring time, total review cost, learning outcomes or patient outcomes.
Source analysis · 1 linked source. Primary analysed source date: 13 October 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The answer: AI produced drafts, not a ready-made question bank
A peer-reviewed benchmark published in Frontiers in Medicine found that current language models could usually reproduce the visible structure of an epilepsy multiple-choice question, but almost none of the resulting items were ready for direct educational use. Two blinded clinical reviewers rated only 2 of 420 questions as usable without revision. Another 119, or 28.33%, could be used after minor revision, while 299, or 71.19%, needed substantial revision or were not recommended.
That is the practical result for medical educators. A correctly formatted stem, four options, one keyed answer and a concise explanation can look finished while still containing weak alternatives, imprecise reasoning or a clinical error. The paper supports using a model as an expert-supervised drafting aid. It does not support publishing an automatically generated examination, question bank or continuing-education test without independent clinical and educational review.[1]
How the researchers made the comparison fair
The China-based research team created 60 prespecified tasks, divided equally across four areas: 15 on foundational knowledge, 15 clinical cases, 15 treatment decisions and 15 on diagnosis, evaluation and longitudinal management. Each task specified a knowledge point, intended difficulty and cognitive level. GPT-5.6, Gemini-3.5, DeepSeek-V4, Qwen3.8, Doubao-2.1, Claude-5 and Kimi-K3 each received all 60 tasks, giving 420 model-question pairs.
Every model was accessed through an official application programming interface and received the same single-turn prompt. The systems could not search the web or use external tools, and the researchers did not optimise the prompt for individual models. A single standard corrective prompt was allowed when a first response failed the requested structure. This design makes the head-to-head comparison easier to interpret, but it tests one controlled workflow rather than the best result a skilled educator might obtain through retrieval, iteration or fine-tuning.[1]
Accuracy varied, and high accuracy did not guarantee a good assessment item
The researchers checked the keyed answer against an epilepsy-specific reference framework assembled before outcome assessment. It drew on current guidelines and consensus recommendations, International League Against Epilepsy classification papers, established textbooks and selected educational references. Keyed-answer accuracy ranged from 37 of 60 for Doubao-2.1, or 61.67%, to 59 of 60 for Claude-5, or 98.33%. GPT-5.6 and Kimi-K3 each reached 57 of 60, or 95%, while Gemini-3.5 reached 54 of 60, or 90%.
Even the upper results leave room for consequential error when questions are used for training or certification. More importantly, knowing the keyed answer is not the whole job. A useful single-best-answer item needs plausible but clearly inferior distractors, an unambiguous stem and alignment with the intended cognitive level. Median distractor-quality ratings were only two or three on the five-point scale across models. That weakness can make a question trivial, confusing or misleading even when the nominal answer is correct.[1]
Two clinicians scored every item, with substantial agreement
Two clinicians with expertise in epilepsy or clinical neurology and experience in medical education independently rated content rigour, clarity, distractor quality, educational usefulness, cognitive alignment and overall usability. They were blinded to which model produced each item. Intraclass correlation coefficients across domains ranged from 0.794 to 0.859, indicating substantial consistency between their ratings without making the judgments objective or universal.
GPT-5.6 had the highest composite expert score, with a median of 3.75 and an interquartile range of 3.50 to 4.50. Claude-5 followed at 3.67 and Kimi-K3 at 3.50. Yet Claude-5 had the highest keyed-answer accuracy while GPT-5.6 led the composite rating and produced the only two directly usable questions. The split illustrates why a single leaderboard number cannot substitute for examining the dimension that matters in a particular educational setting.[1]
What teachers and clinical training teams should do with this evidence
An educator could use a model to create a first draft, then require a named subject expert to verify the source, keyed answer, explanation, difficulty and every distractor. A second review should check whether the question rewards clinically relevant reasoning rather than word patterns or test-taking tricks. Institutions also need version control: model behaviour changes, so an item approved today should not silently be regenerated later and treated as equivalent.
The study does not justify replacing trained item writers. Nor does it show that automated drafting saves time after review and revision are counted. A model that produces many plausible drafts may help a small faculty widen coverage, but a high rejection rate can move work rather than remove it. For examinations affecting progression, licensing or patient-facing competence, local governance should require human sign-off, a traceable evidence source and ordinary psychometric testing with learners.[1]
The limits are as important as the model ranking
Each model generated only one response for each of 60 tasks. The design therefore cannot estimate run-to-run variability or test-retest reliability. The ranking is time-sensitive because model versions and serving configurations change. Two experts supplied the subjective ratings, and the threshold for usability was an expert Likert score rather than observed learner performance. Pooling correlations across models may also reflect model-specific writing styles.
No students answered the questions, so the study cannot establish item difficulty, discrimination, fairness, retention or effects on learning. It did not test whether flawed questions would change clinical judgments or patient outcomes. The work was supported by Chinese public and university research programmes, and the authors declared no commercial or financial conflicts. Those disclosures reduce one concern but do not resolve the design limits.[1]
What would change the assessment
Evidence would strengthen if several independent generations per task produced stable results, if larger and more diverse clinical panels reached similar judgments and if educators measured the total time from prompt to approved item. A stronger validation would give reviewed questions to learners, estimate difficulty and discrimination, compare them with established human-written items and test whether any subgroup faced systematic ambiguity or disadvantage.
For now the defensible conclusion is narrow: structured prompting can make the surface form of specialist medical questions easy to reproduce, but educational readiness remains scarce. The most useful deployment is a transparent drafting workflow with expert accountability, not autonomous assessment production. Better model scores would change that view only if they were accompanied by repeatability, psychometric validity and evidence that review effort and error risk genuinely fall.[1]