Can AI models answer infectious-disease pharmacy exams? Five scored near 90%—but images and harder reasoning exposed gaps
A peer-reviewed comparison gave five commercial language models the same 227 questions from one US Doctor of Pharmacy course. Overall scores clustered between 89.4% and 91.2%, while image questions and the highest Bloom category were materially weaker.
The short version
- What happened
- Five model interfaces answered the same 227 course questions once, producing 1,135 scored responses; overall accuracy was 90.2% and individual model totals ranged from 89.4% to 91.2%.
- Who it affects
- Pharmacy students may use model errors as learning material, but need instructors to check rationales and protect assessment integrity.
- What remains unproven
- The study used 227 questions from one infectious-disease course at one US pharmacy school, limiting generalisability across curricula, countries and clinical settings.
Keep in mind
The study used 227 questions from one infectious-disease course at one US pharmacy school, limiting generalisability across curricula, countries and clinical settings.
Five proprietary interfaces were tested once per item from May to July 2025; model versions, routing and behaviour can change.
Correctness depended on the course instructor's answer key, and the paper did not test guideline retrieval, citations, confidence, abstention or repeated prompt variability.
Only 13 unique questions contained an image and only nine concerned pharmacokinetics or pharmacodynamics, making those subgroup estimates comparatively fragile.
The models may have encountered similar educational material during training; the study did not measure contamination or use a newly created external hidden test set.
No clinicians, patients, clinical workflows or health outcomes were studied.
Source analysis · 1 linked source. Primary analysed source date: 7 November 2026.
Read the full report · 9 min The evidence, methods, implications and limitations
The answer: the five models were similarly strong on this exam set, but the average hides important failure patterns
Five commercial language-model interfaces each answered 227 infectious-disease questions from a required Doctor of Pharmacy course at the University of the Pacific in California. Across the resulting 1,135 model-question attempts, 1,024 were marked correct, an overall accuracy of 90.2%. Individual totals were tightly grouped: ChatGPT-4o-mini answered 207 of 227 correctly, or 91.2%; Gemini 2.5 Flash scored 206, or 90.7%; ChatGPT-4.1 scored 205, or 90.3%; and both Copilot and Claude Sonnet 4 scored 203, or 89.4%.
That clustering is useful because it argues against declaring a decisive winner from this test. A difference of four correct answers separates the highest and lowest totals, and the paper reports no broad model-level superiority. The consequential result is where performance fell. Across models, image-containing items produced 45 correct responses from 65 attempts, 69.2%, versus 979 correct from 1,070 attempts without an image, 91.5%. Questions classed as evaluate/create also scored below remember/understand questions.
The evidence does not show that any model can practise infectious-disease pharmacy safely. These were course questions with an instructor answer key, not patient cases followed through diagnosis, prescribing, monitoring and outcomes. A high average can coexist with a dangerous error on a dose, contraindication, resistance pattern or image. The fair conclusion is that the tested models recalled much of this curriculum, while harder reasoning and visual information exposed gaps that educators and clinicians cannot ignore.[1]
What the researchers actually tested
The source was an Integrated Clinical Sciences infectious-diseases course taken in the second professional year of an accelerated three-year Doctor of Pharmacy programme. The course ran from January to April 2025, and the researchers collected model answers from May through July 2025. The tested interfaces were ChatGPT-4.1 and ChatGPT-4o-mini from OpenAI, Claude Sonnet 4 from Anthropic, Microsoft Copilot accessed through web, mobile or Edge, and Gemini 2.5 Flash from Google.
Every model received the same instruction: answer the question, explain why the selected answer was correct and why the alternatives were incorrect. The first response was recorded; a question incorrectly answered by any model was independently reviewed by two investigators, who agreed on every classification. The answer key supplied by the course instructor determined correctness. The analysis used descriptive statistics, chi-square comparisons and Cohen's kappa to examine agreement among model responses.
The design therefore offers a controlled cross-model snapshot. It does not test prompt sensitivity, repeated runs, conversation history or whether giving the models access to guidelines would change their answers. It also cannot separate learned medical knowledge from possible exposure to similar educational material during training. Because commercial systems change, the named products should be read as the versions and interfaces available during the 2025 collection window, not as permanent capabilities of their brands.[1]
The distinction that matters
How the headline score was built
The 90.2% overall figure combines five answers to each of 227 questions. The smaller subgroups need their own denominators.
Question set
227 course items
One US pharmacy programme
Model attempts
Five models answer each item
1,135 scored responses
Overall result
1,024 responses correct
90.2% attempt-level accuracy
Critical gaps
Separate images and harder reasoning
Lower subgroup performance
The denominator shows where the exam was concentrated
The 227 questions were unevenly distributed across five content areas. Therapeutics accounted for 108 questions, medicinal chemistry for 45, clinical microbiology for 40, pharmacology for 25 and pharmacokinetics or pharmacodynamics for nine. Multiplying each question by five models produced 540 therapeutics attempts, 225 medicinal-chemistry attempts, 200 microbiology attempts, 125 pharmacology attempts and only 45 pharmacokinetic or pharmacodynamic attempts.
Across all five systems, pharmacology had the highest accuracy at 117 correct responses out of 125 attempts, or 93.6%. Therapeutics followed at 502 of 540, or 93.0%; pharmacokinetics or pharmacodynamics at 41 of 45, or 91.1%; clinical microbiology at 175 of 200, or 87.5%; and medicinal chemistry at 189 of 225, or 84.0%. The paper reports that medicinal chemistry and microbiology were significantly lower than therapeutics, while pharmacology and pharmacokinetics were not.
Those percentages are attempt-level aggregates, not independent samples of 1,135 unrelated clinical problems. Each original question contributes five correlated responses because every model saw it. The pharmacokinetic result rests on nine unique questions, so its 91.1% headline is less stable than the therapeutics result from 108 questions. Readers should keep both the question count and the model-attempt count visible rather than treating every percentage as equally precise.[1]
Higher-order questions and images were the clearest warning signs
The researchers grouped the exam using Bloom's taxonomy: 92 questions were remember/understand, 71 apply/analyse and 64 evaluate/create. Across five models, that produced 460, 355 and 320 attempts respectively. Remember/understand questions yielded 423 correct answers, or 92.0%. Apply/analyse yielded 327, or 92.1%. Evaluate/create fell to 274 correct answers, or 85.6%, a statistically significant reduction relative to remember/understand.
The visual comparison was smaller but sharper. Thirteen of the 227 questions included an image, creating 65 model attempts. Only 45 were answered correctly. The remaining 214 questions created 1,070 attempts, of which 979 were correct. The paper reports the 69.2% versus 91.5% difference as statistically significant. Because only 13 unique image questions were used, the result should be treated as a warning that needs replication, not a precise universal estimate of multimodal medical performance.
Inter-model agreement was moderate, with pairwise Cohen's kappa values from 0.497 to 0.613. Similar totals therefore did not mean the systems failed on exactly the same questions. For educators, disagreement can identify items worth discussing rather than justify replacing assessment. For clinicians, it is a reminder that consulting several models does not automatically create independent verification: they may share training sources and failure modes, while moderate agreement provides no guarantee that a majority answer is correct.[1]
What this means for pharmacy teaching and clinical work
For pharmacy educators, the study supports using model answers as material to inspect, challenge and correct. Students can compare a rationale with guidelines, identify omitted patient factors and explain why an attractive distractor is wrong. The weakest categories suggest where supervision matters most: image interpretation, medicinal chemistry, microbiology and questions requiring evaluation or creation. A 90% average is not a reason to outsource grading or permit unsupervised advice.
For practising pharmacists and other clinicians, the study is much further from deployment evidence. Course questions are bounded, usually contain the information required and have a predetermined answer. Real care involves incomplete histories, local resistance data, allergies, interactions, renal and hepatic function, changing guidance and responsibility for follow-up. The paper did not measure citation accuracy, calibration, hallucination, abstention, dose calculation, guideline retrieval, patient communication or harm.
People seeking health information should not infer that a chatbot has a nine-in-ten chance of giving safe infectious-disease advice. The denominator is exam responses, not patients, and a wrong answer may have very different consequences depending on the topic. The practical safeguard remains qualified professional review with access to current local evidence. Model output can support learning or drafting only when a person with appropriate expertise checks the reasoning and the final action.[1]
What would change the assessment
Confidence would rise with preregistered replication across several pharmacy schools, countries and languages, using larger banks of unseen questions and stable archived model versions. Researchers should report repeated runs, prompt variants, confidence or abstention and item-level error types. A separate external answer panel could reduce dependence on one course key, while tests using current guidelines could reveal whether retrieval improves accuracy or merely adds plausible citations.
The assessment would become clinically consequential only if models were tested prospectively on realistic cases with missing and conflicting information, compared with pharmacists and trainees, and evaluated for safe recommendations rather than multiple-choice selection. Useful outcomes would include severe-error rates, dosing and interaction mistakes, sensitivity to images and laboratory data, appropriate escalation, calibration and the effect on human decision time and quality. Patient outcomes would require a much more cautious study design.
For now, the durable finding is a bounded one: five widely used model interfaces produced similar high averages on one infectious-disease pharmacy exam set, but performance fell on visual items and the highest-order reasoning category. The paper gives educators a concrete map of strengths and weaknesses. It does not establish clinical competence, and it should not be used to lower the standard of human oversight.[1]