Can AI grade medical students without hiding mistakes?
A prospective US deployment processed 72,907 OSCE rubric scores for 222 students and sharply reduced human scoring passes, but validation focused on the low-scoring tail and physician adjudicators could see which score came from AI.
The short version
- What happened
- MAPLES produced 72,907 retained item-level scores for 222 students across written notes, audio and video during a Fall 2025 pre-clerkship OSCE deployment.
- Who it affects
- Students could receive faster scoring, but a hidden error could affect remediation or progression if review is too narrowly routed.
- What remains unproven
- The deployment involved one US medical centre and one Fall 2025 pre-clerkship OSCE context.
Keep in mind
The deployment involved one US medical centre and one Fall 2025 pre-clerkship OSCE context.
Independent human rescoring focused on the routed low-scoring tail rather than a random sample of all 72,907 retained item scores.
The 616-item physician adjudication set was selected, and adjudicators could see which score came from AI and which from the standardized-patient evaluator.
The 92.3% reduction compares scoring-pass counts with a modelled single-pass manual workflow, not measured staff time or total cost in a parallel trial.
Only 33 video disagreements were included in the retained adjudication set, limiting precision for that modality.
UT Southwestern filed patent applications related to the technology, with several authors named as inventors.
Source analysis · 1 linked source. Primary analysed source date: 7 October 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The direct answer: it reduced workload, but did not remove the need for human judgement
A multimodal AI system handled the first scoring pass for a real medical-school examination and routed lower-scoring cases to people, showing that this workflow can operate at meaningful scale. In the Fall 2025 deployment at UT Southwestern Medical Center, MAPLES processed written notes, audio and video from 222 students and retained 72,907 rubric-item scores. The reported number of human scoring passes was 92.3% lower than a modelled workflow in which a person scored every retained item once.
That is an operational result, not proof that AI can safely become an autonomous examiner. The study concentrated human validation in the low-scoring tail, where errors could affect remediation or progression. It did not obtain an independent human reference score for every one of the 72,907 items. The defensible conclusion is therefore that AI can triage and score within a supervised local workflow, while the fairness and accuracy of the much larger unreviewed set remain less directly observed.[1]
How the routed assessment worked
MAPLES is a rubric-driven, zero-shot multimodal large-language-model pipeline. It scored post-encounter notes, audio-derived communication evidence and video evidence of physical examination against item-specific criteria. Students whose aggregate scores fell below prespecified thresholds were routed for an independent rescore by standardized-patient evaluators. Those evaluators were blinded to AI scores, model rationales, aggregate scores and the reason for routing, and did not rescore encounters in which they had acted as the simulated patient.
The routed review contained 4,973 independently rescored items. Tolerant AI–human agreement ranged from 85.7% to 92.4% across modalities. Tolerant agreement allows a defined amount of score difference, so it should not be confused with exact identity. It is relevant to a rubric where adjacent scores may have similar educational meaning, but institutions need to decide in advance which disagreements can safely be tolerated and which can alter a student's outcome.[1]
The physician comparison favoured AI, with an important source of bias
Items outside the tolerance rule were escalated. After excluding disagreements attributed to rubric or data-input problems, the authors retained 616 genuine disagreements for the main physician-adjudication analysis: 376 note items, 207 audio items and 33 video items. Physicians matched the AI score on 76.0% of those items and the standardized-patient evaluator on 19.0%; the remainder matched neither. This suggests that AI often identified scoring evidence missed or interpreted differently by the first human reviewer.
However, physicians could see both scores and their sources. The adjudication was therefore selected and non-blinded. Knowing which value came from AI can influence judgement in either direction, especially in a team that designed and deployed the system. The 616 items also represent disagreements from a routed subset, not a random sample of all outputs. A stronger test would hide score provenance, sample cases across the full score distribution and use several independent adjudicators with prespecified tie-breaking rules.[1]
A 92.3% workload reduction is a modelled comparison
The AI-assisted workflow required 5,616 human scoring passes: 4,973 standardized-patient rescoring passes and 643 physician adjudication passes. The counterfactual comparator assumed one human pass for each of the 72,907 retained items. Against that benchmark, human passes fell by 92.3%, with reported reductions of 94.0% for notes, 88.3% for audio and 88.9% for video. That could release educator time for feedback, remediation and quality assurance.
It is not a measured time-and-cost trial against a parallel manual examination. A human pass on a simple checklist item and a physician adjudication are not interchangeable units of labour, and model operation, data preparation, oversight, appeals and system maintenance also consume resources. The study supports a plausible efficiency gain, but procurement decisions need elapsed time, staff grades, computing costs, failure recovery and the cost of correcting consequential errors.[1]
What this means for students and assessment leaders
For students, the central issue is not whether AI participates but whether the assessment remains contestable and educationally valid. Learners should know which evidence was processed, when a human reviews the result, how to request reconsideration and whether recordings are retained for model development. A low aggregate score should trigger careful review rather than automated labelling. Institutions also need subgroup audits to detect whether accent, speech pattern, disability, skin tone, camera framing or documentation style changes error rates.
Assessment leaders can treat the paper as evidence for a bounded pilot: AI first pass, prespecified routing, blinded secondary review where possible, audit samples from every score band and final human authority over high-stakes decisions. The system was evaluated at one US medical centre in one pre-clerkship OSCE context. Different stations, rubrics, languages, licensing standards and recording conditions could change performance. Local validation is not optional simply because the architecture is described as zero-shot.[1]
Commercial interests and the evidence that should come next
The work was supported by UT Southwestern institutional funds. The authors disclose that UT Southwestern filed patent applications related to AI-based assessment technologies described in the work and that several authors are named inventors. This does not negate the results, but it raises the value of independent replication and blinded adjudication. The authors also describe plans to make tools available for non-commercial academic use, which could help other teams reproduce the workflow.
The assessment would change most with a preregistered multicentre study that independently scores a random sample from the entire distribution, masks adjudicators to provenance and reports exact as well as tolerant agreement, pass/fail changes, appeals, subgroup errors and downstream remediation. A parallel resource evaluation should measure staff time and cost rather than infer them from scoring-pass counts. Until then, MAPLES is credible evidence that supervised AI scoring can be embedded in one medical school—not proof that an AI examiner is ready to make final decisions about students.[1]