Did AI patients train medical students as well as human actors?
A three-arm pilot randomized 61 students and analysed 58. All groups improved on the main communication checklist, with no detectable between-group difference—but the study was too small to establish equivalence and compared text-based AI with voice-based human practice.
The short version
- What happened
- The study combined 350 blinded expert assessments across five language models, 155 live AI consultations by 31 students and a three-arm pilot randomized study that analysed 58 participants.
- Who it affects
- Medical students could practise difficult questions on demand and educators could see which safety-critical details were missed. Patients would benefit only if that practice later improves real consultations; this study did not test that transfer.
- What remains unproven
- The pilot analysed 58 participants and was underpowered for small between-group differences or an equivalence claim.
Keep in mind
The pilot analysed 58 participants and was underpowered for small between-group differences or an equivalence claim.
AI training used text chat while human standardized patients used voice calls, so agent and communication modality cannot be separated.
Training and assessment reused the same five scenarios, increasing familiarity and potentially compressing differences between arms.
The single-institution Chinese-language study measured immediate performance, not retention, real-patient communication or clinical outcomes. The global rating scale had poor reliability (ICC 0.15).
The educational study was not prospectively registered. The authors report no specific funding and no competing interests.
Source analysis · 1 linked source. Primary analysed source date: 23 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The pilot found comparable gains, not equivalent training
A peer-reviewed Chinese study found no detectable difference in immediate communication-skill gains among students assigned to text-based AI patients, voice calls with human standardized patients or the usual curriculum. Sixty-one medical students were randomized; three withdrew before training, leaving 21 in the AI arm, 18 in the human-patient arm and 19 controls. Blinded examiners scored recorded consultations.
All three groups improved on the prespecified 14-item checklist, whose inter-rater reliability was acceptable at ICC 0.72. The between-group comparison was not significant (p=0.483), and baseline-adjusted analysis reached the same conclusion. But this was a pilot: post-hoc power was below 25% for the observed between-group effect, and the authors estimate at least 216 participants would be needed for 80% power. A null result therefore cannot show that AI and human practice are equivalent.[1]
Structured disclosure made the simulation responsive to better questioning
The system withheld some clinically important details until a student asked directly or used empathetic probing. In an earlier live phase, 31 students completed five consultations each, for 155 sessions, and discovered 65.6% of hidden items. The wider validation included 350 blinded expert assessments across five models, but every model used the same architecture, so the study cannot isolate the contribution of that design from prompting, persona writing or language style.
For educators, AI patients could make practice available without booking actors or simulation rooms. Yet the trial reused the same five scenarios for training and assessment, tested immediate outcomes at one Chinese institution and entangled agent with modality: AI was text-based, people used voice. The study did not measure retention, real-patient communication or clinical outcomes. Before replacing human practice, a larger registered multicentre trial would need new assessment cases, matched modalities and longer follow-up.[1]
The distinction that matters
What each phase established—and what it did not
The programme moved from scripted validation to live use and then a pilot comparison.
Phase 1
Seven educators score 350 scripted assessments across five models
Tests fidelity, not learning
Phase 1b
31 students complete 155 AI consultations
65.6% of hidden items discovered
Phase 2
Randomize 61 students; analyse 58
No detectable checklist difference; equivalence remains unproven