Did the AI write a better clinic note?
A prospective single-centre study of 62 adults with abdominal pain found that AI-generated notes were more complete than clinicians' notes but scored worse on factuality. The assistant also produced only 50% diagnostic accuracy in the prospective cohort, so the result supports supervised documentation research—not autonomous clinical use.
The short version
- What happened
- The prospective cohort enrolled 62 adults attending four consultants at one Wuhan hospital with abdominal pain as their chief complaint.
- Who it affects
- Patients could benefit from fewer omitted history details, but unsupported additions could contaminate future care if they are not caught.
- What remains unproven
- The prospective study included 62 adults with abdominal pain at one hospital and was observational rather than randomised.
Keep in mind
The prospective study included 62 adults with abdominal pain at one hospital and was observational rather than randomised.
The development data and prospective cohort were Chinese-language records from the same institution, limiting transfer to other languages and workflows.
The theoretical completion rate assumes clinicians correctly follow every useful prompt; it is not a direct measure of the final signed record.
AI-generated notes scored lower for factuality and the prospective diagnostic accuracy was 50%.
The study did not measure consultation time, verification workload, treatment changes, adverse events or patient outcomes.
The transcription workflow required relatively clean, turn-taking speech without dialect or overlapping speakers.
Source analysis · 2 linked sources. Primary analysed source date: 9 October 2026. Earliest linked supporting source: 29 July 2025.
Read the full report · 9 min The evidence, methods, implications and limitations
The direct answer: more complete notes, but a serious factuality trade-off
The AI assistant produced outpatient records that clinicians rated as more complete than the records written without it, but those AI-generated records were also less factual. In the prospective cohort, the mean integrity score was 3.50 for AI-generated notes and 3.21 for human-generated notes. The mean factuality score moved in the opposite direction: 3.34 for AI-generated notes versus 4.01 for human-generated notes. Both differences were statistically significant. For a clinical record, an unsupported addition can matter more than a missing stylistic detail, so completeness cannot be treated as the sole definition of quality.
The system, called SMART-ASSISTANT, combined speech transcription, conversion of dialogue into a structured electronic medical record, prompts about missing information and a list of possible diagnoses. Its prospective diagnostic accuracy was 50%, lower than in the retrospective test. The authors attributed part of the drop to the consultants' cohort containing more rare and complicated disease. Whatever the reason, the finding shows why a good extraction score does not make a system a reliable diagnostic authority.
The appropriate conclusion is therefore narrow. This study shows that a fine-tuned assistant can capture and organise parts of a Mandarin outpatient conversation under supervision. It does not show that clinicians should accept generated notes without checking them, that the software reduces workload, or that it improves diagnosis or care. The factuality result instead makes human verification the central safety requirement.[1][2]
What was tested in the 62-person prospective cohort
Four consultants at Renmin Hospital of Wuhan University recruited consecutive eligible outpatients between 22 July and 15 September 2024. Participants were adults attending for a first visit with abdominal pain, without a previous definitive diagnosis, and able to consent. Sixty-two people entered the prospective cohort. This was an observational, single-centre evaluation rather than a randomised comparison of two care pathways.
The assistant transcribed consultation audio and converted the dialogue into structured records. Mean semantic textual similarity for transcription was 0.9253 and the character error rate was 0.2058. For recognising predefined quality-control points, accuracy was 94.40%. The authors also calculated a theoretical completion rate: they assumed a missing item would be completed whenever the system correctly prompted the clinician to ask about it. That rate rose from a human record completion measure of 59.14% to a theoretical 97.85% with prompts.
The word theoretical is important. The calculation did not itself prove that every prompt was followed correctly, that the answer was recorded accurately or that the added information changed a decision. A real implementation study would need to log accepted, edited and rejected suggestions, identify unsupported additions and compare the final signed record with the conversation and other source data.[2]
How the assistant was built and compared
Development used 1,249 anonymised records, 119 paired free-text and structured-record examples, and a dictionary of medical terms. The researchers evaluated three base large language models before and after task-specific fine-tuning and compared them with GPT-4 and DeepSeek on extraction of quality-control points. After fine-tuning, HuatuoGPT-II achieved the strongest reported quality-control accuracy and became the model inside SMART-ASSISTANT.
The workflow also included a multi-reader, multi-case study. Five attending physicians used the assistant's prompts to revise 125 records. Ten physician readers were divided into two reading orders and, after a three-week washout, rated original and AI-assisted records for completeness and diagnostic correlation. Ratings were higher for the assisted set. This design helps reduce simple order effects, but it still evaluates records produced from one institution's data and a carefully constructed research interface.
Audio processing was separately tested on 30 simulated conversations. The selected transcription pipeline used FunASR with entity correction and was then taken into the prospective cohort. The pipeline depended on relatively controlled speech: the authors said it required little background noise, no overlapping speakers, limited speaker numbers and no dialect. Those conditions may be difficult to maintain in busy clinics and reduce confidence that the same performance will transfer to other languages, accents and settings.[2]
Why the factuality score matters more than fluency
Readers found no statistically significant difference between AI- and human-generated records for normativity, readability or logicality. That apparent parity could be reassuring until factuality is examined. A note can be clear, well structured and clinically plausible while still containing a detail that the patient never said. Because later clinicians may treat the record as evidence, an invented symptom, negative finding or medication history can propagate through handovers and future decisions.
The authors explicitly acknowledged that the model could generate implausible information and illusory outputs when it had only the doctor-patient dialogue. Their evaluation penalised values inserted into fields when the source record contained no supporting information. That is a stronger approach than scoring surface quality alone, but the prospective denominator remains small and the paper does not report downstream correction rates or whether any unsupported item reached a signed record.
Safe use would require provenance and review at field level: users should be able to see which words came from the transcript, which were reformatted and which were generated or inferred. High-risk elements such as allergies, medication, examination findings and diagnostic claims need explicit confirmation. A system should also abstain when audio quality is poor or when its confidence cannot be calibrated, rather than filling a template because the interface expects a value.[2]
What the study means for patients, clinicians and hospitals
For clinicians, the strongest potential benefit is a structured prompt that makes omissions visible during a consultation. That may help a busy team remember important questions. But the study did not measure documentation time, after-hours work, interruption, alert fatigue or the time required to verify a generated note. A more complete draft that takes longer to check—or that encourages automation bias—may not reduce workload.
For patients, this evidence does not establish better diagnosis, faster treatment or safer follow-up. It covers adults with one chief complaint at one hospital and excludes many people who could complicate a first deployment. Patients should be told when audio is processed, how long it is retained, who can access it and how an inaccurate record can be corrected. Consent to a study is not the same as consent to routine recording across a health system.
For hospitals, the deployment question is not whether a model can draft a note. It is whether the full process produces a verified record with fewer clinically meaningful errors at acceptable cost and workload. Procurement should require local prospective evaluation, audit logs, security controls, subgroup analysis, an incident route and clear accountability for the signed record. The assistant should support the clinician, not obscure who made a clinical claim.[1][2]
What would change the assessment
Confidence would increase with a preregistered, multicentre trial that randomises comparable consultations to usual documentation or supervised AI assistance. The primary outcome should count clinically meaningful unsupported statements and omissions in final signed notes, assessed by independent reviewers against recordings and source data. Secondary outcomes should include time, after-hours work, edits, rejected suggestions, diagnostic and treatment changes, patient experience and safety incidents.
The model should be frozen before evaluation and tested across complaints, languages, accents, ages and noisy real clinics. Hospitals need to know how often it abstains, how performance changes when speakers overlap and whether clinicians catch errors under time pressure. Analysis should separate transcription errors from generation errors because the remedy is different: better audio cannot fix a model that invents facts, and better generation cannot recover words the microphone missed.
The research was supported by the Hubei Province Key Research and Development Program and a Wuhan University college-enterprise reform project; the detailed manuscript said the funder had no role and reported no competing interests. Independent replication would help test those findings. Until it exists, the balanced reading is that SMART-ASSISTANT improved one measure of completeness while exposing the more consequential weakness that a fluent note can still be less factual.[1][2]