Can consistent AI replace sarcoma tumour-board judgement?
No. A structured Claude workflow reproduced 94% of 1,071 decision-code instances across repeated runs, but the study did not test whether those recommendations were clinically correct or improved care.
The short version
- What happened
- The pilot processed 51 de-identified sarcoma cases in 154 runs and compared 21 decision codes per case, producing 1,071 case-code instances.
- Who it affects
- Specialists may gain a more auditable research interface for testing AI recommendations before any clinical trial.
- What remains unproven
- The primary endpoint was exact repeated-output stability, not clinical correctness or guideline concordance.
Keep in mind
The primary endpoint was exact repeated-output stability, not clinical correctness or guideline concordance.
Only Claude Opus 4.5 and one final prompt/schema were evaluated on 51 cases from one institution.
Ten cases informed framework development, and the remaining 41 were not a prospectively declared held-out set.
A single reviewer screened outputs for hallucination categories; multi-rater clinical adjudication was not performed.
Unsuccessful schema-validation attempts were not logged, so reliability of all attempted invocations cannot be calculated.
No patient outcome, workflow burden, treatment decision or specialist-panel comparison was measured.
Source analysis · 1 linked source. Primary analysed source date: 11 October 2026.
Read the full report · 6 min The evidence, methods, implications and limitations
The answer: structured output made the model steadier, but steadiness is not medical validity
A Claude Opus 4.5 workflow produced the same value across repeated runs for 1,007 of 1,071 case-level decision codes, an overall reproducibility estimate of 94.0% with a cluster-bootstrap 95% confidence interval of 92.0% to 95.8%. That is evidence that a constrained schema, explicit semantics and conditional rules can reduce run-to-run variation in a clinical simulation. It does not establish that the recommendations were appropriate.
A system can consistently return the same wrong answer. The researchers deliberately framed this as a methodological pilot and made concordance with historical multidisciplinary-team decisions follow-up work. No clinician panel scored the recommendations for correctness, no patient entered an AI-guided pathway, and no outcome improved. The immediate result is therefore about building an auditable experimental instrument for later comparison—not about replacing surgeons, oncologists, radiologists, pathologists or specialist nurses.[1]
How the 51-case experiment was organised
The dataset contained 51 fully de-identified sarcoma cases spanning 17 histological subtypes from 2020 to 2025. Ten cases were used during three framework-development phases. The final schema and prompt were then frozen before the remaining 41 cases were processed, although development was not prospectively logged and the authors appropriately avoid calling those 41 a formal held-out test set. All cases were later run under the same final version.
Fifty cases had three completed runs and one case had four, giving 154 analysed outputs. Claude Opus 4.5 was accessed through the official SDK at temperature zero with a maximum of 4,000 output tokens. Tool-use output was validated against a Pydantic schema; all saved runs returned the required block and passed validation. The prompt encoded meanings and conditional relationships across surgical, reconstructive, radiotherapy, imaging, staging and systemic-treatment fields.[1]
What 94% reproducibility means—and what it hides
The denominator was 51 cases multiplied by 21 decision codes. A code counted as stable only when every available run for that case returned exactly the same value. There was no case folding, whitespace harmonisation or semantic matching, so equivalent free-text wording could count as unstable. Reproducibility was 94.2% among the 41 cases not used to specify the framework and 93.3% among the ten development cases. Seventeen cases, one third of the total, were stable on every code.
Variation was not evenly distributed. Reconstruction decisions were stable in 72.5% of case-code comparisons, free-text staging recommendations in 76.5%, and radiotherapy dose specifications in 86.3%. The lowest-stability case reached 61.9% and involved an older patient for whom outputs alternated among observation, wide excision and different reconstruction options. These are not trivial formatting differences; they sit close to choices that would require specialist deliberation, patient preferences and knowledge of local capability.[1]
The ablation shows where rules helped
A post-hoc ablation reran 15 cases with and without the semantic and conditional-logic rules. Reproducibility was 94.6% with the full framework and 82.5% without it, a 12.1-percentage-point difference with a 95% confidence interval of 7.6 to 17.1 points. Most of the improvement came from exact wording in three free-text fields, which rose by 60 points. Across the 18 categorical and numeric fields, the difference was 4.1 points and its confidence interval crossed zero.
That nuance prevents an exaggerated conclusion. Schema enforcement clearly standardised the container and language, while the evidence for better stability in coded clinical choices was smaller. Structured output still has practical value: it makes missing fields, impossible combinations and version changes easier to audit. But schema conformity is a software property, not a guarantee that a treatment plan matches guidelines, multidisciplinary judgement or an individual patient's goals.[1]
What this means for clinicians and patients
For researchers, the framework offers a way to measure a model's behaviour repeatedly before asking experts to judge it. Future evaluations can lock the model and prompt, compare outputs with documented tumour-board decisions, inspect disagreement by speciality and test multiple providers. Hospitals considering decision-support research should also log prompt versions, model identifiers, API failures, schema validation, manual overrides and the provenance of every case field.
For clinicians and patients, nothing in this pilot supports using the output as an independent recommendation. Sarcoma care is uncommon, multidisciplinary and preference-sensitive; reconstruction and radiotherapy—the areas with greatest remaining variation—depend on anatomy, function, timing, prior treatment and local expertise. A safe study would present outputs to a specialist team as research material, preserve ordinary care, and measure whether the system adds information or merely creates extra review work.[1]
Limits, disclosures and evidence that would change the assessment
The study used one proprietary model, one institution's de-identified cases and one final prompt. Temperature zero reduces sampling variation but does not guarantee determinism. Failed schema-validation attempts were not logged because they produced no saved file, so the analysed denominator describes successful outputs. Hallucination screening was performed by a single author, and absence of detected fabricated content under five categories does not prove that every coded recommendation was factually or clinically correct.
The model was not compared with a multidisciplinary panel, guidelines, alternative models or patient outcomes. Source cases remain at the institution, limiting independent case-level replication, although aggregated results and framework materials are reported as deposited. Confidence would rise with prospectively logged, multi-centre, multi-model evaluation; blinded expert adjudication; predefined safety-critical disagreement rules; and workflow studies. Evidence of improved plan quality without added delay or automation bias would be required before considering clinical benefit.[1]