Can an AI agent prepare clinical data safely? THReD succeeded often—but open-ended work remained harder
A synthetic benchmark based on clinical-data preparation reported 92.7% first-attempt task completion for one agent configuration across 141 tasks. Performance fell without documentation, and the study did not let the model handle real patient records or test clinical outcomes.
The short version
- What happened
- THReD contains 141 data-preparation tasks derived from 100 templates and seven categories. With repeats and four documentation settings, the reported evaluation covered 2,550 agent episodes.
- Who it affects
- Clinical researchers could automate repeatable preparation steps, but domain experts still need to approve cohort and outcome definitions that can change results.
- What remains unproven
- The paper is a preprint and has not been peer reviewed.
Keep in mind
The paper is a preprint and has not been peer reviewed.
One agent configuration was tested on synthetic tables designed by the study team.
The authors created the prompts, gold answers and automated checks, which can favour the benchmark's own assumptions.
Real clinical records were used for local tool validation, not given to the agent; no clinical outcomes were tested.
Eight injection tasks are too few to establish security, and one did not meet the pre-specified resistance criterion.
Source analysis · 2 linked sources. Primary analysed source date: 2 November 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The direct answer: the agent performed well on a bounded synthetic benchmark, not on live patient data
THReD is an attempt to test a less visible part of clinical AI: the work required before a model or statistician can analyse health records. Researchers routinely have to discover fields, reconcile coding systems, define cohorts, handle dates and missing values, and produce an auditable dataset. The benchmark assembled 141 tasks derived from 100 templates across seven categories. Repeated runs and four documentation conditions produced 2,550 evaluation episodes for one deployed agent configuration using Claude Haiku 5.5.
The agent completed 92.7% of tasks on its first attempt, with a reported 95% confidence interval of 88.1% to 96.4%. Allowing up to three conversational turns raised completion to 95.6%, and the paper reports 93.6% pass-at-five reliability across repeat trials. Those figures show that a tool-using model can execute many specified transformations in a controlled environment. They do not show that it can independently decide what a clinical study should measure, safely access a hospital system or produce a trustworthy research conclusion.[1][2]
What was tested and why documentation mattered
The tasks covered schema discovery, coding and label reconciliation, cohort construction, temporal alignment, data-quality checks, feature engineering and export. Synthetic tables were calibrated to patterns in HealthShare Exchange data, allowing the authors to create known answers without exposing patient records to the agent. A simulated researcher asked for a transformation, the agent selected tools or code, and rule-based checks compared the result with a gold answer.
Four context settings varied how much reference material the agent received. With the intended documentation, examples and tool descriptions, performance was high. In the no-context condition, success fell to 58%. That difference is operationally important. It suggests that the surrounding data dictionary, interface contract and examples are part of the system, not optional prose added after deployment. A hospital with inconsistent field definitions should not expect a general model to infer local meaning reliably.
The most open-ended goal-level tasks were harder, with 80.6% completion. These tasks required the agent to work from a research objective rather than follow a tightly specified operation. That gap matches a familiar human distinction: converting a clear rule into code is different from deciding which rule is scientifically justified. Clinical governance becomes more important, not less, as instructions move from exact transformations to broad analytical goals.[1][2]
The distinction that matters
What the headline percentages do—and do not—measure
THReD separates a synthetic agent benchmark from real-record validation of the underlying tools.
2,550 episodes
Synthetic agent-task evaluations
Known answers, no patient records sent to model
92.7%
First-attempt task completion
One agent configuration in the benchmark
58%
Success without reference context
Documentation materially affected performance
Real records
Local tool and pipeline validation
Not an agent test on patient data
The tool layer was checked on real records, but the model was kept away from them
The authors separately validated parts of the tool layer against real electronic health records from two chronic-disease cohorts. The tools reconstructed patient classes at roughly 91% to 92% and retained 99.7% to 99.8% of vital-sign labels. A terminology exercise began with 2,208 raw variants and mapped 99% of them. These measurements support the usefulness of the underlying preprocessing utilities, but they are not an agent accuracy score on real patients.
For reproducibility, the team replayed 2,176 executions from 128 pipelines and obtained byte-identical outputs when dependencies were pinned. A change in the pandas software version altered results until the environment was fixed. That is a valuable warning for any research group: a fluent explanation from an agent does not guarantee that a pipeline will reproduce later. Software versions, parameters, inputs and intermediate outputs need to be recorded alongside the conversation.
The authors also show why preprocessing choices can affect a substantive conclusion. Different ways of consolidating a cohort changed the estimated 90-day emergency-department revisit rate by as much as 3.2 percentage points. The benchmark does not say which estimate is clinically correct. It demonstrates that apparently technical choices can move an outcome enough to matter, so a domain expert must approve definitions before an agent automates them.[1][2]
Prompt injection and privacy need a cautious reading
THReD included eight tasks with malicious instructions planted in data or context. Under the authors' pre-specified resistance criterion, the agent resisted seven of eight. The paper also reports that it did not propose the particular harmful action requested by the planted text. With only eight tasks, neither statement establishes robust security. A single missed injection can be consequential in a system that can read files, alter data or export results.
The synthetic design reduces privacy risk and makes exact scoring possible. It also leaves open how the agent would behave with messy, rare or contradictory patterns in live clinical records. In the real-data tool-validation experiments, records remained within the local environment and were not sent to the language model. This distinction should remain visible: the paper validates a benchmark and a tool-assisted workflow, not a permission model for sending identifiable health information to a commercial model.
Any operational pilot would need least-privilege tool access, approved data boundaries, logging, independent output checks and a clear human owner for cohort definitions. It should also test attacks that are tailored to the local tools rather than relying on eight benchmark prompts. Privacy review, data-use agreements and security assessment remain prerequisites.[1][2]
Evidence limits, funding and what would change the assessment
The paper is an unreviewed preprint submitted on 2 November 2026. One model configuration was evaluated, and the authors designed the prompts, tasks, gold answers and checks. The simulated researcher followed a fixed interaction pattern. The synthetic tables cannot reproduce every ambiguity in real clinical data, and the real-data experiments sampled 5,000 records per cohort for some analyses. Multiple subgroup comparisons were not adjusted, while the terminology reference relied on one expert plus model-assisted annotation.
The authors report funding from the Robert Wood Johnson Foundation Health Data for Action programme, grant 78961, and disclose no conflicts of interest. They also disclose using an AI writing assistant. The persistent Zenodo archive helps others inspect or reproduce the benchmark, but open materials do not substitute for independent replication or prospective evaluation.
The assessment would strengthen if independent hospitals ran the benchmark on additional models and data systems, if experts blinded to agent condition judged scientific validity, and if prospective studies measured time saved, correction burden, security events and downstream research errors. For now, THReD supports a bounded conclusion: a documented tool environment let one agent complete many synthetic preprocessing tasks, while missing context and open-ended goals exposed clear weaknesses. That is useful current research, not proof of autonomous clinical-data handling and not a Breaking development.[1][2]