Can AI agents configure complex contract workflows?
OpenAI reports that GPT-6 Astra scored 55.0% across 11 Ironclad tasks versus 41.6% for GPT-5.6 Sol, while simulated time fell from 37.0 to 19.2 minutes. The company-run benchmark shows progress—not autonomous legal work or measured customer productivity.
The short version
- What happened
- OpenAI and Ironclad defined 11 legal, commercial and procurement tasks, each assessed against 8 to 50 task-specific criteria.
- Who it affects
- Legal-operations and procurement staff may spend less time on repetitive configuration but more time specifying and validating rules.
- What remains unproven
- The evaluation covered 11 tasks on one contract-management platform and was designed with the platform provider.
Keep in mind
The evaluation covered 11 tasks on one contract-management platform and was designed with the platform provider.
The model developer trained and evaluated the systems; there is no reported peer review or independent replication.
Astra’s mean score was 55.0%, so many task criteria remained unmet even in the research environment.
Time per attempt was simulated and does not include human review, correction, deployment or rework.
The publication does not report repeated-run variance, a severity-weighted error analysis or live customer outcomes.
Source analysis · 1 linked source. Primary analysed source date: 6 October 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The answer is partial capability, not autonomous legal operations
GPT-6 Astra completed more of the specified requirements than GPT-5.6 Sol on an 11-task contracting benchmark, but its mean rubric score was 55.0%. That means the stronger model still missed a substantial share of the criteria in the evaluation designed by OpenAI with Ironclad. The result supports measurable progress on long, rule-bound computer tasks; it does not support handing an agent unsupervised authority over contracts, approvals or procurement.
The comparison is useful because the tasks went beyond drafting text. They required a model to configure forms, templates, approval rules and reusable clauses while preserving conditions across a workflow. Those are closer to operational work than a one-turn question. They are still research tasks inside a hosted software environment, not observations of live customer teams, signed agreements or financial and legal outcomes.[1]
Eleven tasks were scored against detailed requirements
Ironclad employees and people who use Ironclad at OpenAI helped identify 11 tasks spanning legal, commercial and procurement work. Examples included setting up a nondisclosure-agreement workflow, creating a procurement approval process and updating a reusable clause so it changed with a requester’s jurisdiction. OpenAI estimated that an experienced user would need about 30 to 40 minutes for each task.
Each task had between 8 and 50 criteria. This is more informative than declaring a task simply passed or failed: an agent may create the right form but miss a security review, spending threshold or jurisdiction-specific branch. In contract operations, one missed condition can matter more than several correctly completed cosmetic steps, so future reporting should distinguish critical controls from lower-consequence rubric items rather than relying only on an unweighted average.[1]
Astra improved the score, but both models were incomplete
OpenAI compared GPT-6 Astra at Max reasoning with GPT-5.6 Sol at High reasoning, the setting where each model performed best. Astra’s mean rubric score was 55.0%, compared with 41.6% for Sol—a relative increase of about 32%. An internal model used during Astra’s development scored 63.7%, but that model was not the deployed comparison and should not be treated as a product result.
A single illustrated task looked much stronger: Astra met about 94% of its criteria and Sol about 85%. That example cannot replace the 11-task mean. Highlighting the best-looking workflow may obscure harder failures elsewhere. Buyers and workers need the distribution of scores, repeated-run variability and the specific types of requirement missed, especially where an omission could bypass an approval or apply the wrong contractual term.[1]
The time result is simulated, not saved staff time
Estimated average time per attempt fell from 37.0 minutes for Sol to 19.2 minutes for Astra, a reported reduction of 48%. OpenAI explicitly says those figures are simulations based on assumed model processing and generation speeds. They are not stopwatch measurements of employees using the systems, and they exclude time required to review errors, correct configuration and investigate uncertain results.
That qualification changes the practical interpretation. A faster attempt that misses a material approval rule may create more downstream work than a slower human setup. A credible productivity study would measure the complete human-agent workflow: initial instruction, agent execution, review, correction, validation, deployment and later rework. It would also record error severity and whether reviewers detect every defect before a workflow reaches production.[1]
Training and evaluation were closely linked to one platform
Ironclad provided hosted environments in which the models could practise. OpenAI created synthetic training tasks around representative workflows and used reinforcement learning. It says the simulated contracts came from public SEC EDGAR filings after filters intended to remove personal information, and that it did not use OpenAI customer data, internal contracts or non-public Ironclad customer contracts for training or evaluation.
The partnership gives the benchmark realism, but also limits independence. The platform provider helped define the tasks and success criteria, while the model developer trained and evaluated the systems. There is no peer review, external replication or comparison with other agents and platforms. Performance may depend on interface structure, environment stability and familiarity created during training. Generalisation to another contract-management system remains untested.[1]
The practical impact is job redesign before job replacement
If performance improves, legal-operations and procurement staff may spend less time configuring repetitive forms and routing rules. Their work would shift toward specifying requirements, reviewing exceptions, testing workflows and deciding when automated action is inappropriate. The 55% average, however, is a reminder that domain expertise is still needed to recognise what the agent omitted and to understand the legal or commercial consequence.
Organisations should begin with bounded, reversible tasks and test the agent against historical scenarios before enabling production changes. Approval authority, audit logs, separation of duties and rollback should remain human-controlled. Staff need time and training for verification; presenting review as effortless would hide the most important new workload created by partial automation.[1]
What would change the assessment
Confidence would rise if OpenAI or an independent evaluator released the full task specifications, criterion weights, repeated-run results and error taxonomy. A held-out benchmark created after training would help separate genuine workflow reasoning from adaptation to a familiar task family. Cross-platform tests and comparisons with experienced users, other agents and non-agent automation would show where the capability actually adds value.
The decisive evidence is a prospective workplace study measuring total human time, critical error rates, caught and uncaught defects, rework, user trust and downstream contract outcomes. Results should be segmented by task complexity and reviewer expertise. Until then, the benchmark is evidence that a newer model handled more requirements with lower simulated time—not that AI has automated contracting or delivered customer productivity gains.[1]