Can AI agents read routine wet-lab evidence? A 100-task benchmark left Astra short of full credit
OpenAI's GPT-6 Astra earned 66.7% of available credit at xhigh effort across gels, microscopy, sequencing and other raw evidence, versus 43.8% for GPT-5.6 Sol. The company benchmark shows progress, but its own examples still include wrong controls, reversed interpretations and inaccurate colony counts.
The short version
- What happened
- BioBasicsBench contains 100 expert-reviewed tasks spanning five areas of routine wet-lab work and multiple raw evidence formats, from gels and microscopy to sequencing files and protocol videos.
- Who it affects
- Wet-lab scientists may gain help inspecting raw files and troubleshooting controls, but they still need to verify observations and retain authority over experimental decisions.
- What remains unproven
- The benchmark was developed by OpenAI with Ligo and evaluated only OpenAI models; there is no independent provider comparison.
Keep in mind
The benchmark was developed by OpenAI with Ligo and evaluated only OpenAI models; there is no independent provider comparison.
The public report does not provide a human baseline, inter-rater reliability or an independently peer-reviewed methods paper.
Selected examples illustrate strengths and failures but do not give a complete error-type denominator.
Mean rubric reward is not a laboratory success, safety or discovery rate.
No prospective study measured real team productivity, experimental validity, biosafety or scientific outcomes.
Source analysis · 1 linked source. Primary analysed source date: 6 November 2026.
Read the full report · 9 min The evidence, methods, implications and limitations
The result: a stronger score, not a reliable laboratory scientist
OpenAI's new BioBasicsBench asks AI agents to interpret the untidy evidence that working scientists see before a figure reaches a paper. At the highest reasoning setting reported for all seven tested models, GPT-6 Astra earned 66.7% of the available rubric credit. GPT-6.1 Sol scored 62.0%, while GPT-5.6 Sol earned 43.8%. The whole-task bootstrap interval for Astra ran from 60.3% to 73.1%; GPT-5.6 Sol's ran from 37.6% to 50.0%. That is a material within-company model improvement on the benchmark, but it also means the top system left roughly a third of available credit unearned.
The company is explicit that a higher score does not mean an agent can make reliable judgements in a laboratory. Its published failure cases include choosing the wrong colonies after reversing the meaning of control bands, discounting microscope regions that contradicted a preferred explanation and overcounting colonies by about 12% on one plate. Those are not cosmetic errors. Selecting the wrong construct can waste downstream purification work, and a colony-count error directly changes an estimate of transformation efficiency when the dilution and plated volume are fixed.[1]
What the 100 tasks contain
The benchmark covers protein methods, imaging and cytometry, quantification and assays, nucleic-acid methods, and molecular cloning. Inputs preserve raw laboratory formats rather than polished publication figures. Thirty-one tasks contain gels or blots, 27 include protocols, notes or specifications, 20 use tables or instrument exports, 18 include photographs of benches or reagents, 14 use microscopy, 13 use instrument screens or plots and 12 include sequencing files. Seven tasks contain protocol videos and seven use agar plates or colonies. These categories overlap, so they cannot be added to produce a larger denominator than the 100 unique tasks.
The task range matters because laboratory interpretation is often multimodal and contextual. An agent may need to read a handwritten label, match a lane to a ladder, notice that an antibiotic does not fit the experiment, interpret a time-lapse or inspect raw flow-cytometry data. Reference sheets such as plasmid maps and pH charts are supplied where relevant. The benchmark therefore tests more than factual recall, but it remains a selected task collection. It does not represent every organism, instrument, protocol, image quality, safety regime or tacit practice found across real laboratories.[1]
The distinction that matters
How to read the 66.7% result
The score is rubric credit across a selected benchmark, not a laboratory success rate.
Tasks
Interpret 100 raw wet-lab problems
Five overlapping laboratory domains
Runs
Attempt each task five times per setting
18,500 results across 37 settings
Score
Award partial credit against expert rubrics
66.7% mean reward for Astra xhigh
Limit
Inspect selected failure cases
No proof of safe autonomous lab work
How tasks were written, reviewed and scored
Scientists created tasks from their own experiments and paired each question with an expert-written grading rubric. A Ligo scientist who had not written the task reviewed it, followed by an external PhD-level scientist working in a wet lab and then a final internal review. Reviewers examined the prompt, experimental evidence, rubric, author notes and a graded model attempt. Answers were structured to provide the requested conclusion, measurement or recommendation and, where required, supporting reasoning. Rubric criteria could reward a conclusion, numerical result, explanation or recognition that the available evidence was insufficient.
OpenAI evaluated seven models across 37 model-and-effort settings. Each setting attempted all 100 tasks five times, producing 18,500 scored results. The reported mean reward is the average percentage of available rubric credit. Confidence intervals resampled whole tasks while keeping their five attempts together, which treats the task—not each attempt—as the independent unit. That is a sensible way to avoid pretending repeated answers are 500 independent laboratory problems. The report does not provide inter-rater reliability, a human baseline, an external model comparison or an independent audit of the rubrics.[1]
Where the newer model appeared to improve
OpenAI highlights three types of improvement over GPT-5.6 Sol: respecting the limits of evidence, handling multimodal detail and keeping experimental context active. In one microscopy example, two fluorescent detection windows overlapped. GPT-5.6 Sol treated brightness in a dead-cell dye channel as proof that cells were dead; Astra noticed that another dye could contribute signal and asked for controls. In a cloning example, Astra read a plate rim labelled for kanamycin and rejected it as a valid control for an experiment using ampicillin. In a depth scan, it recognised cells moving in and out of focus rather than inventing a biological disappearance.
These examples clarify what the benchmark is trying to reward: disciplined interpretation rather than a plausible narrative. They do not establish how often each improvement occurred across the 100 tasks, because the public report presents selected cases rather than a full error taxonomy with denominators. They also compare generations from one provider. A model can improve because of better visual perception, stronger reasoning, benchmark familiarity or changes in the evaluation harness; the design does not isolate those mechanisms. The examples are useful diagnostics, not causal evidence about why Astra scored higher.[1]
The remaining failures are exactly where oversight matters
Astra sometimes made a correct intermediate observation and then failed to use it. In the reported cloning case, it separated band groups visually but assigned the experimental identities in reverse, selecting lanes 4, 6, 7 and 8 instead of the reference selection of 5, 9 and 10. In a microscope image, it noted that contrast varied with focus yet generalized that vacuoles were darker, discounting lighter regions visible in the same field. Its colony-count marks missed visible colonies, merged close neighbours and included empty agar. The net count was approximately 12% above the reference.
For working scientists, this pattern is more important than the headline average. A fluent final answer can conceal the point where evidence and conclusion separated. Review systems should therefore preserve raw inputs, intermediate calculations and control checks; require a human to approve consequential next steps; and use independent measurements where possible. A scientist also needs authority to reject an agent's confident recommendation without being pushed into automation bias. The benchmark does not test whether those governance controls work, but its failure cases show why a laboratory deployment needs them.[1]
What the comparison does not establish
BioBasicsBench is first-party evidence. OpenAI created it with Ligo, assessed OpenAI models and chose the examples and presentation. The source reports an external scientific review for each task, but not an independent evaluation of the final benchmark or a peer-reviewed paper. It does not state that the complete task data, model traces and rubrics are publicly available for replication. Commercial interests are direct: stronger results may support the case for using OpenAI systems in science, while poor results can inform product development and safety research.
The benchmark also measures task answers, not scientific outcomes. It does not compare agent-assisted and unassisted laboratory teams, track experimental throughput, calculate error costs, test biosafety compliance or measure whether a discovery replicates. The five attempts characterize response variability within the harness, but do not recreate months of changing samples, instruments and staff. A 66.7% mean cannot be converted into a probability that an experiment will succeed or a claim that one-third of laboratory work would fail. The rubrics assign partial credit across heterogeneous tasks.[1]
What would make the evidence decision-ready
Independent groups could rerun the benchmark across providers, publish per-task traces and compare scores with practising scientists at different experience levels. Prospective trials should randomize comparable laboratory tasks to human-only and agent-assisted workflows, then measure time, corrections, unusable samples, protocol deviations and the reproducibility of resulting conclusions. Evaluation should include unfamiliar equipment, degraded images, contradictory records and deliberately incomplete controls. Safety-sensitive work needs separate analysis rather than being averaged with low-consequence measurements.
For now, BioBasicsBench supports a narrower conclusion: newer OpenAI models earned more rubric credit on 100 selected routine wet-lab interpretation tasks, while the best reported model still made errors that could send an experiment down the wrong path. That is enough to justify careful research assistance and structured checking, not autonomous laboratory authority. Evidence would strengthen if the gains replicate independently and translate into fewer errors or faster work without new safety failures. It would weaken if performance falls on unseen laboratories or if human reviewers cannot reliably detect the model's plausible mistakes.[1]