Biology agents could look up protein facts—but geometry exposed a hidden weakness
A 131-task preprint found that three frontier agents were much stronger at metadata lookup than coordinate calculations. Overall scores concealed the split, and rankings changed with evaluation choices.
The short version
- What happened
- ProteinGeoBench contains 131 expert-written tasks: 36 metadata, 51 sequence, 28 numeric and 16 concept-reasoning questions; the four-model comparison used 4,249 rollouts.
- Who it affects
- Researchers should verify coordinate calculations and inspect execution traces even when an agent gives fluent, correct-looking protein metadata.
- What remains unproven
- The preprint is company-affiliated and has not been peer reviewed.
Keep in mind
The preprint is company-affiliated and has not been peer reviewed.
The 131 tasks lean towards biophysics, and different experiments use different, sometimes purposively selected task subsets.
Thirty unpublished-structure tasks derive from two substrates, so they do not represent confidential or novel structures generally.
Source analysis · 1 linked source. Primary analysed source date: 19 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
One score hid two very different abilities
An unreviewed benchmark study found that frontier agents could retrieve protein-structure metadata more reliably than they could calculate distances or angles from coordinates. ProteinGeoBench supplies structure files, a natural-language question and an executable verifier for 131 tasks: 36 metadata lookups, 51 sequence checks, 28 numeric calculations and 16 concept-reasoning tasks. The main comparison ran 4,249 rollouts across Claude Opus 5, GPT-5.6-sol, Grok 4.5 and Gemini 3.1 Pro Preview.
Overall means ranged from 0.707 to 0.816. The leading three were within 0.031, and bootstrap intervals did not resolve their order. Yet the family split was material: for example, Claude Opus 5 averaged 0.907 on lookup and 0.716 on numeric tasks. Twelve tasks scored below 0.5 for every model; numeric questions were 28 of 131 tasks but half of those shared failures.[1]
Evaluation settings changed the apparent result
With only one run per task, 23.9% of bootstrap replicates put a different model first. On a selected 15-task slice, changing the agent harness moved scores from -0.071 to +0.086, while removing network access hurt more than blocking eight structural-database domains. Those access contrasts were graded in different passes, so the residual cannot be cleanly attributed. Across a wider 26-model sweep, numeric and non-numeric performance rose together rather than revealing a fixed geometry ceiling.
For structural biologists, the useful lesson is operational: demand per-family results, repeat counts, tool permissions and network conditions before trusting an aggregate benchmark. Thirty tasks used structures absent from public databases during evaluation, but they came from only two substrates. The suite leans towards biophysics and several comparisons use small, purposively chosen subsets. The released tasks and verifiers make replication possible; independent labs should add broader unpublished structures and prospective research workflows before these agents are trusted with consequential analysis.[1]
The distinction that matters
Why the aggregate score can mislead
The benchmark combines tasks that depend on different abilities and resources.
Lookup
Retrieve recorded database metadata
High scores for three agents
Geometry
Calculate from atomic coordinates
Lower and shared failures
Evaluation setup
Change runs, tools or network access
Shift scores and rankings