Can AI agents safely sign off real engineering work? This benchmark says not yet
Google and Mercor tested agents on 85 civil, structural, electrical and mechanical assignments. The strongest native setup averaged 48.3%, while only 41.3% of its deliverables cleared every hard safety and feasibility check.
The short version
- What happened
- The benchmark contains 85 long-horizon tasks: 30 mechanical, 27 electrical, 18 civil and 10 structural, with an average of 61 project files per task.
- Who it affects
- Engineers may gain drafting and checking assistance, but they need enough time, authority and traceability to challenge the tool rather than rubber-stamp it.
- What remains unproven
- The paper is an unreviewed preprint and the benchmark, release and leaderboard are operated by its creators.
Keep in mind
The paper is an unreviewed preprint and the benchmark, release and leaderboard are operated by its creators.
The 85 tasks are curated office deliverables, not a representative sample of all engineering practice or built projects.
Open-ended outputs use an LLM rubric judge; automated checks cannot encode every relevant professional judgment.
Only 29 tasks are public, while the 56-task hidden set cannot be independently inspected in full.
No matched human baseline, field safety outcome, productivity denominator or employment effect is established.
Multi-source analysis · 4 linked sources. Primary analysed source date: 3 November 2026. Earliest linked supporting source: 21 October 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The direct answer: these agents are not ready to approve physical designs on their own
No. Physical-Engineering's Last Exam gives current AI agents something closer to an engineering project folder than a textbook question, and the strongest native model-and-tool setup still averaged 48.3% on the hidden test set. More importantly, a design earned cost-efficiency credit only after clearing every hard safety and feasibility check. The top configuration met 91% of individual constraints on the co-authored tasks, yet only 41.3% of its deliverables passed all hard checks. That difference is the practical finding. A system can look broadly competent while missing one requirement that prevents a drawing, calculation or technical memo from being acceptable. The benchmark does not prove that a 48% score corresponds to a 52% chance of unsafe construction, and none of the outputs became signed real-world designs. It does show why a high average, fluent explanation or nearly complete checklist cannot substitute for licensed review. For engineers, the plausible near-term role is bounded assistance—searching files, drafting calculations or checking revisions—with responsibility, verification and sign-off remaining with people who can inspect the complete project context.[1][2][3]
What the 85 tasks ask an agent to do
The benchmark covers 30 mechanical, 27 electrical, 18 civil and 10 structural tasks. Examples include choosing a foundation strategy from geotechnical records, checking roof diaphragms for wind or seismic loads, sizing electrical feeders, coordinating cable tray and chilled-water routes, reviewing a non-converging fire simulation and preparing an engineer's estimate. Each task runs in an isolated container and starts with a project folder containing an average of 61 files; the complete benchmark holds 5,170 files. Those files include PDFs, spreadsheets, emails, specifications, two-dimensional CAD drawings, three-dimensional building models and document revisions. The agent must decide which information governs, use local tools, run calculations and create a specified deliverable. It has no open internet access and may use up to six hours. This design improves on short-answer benchmarks because much professional work is information selection and document control, not just recall. But it still represents selected office deliverables inside a controlled environment. It excludes site visits, conversations with clients and contractors, changing conditions, formal professional liability and the full chain by which engineering decisions are checked, approved, built and maintained.[1][2][3][4]
The distinction that matters
Why an average score is not enough for engineering approval
PLE separates broad task performance from the requirement to clear every safety and feasibility gate.
Project folder
Agent searches an average of 61 files
Relevant inputs and revisions must be selected
Deliverable
Agent calculates, checks or drafts a design record
Output is scored by code, an LLM rubric judge or both
Hard gates
Every safety and feasibility check must pass
Only then can cost-efficiency points count
Human sign-off
Licensed reviewer verifies context and responsibility
Benchmark performance does not replace approval
How the tasks and graders were constructed
The creators used two routes. Practising engineers wrote 44 expert-authored tasks, including the starting files, prompt and grading rubric. For 41 co-authored tasks, engineers set the initial problem while an agent pipeline assembled the environment, reference deliverables and executable grader from licensable standards; engineers then validated the task and increased its difficulty. Twenty-nine tasks are public development material and 56 form the hidden test set. Quantitative outputs can be checked by software, while open-ended work is scored against an expert rubric by an LLM judge. Some tasks use both. Every tested model-harness pair is run three times and results are averaged, reducing but not eliminating run-to-run variation. This is more transparent than a single demonstration because the public repository exposes development tasks and evaluation code. Still, the denominator is 85 curated assignments, not the universe of physical-engineering work. Co-authored tasks may preserve assumptions introduced by the same class of systems being evaluated, and an LLM judge may reward or miss features differently from a licensed human reviewer. The paper reports sensitivity work with alternative judges, but independent reproduction on the hidden set is not yet available.[1][2][3]
How to read the 48.3% result—and the paper's different 50.6% figure
The public release ranks each model on its native harness, the software layer that gives a model tools and manages its work. On that comparison, Claude Opus 5.5 with Claude Code scored 48.3% across the 56 hidden tasks, followed by GPT 6 Astra with Codex at 45.8% and Gemini 4 Argon with Antigravity at 40.6%. The preprint reports a 50.6% strongest configuration across its broader matrix of model-harness combinations. These are not interchangeable league-table numbers: one is the native-harness headline used for the public competition, while the other reflects the paper's best evaluated pairing. The creators tested 14 frontier models across seven public harnesses. They say model choice produced spreads of up to 50 percentage points, while switching harnesses for the same model changed scores by a median of about three points, although harness choice affected cost. The strongest reported setup used about 27 million tokens and 63 minutes per task on average. That is evidence about benchmark resource use, not a comparison with an engineer's time, fees or quality. No human baseline with the same folders, time limit and scoring procedure is reported on the public page.[1][2][3]
What this means for engineers, firms and the public
For working engineers, the benchmark points toward task decomposition rather than replacement. A firm could test an agent on document search, drawing-change logs, first-pass calculations or specification checks, then require a named professional to verify inputs, reproduce the calculation and approve the output. Tasks with executable checks may be easier to bound than ambiguous judgments about constructability, unusual site conditions or acceptable risk. Employers should measure rework, missed constraints and review time, not merely how quickly an agent produces a polished file. Workers also need clarity about who is responsible when automation changes a calculation or selects the wrong revision. For clients and the public, a professional seal must continue to mean accountable human judgment under the applicable law. A benchmark score does not authorize a product for structural, electrical, civil or mechanical practice. Procurement teams should ask what task was evaluated, which standards and file formats were used, what failure rate occurred at the hard gates, how the tool logs its actions and how a reviewer can detect incomplete work. The evidence supports carefully supervised trials; it does not support unattended sign-off or staffing promises.[1][2][3][4]
Evidence limits, commercial interests and what would change the assessment
PLE is an unreviewed preprint and a benchmark built by Google researchers with Mercor, a company that supplies expert labour and operates the task environment. The public release, benchmark site and leaderboard are controlled by the creators. The 29 development tasks and code can be inspected, but the 56 hidden tasks cannot be audited in full without undermining the test. The leaderboard can change as models and harnesses change. Automated checks are valuable where requirements can be encoded; rubric-scored work inherits the limits of the selected rubric and judge. The tests do not measure safety after construction, rare environmental conditions, coordination with other disciplines, regulatory acceptance, liability, cyber security of project data, worker displacement or whether assistance improves a human team's outcomes. Confidence would rise with independent teams reproducing the public tasks, licensed engineers blind-reviewing model and human outputs, a matched human baseline, disclosed error categories and field trials that measure review time and caught versus missed defects. Evidence that an agent repeatedly clears all hard checks on unfamiliar projects and improves a team without weakening accountability would change the assessment. Until then, the result is consequential research, not Breaking news and not permission to automate professional sign-off.[1][2][3][4]