Can an AI spot a hazard but still build an unsafe household plan?
CAPSafe found that several language models recognised unsafe single actions above 90%, yet the best safe-and-successful result on 243 longer household tasks was 78.19%. The benchmark is a symbolic simulator, not a robot trial, and was created by Toshiba researchers.
The short version
- What happened
- CAPSafe contains 1,000 balanced single-action examples and 243 longer household task instances drawn from symbolic environments, with about 12 hazards, 96 objects and 21.8 required steps per scene on average.
- Who it affects
- Anyone considering an embodied assistant should distinguish a model's ability to describe a hazard from a system's ability to avoid harm throughout a task. Certification needs perception, hardware, recovery and human-supervision tests in addition to text planning.
- What remains unproven
- The accessible paper is a preprint with a REALM 2026 reference; independent peer-review status was not confirmed.
Keep in mind
The accessible paper is a preprint with a REALM 2026 reference; independent peer-review status was not confirmed.
The authors work at Toshiba Europe, and no separate funding or competing-interest statement was located in the accessible manuscript.
The simulator is symbolic and fully observable, omitting sensing, navigation and low-level control failures.
Hazards are rule-defined; severity labels began with two LLM ratings and were checked by three internal judges on 30 pairs.
Household layouts primarily reflect European and North American practices.
Model versions, prompts and vendor APIs can change after the benchmark date.
Source analysis · 1 linked source. Primary analysed source date: 20 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
The benchmark makes hazards depend on context
A stove-wiping action can be safe or dangerous depending on whether the burner is hot or switched on. CAPSafe generates safe and unsafe variants from 181 simulated household environments, producing 732,370 possible action-environment pairs before diversity sampling 500 of each. Its longer test uses 18 task types and 243 hazardous scenes, checking hazards during execution and after completion—such as leaving an appliance on or a storage unit open.[1]
Recognising one unsafe action did not guarantee a safe plan
Qwen3-32B achieved 92.6% accuracy on the balanced single-action set, while Gemini-3-pro recalled 99.2% of unsafe actions but was more likely to reject safe ones. In plan generation, Gemini-3-pro reached 93.83% task success, yet its strongest reflection setting completed tasks both safely and successfully only 78.19% of the time. Qwen3-32B's planning safety rate was 51.65%. Across reflection runs, 57% of encountered hazards remained unmitigated.[1]
The distinction that matters
Why long plans are harder than single checks
Safety has to hold across every step and the final state.
Recognise
Judge one action in a known state
Best models exceed 90% on the sampled set
Plan
Complete about 22 steps amid many hazards
Errors accumulate across the sequence
Finish
Leave appliances and storage safe
Post-task hazards still reduce safe success
Demonstrations helped, but the test omits the physical world
Few-shot examples of safe behaviour consistently improved safe task completion. General safety guidelines mainly helped the stronger proprietary models, and reflection could improve safety while reducing task success. These patterns are useful for designing evaluation, not certifying products. CAPSafe is a fully observable text simulator with rule-defined hazards and no perception or motor-control errors. Its household layouts mainly reflect European and North American settings, while severity scores began with two model ratings and a 30-pair internal human check.[1]