An AI agent can act unsafely when one true fact is missing—even if nothing it sees is false
EpiSEM-Bench paired 300 permitted and prohibited tasks across six simulated domains. Giving agents an audit tool reduced unsafe commits for every tested model, but useful recovery remained uneven.
The short version
- What happened
- The benchmark contains 300 matched prohibited/permitted task pairs—600 simulated worlds—across cloud operations, commerce, communications, document access, finance and travel.
- Who it affects
- A missing permission record can turn a routine request into an unsafe automated action. People need systems that block the action independently and make the evidence trail reviewable, rather than relying on the agent to notice absence.
- What remains unproven
- The work is an unreviewed preprint using synthetic executable environments, not live organisations or real customers.
Keep in mind
The work is an unreviewed preprint using synthetic executable environments, not live organisations or real customers.
A task counted as affected if any retained critical-omission rollout produced an unsafe commit; failure frequency across rollouts was lower and must be read separately.
Results depend on the benchmark’s rules, fact annotations, tool schemas, serving configurations and single retained rollouts.
The accessible paper does not provide a clear funding or competing-interest statement.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
Missing information is a different failure from a lie
An agent may need an authorisation limit before sending money or a membership list before sharing a document. EpiSEM-Bench hides selected facts while keeping every visible statement true. Each task has a prohibited world and a closely matched permitted twin, so researchers can see both dangerous action and excessive refusal. The inventory contains 944 critical and 419 irrelevant facts across 60 task families.[1]
Access to an audit source helped, but did not solve recovery
Across the same 300 pairs, adding a recovery tool reduced the share of tasks with unsafe commits by 0.7 to 45.3 percentage points depending on the model. Nine models also completed more permitted tasks; one completed slightly fewer. For GPT-6 Luna, critical omissions produced an unsafe commit on 21% of tasks, although only 7.9% of its retained omission rollouts were unsafe, showing why one headline percentage cannot describe the whole profile.[1]
The distinction that matters
The matched-world test
The request and tools stay the same; the hidden world determines whether action is allowed.
Observe
Show complete or selectively incomplete true facts
Tests sensitivity to missing evidence
Recover
Offer a separate source for omitted facts
Tests whether the agent seeks and uses evidence
Act
Compare prohibited and permitted twins
Separates useful restraint from blanket refusal
Production systems should enforce the rule outside the agent
A useful agent should fetch missing evidence, withhold a prohibited action and still finish the permitted twin. The benchmark shows those abilities do not automatically travel together. Operators should keep payment limits, access rights and destructive-action checks in independently enforced controls, record which facts supported an action, and test omission cases. The preprint does not show that any named product failed in a live service.[1]