Can a tool output impersonate the user? Six frontier models followed the spoof
A University of Pennsylvania preprint reports that malicious tool text could pose as a user and redirect agent actions. The controlled attacks are not production breaches, and the proposed detector has so far been tested only on role-spoofing injections.
The short version
- What happened
- In a purpose-built tool-use testbed, the attack succeeded more than 95% of the time against five of six tested frontier models and 72.5% against the sixth.
- Who it affects
- An agent with authority over email, calendars, files or payments can turn a mistaken role assignment into a real action. Users need permission boundaries, confirmation for consequential steps and logs that show which source supplied each instruction.
- What remains unproven
- Unreviewed preprint
Keep in mind
Unreviewed preprint
Controlled testbeds
No reported production incident
Modified AgentDojo formatting
Task pools are synthetic
Model versions can change
Probe tested on two open models
Probe requires internal activations
Detector limited to role spoofing
No independent replication
No funding or competing-interests statement surfaced
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
The attack makes untrusted text look like an authorised instruction
The researchers studied ‘role spoofing’: malicious text inside a tool result imitates both the wording and formatting of a user message. In a five-setting testbed covering bank accounts, documents, calendars, email and climate control, an agent first read information and then issued one checked action. The injected text asked for a different action drawn from the same task pool. GPT-5.4, Claude Opus 4.8, Claude Opus 5, Claude Sonnet 5 and Gemini 3.5 Flash followed the spoof in more than 95% of trials; Gemini 3.1 Pro did so in 72.5%. Clean-task accuracy exceeded 95%.[1]
The mechanism matters more than one attack string
Experiments suggest models infer ‘who said what’ from a mixture of message meaning, writing style and apparent structure rather than relying only on protected role boundaries. In a toy training setting, adversarial training against content-based injections improved resistance to those attacks but left fake-structure attacks effective. Fine-tuning Qwen3.5-27B showed the same gap. AgentDojo-style tests also remained vulnerable, although the authors changed the default output formatting and warn that their figures are not directly comparable with the benchmark's standard harness.[1]
The distinction that matters
From tool result to unauthorised action
The attack succeeds when a model treats untrusted data as if it came from the user.
1. Tool
Returns untrusted text
Injection is inside data
2. Spoof
Mimics user content and structure
Role boundary is confused
3. Agent
Selects an action
Attacker's task may run
4. Probe
Reads internal role signal
Candidate detector
A detector looks promising but is not a complete defence
The team trained linear probes to identify the role a model internally assigns to message tokens. On held-out data, the probes reportedly exceeded 99% true-positive detection with zero false positives for Gemma 4 31B-IT and Qwen3.5-27B. The positive test pools included 288 AgentDojo and 3,621 synthetic runs for Gemma, and 253 AgentDojo and 6,490 synthetic runs for Qwen; clean negatives came from 2,156 conversations across three sources. That is substantial controlled evidence, but the detector was selected and tested within the same research programme, requires model activations and only targets role spoofing. Peer review, independent reproduction and live shadow testing are needed before providers treat it as a security control.[1]