An autonomous-driving model can explain a turn it does not actually make
Counterfactual tests of four driving models found that fluent stated reasons were often weakly coupled to predicted trajectories. This is an offline explanation test, not evidence about on-road crashes or driving quality.
The short version
- What happened
- Researchers edited the stated reasoning or removed action-relevant objects from camera scenes, then reran four released driving vision-language-action models on 799 DriveLM scenes and 349 PhysicalAI-AV events.
- Who it affects
- A driver or safety operator may read a plausible explanation as evidence that the system acted for that reason. The study shows that assumption needs a causal test before explanations are used for trust or accountability.
- What remains unproven
- The paper is an unreviewed preprint.
Keep in mind
The paper is an unreviewed preprint.
Only four released models and two offline validation datasets were evaluated; object-removal tests covered three models.
Trajectory displacement measures change from the model's own baseline, not correctness, comfort, collision avoidance or on-road safety.
Object-removal images and consistency judgments involved manual review; the accessible paper does not report a separate funding or conflict declaration.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
The researchers changed the reason, not the road
Each content edit—mistake, negation or a reason borrowed from another scene—was compared with a meaning-preserving paraphrase passed through the same interface. Results used three seeds and paired every edited trajectory with the same model and scene. AutoVLA and ORION reacted more to whether text occupied the reasoning slot than to the driving meaning of that text.[1]
A stated left turn often did not reach the left
AutoVLA's stated left/right decision matched its predicted endpoint in 187 of 202 DriveLM scenes and 66 of 74 PhysicalAI-AV scenes. Alpamayo-1.5 matched in 71 of 155 scenes; Alpamayo-R1 in 59 of 121. Removing a named object made all three tested models update their words, but manual inspection still found cases where the new trajectory did not implement the revised explanation.[1]
The distinction that matters
Two checks for a faithful driving explanation
A reason should influence the action and update with the scene.
Reason edit
Change the claimed maneuver
Does the predicted trajectory follow?
Scene edit
Remove a named object
Do both reason and trajectory update?
Deployment
Test closed-loop driving
Still needed to establish safety
A readable rationale is not an audit trail
For passengers, engineers and regulators, the result is a warning about interface trust: a polished explanation may describe the scene without revealing what drove the action. Deployment evidence would need closed-loop driving tests, safety outcomes and independently reproducible interventions—not explanation fluency alone.[1]