Can interpretable AI tell when a pedestrian will cross?
New analysis today of a peer-reviewed 3 October study on 1,800 labelled pedestrian tracks. A jointly trained concept-bottleneck model produced a small F1 improvement, but one supposedly meaningful concept failed and human intervention behaved unexpectedly; the study does not show safer vehicles or fewer collisions.
The short version
- What happened
- The experiment used 1,800 labelled pedestrian tracks from the public PIE urban-driving dataset and compared image baselines with concept-bottleneck variants.
- Who it affects
- Pedestrians could benefit if interpretable models expose failures before deployment, but an appealing explanation can also create false confidence.
- What remains unproven
- The analysis uses one public video dataset with 1,800 labelled tracks and no independent external dataset or live road deployment.
Keep in mind
The analysis uses one public video dataset with 1,800 labelled tracks and no independent external dataset or live road deployment.
The absolute F1 difference between the best concept model and ResNet baseline was small and cannot be converted into collision reduction.
One imbalanced behavioural concept collapsed to F1 of zero, while concept interventions behaved unexpectedly in joint training.
Soft concepts may carry information beyond their human-readable labels, weakening the claim that the bottleneck is fully interpretable.
Integrated-gradients maps depend on a chosen black baseline and show model sensitivity rather than causal reasoning.
Source analysis · 2 linked sources. Primary analysed source date: 3 October 2026.
Read the full report · 6 min The evidence, methods, implications and limitations
The model predicts intention from short image sequences
Pedestrian-intention systems try to answer a difficult question before a person enters a vehicle's path: are they about to cross? The study evaluates that question on 1,800 annotated pedestrian tracks from the public PIE dataset, built from urban driving video. The outcome is a binary crossing-intention label. The models receive short sequences of cropped images and are tuned across one, three, five or 15 frames rather than controlling a vehicle in real time.
This is narrower than pedestrian detection and far narrower than autonomous-driving safety. A vehicle still needs reliable tracking, distance and speed estimation, path planning, braking and safe failure behaviour. The paper tests a recognition component offline. It reports no road trial, collision, near miss, braking decision or comparison with a human driver, so its results cannot establish that a vehicle became safer.[1][2]
Concept bottlenecks make intermediate claims visible
A conventional image model can output a crossing probability without exposing what it believes about the person. The concept-bottleneck design adds intermediate predictions for behaviours that people might recognise: whether the pedestrian is walking or standing, whether they appear to look toward traffic, and whether a traffic signal is red. The final crossing decision is then derived from those predicted concepts, either through staged training or joint optimisation.
That structure promises two kinds of accountability. Engineers can measure whether the intermediate concepts are correct, and in principle a human can intervene on a concept to see whether the final prediction changes appropriately. But the bottleneck is only as meaningful as its labels and constraints. A soft numeric concept channel can carry information unrelated to the named concept, while an incomplete concept set can omit posture, curb geometry, vehicle motion and social context that influence crossing.[1]
The headline performance difference is small
The best jointly trained concept-bottleneck model used three frames sampled at six frames per second and achieved mean F1 of 0.909 across repeated runs. The ResNet image baseline reached 0.902. A Kolmogorov-Smirnov comparison reported p = 0.0079, suggesting the run-level distributions differed, but the absolute gap is seven thousandths. Statistical detectability on repeated benchmark runs should not be translated into a meaningful reduction in road risk.
The study also explores a more efficient MBConv image backbone and hard versus soft concept representations. Soft concepts preserve continuous signals and generally perform better, but that advantage can undermine interpretability if the values encode residual image information beyond the stated label. Hard concepts are easier to read but discard nuance and reduce performance. The trade-off is central: a labelled layer does not automatically make the whole model causally understandable.[1]
A failed concept exposes the value of looking inside
The behavioural labels were imbalanced: roughly 43% of examples were positive for action, while only 18% were positive for looking. In one jointly trained model, the looking concept collapsed to F1 of zero. The final intention score could remain strong because other signals compensated or because the soft bottleneck carried information not captured by the concept label. That is precisely the kind of failure an intermediate audit can reveal and an end-to-end score can hide.
Concept intervention produced another warning. Replacing predicted concepts with ground-truth values did not reliably improve the final prediction and in some joint-training experiments made it worse. A genuinely interpretable pipeline should usually respond coherently when an erroneous intermediate belief is corrected. The unexpected result suggests the final layer had adapted to the model's imperfect concept distribution, limiting a human operator's ability to treat those concepts as stable controls.[1]
Integrated gradients show attention, not reasoning
The paper also applies integrated gradients to visualise which pixels influenced predictions. The concept model placed more attribution inside pedestrian bounding boxes than the baseline, supporting the limited claim that its decisions were more spatially focused on the person. Yet attribution maps do not prove that the model used the right causal cue. A bright region can reflect texture, contrast or correlations that fail in another city, camera or weather condition.
Integrated gradients also require a reference image; this study used a black baseline. Different baselines and preprocessing can change attributions. Bounding-box concentration may be desirable for concepts about gaze and movement, but context such as traffic lights, curb position and approaching vehicles can legitimately matter. The method helps investigators ask where sensitivity lies. It does not explain why a pedestrian will act or guarantee a faithful human-readable rule.[1]
What would change the assessment
Stronger evidence would freeze the models and test them on independent datasets from different countries, cameras, lighting, weather and pedestrian groups. Evaluation should report time-to-crossing, calibration, missed vulnerable road users and false alerts, not only aggregate F1. Concept labels need inter-rater reliability, subgroup analysis and tests showing that human corrections cause predictable changes without hidden information leaking through soft channels.
A prospective shadow-mode road study would then connect intention estimates to actual downstream planning while keeping the model out of control. Only carefully governed trials could establish whether explanations help safety engineers detect faults faster or whether the system reduces braking errors. For now, the paper demonstrates a promising audit architecture and a useful failure case—not a deployable pedestrian-safety claim.[1][2]