MulCPred: Learning Multi-Modal Concepts for Explainable Pedestrian Action Prediction

Pedestrian action prediction is crucial for many applications such as autonomous driving. However, state-of-the-art methods lack the explainability needed for trustworthy predictions. In this paper, a novel framework called MulCPred is proposed that explains its predictions based on multi-modal conc...

Full description

Saved in:
Bibliographic Details
Published in:Sensors (Basel, Switzerland) Switzerland), 2024-10, Vol.24 (20), p.6742
Main Authors: Feng, Yan, Carballo, Alexander, Fujii, Keisuke, Karlsson, Robin, Ding, Ming, Takeda, Kazuya
Format: Article
Language:English
Subjects:
autonomous driving
Behavior
computer vision
Decision making
explainable AI
Learning
multi-modal learning
neural networks
pedestrian action prediction
Pedestrians
Surveillance
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Pedestrian action prediction is crucial for many applications such as autonomous driving. However, state-of-the-art methods lack the explainability needed for trustworthy predictions. In this paper, a novel framework called MulCPred is proposed that explains its predictions based on multi-modal concepts represented by training samples. Previous concept-based methods have limitations, including the following: (1) they cannot be directly applied to multi-modal cases; (2) they lack the locality needed to attend to details in the inputs; (3) they are susceptible to mode collapse. These limitations are tackled accordingly through the following approaches: (1) a linear aggregator to integrate the activation results of the concepts into predictions, which associates concepts of different modalities and provides ante hoc explanations of the relevance between the concepts and the predictions; (2) a channel-wise recalibration module that attends to local spatiotemporal regions, which enables the concepts with locality; (3) a feature regularization loss that encourages the concepts to learn diverse patterns. MulCPred is evaluated on multiple datasets and tasks. Both qualitative and quantitative results demonstrate that MulCPred is promising in improving the explainability of pedestrian action prediction without obvious performance degradation. Moreover, by removing unrecognizable concepts, MulCPred shows improved cross-dataset prediction performance, suggesting its potential for further generalization.
ISSN:1424-8220
1424-8220
DOI:10.3390/s24206742