Action Recognition of Traffic Police by Attentive for Self-Driving Vehicles

• Main Authors: Ha, Manh-Hung, Le, Minh-Huy, Dang, Khoa Nguyen, Kim, Dinh-Thai, Tran, Van Luan
• Format: Conference Proceeding
• Language: English
• Subjects: action recognition; convolutional neural network; Deep neural network; human pose estimation; Image segmentation; Law enforcement; Motion segmentation; Optical imaging; recurrent neural network; Roads; Skeleton; spatiotemporal attention; subject identification; Transfer learning
• ISSN: 2162-1039
• DOI: 10.1109/ATC55345.2022.9942963

This work develops a Deep Neural Network (DNN) with the spatiotemporal skeleton-based attentions to effectively perceive traffic officers for self-driving vehicles. The DNN framework includes two Convolutional Neural Networks (CNNs), Attention-Jointed Appearance (AJA) and Attention-Based Motion (ABM) layers, Recurrent Neural Networks (A_RNN), and Feed-Forward Networks (FFNs) where RGB and optical-flow streams are inputs accompanied with pose joint maps of two-dimensional subject skeletons. The AJA, and ABM layers pay attention to poses, and motions of subjects, respectively. The A_RNNs generate the attention weights over time steps to highlight rich temporal context. In FFN s, one takes the outputs of A_RNNs to determine the action type, and the other processes the outputs of the AJA layer together with the majority voting to enhance subject identification. Based on transfer learning, the initial parameters of two CNN s are from the converged network of Google Inception V3 trained by ImageN et and Kinetics. The experimental results reveal that the proposed DNN achieves the average accuracies around 100.0%, and 97.6% for subject, and action recognitions, respectively, at the traffic police dataset. Comparing to the conventional work, our DNN with superior performance can be a great context-aware system for self-driving vehicles.