Interactive Learning of a Dual Convolution Neural Network for Multi-Modal Action Recognition

• Published in: Mathematics (Basel) 2022-11, Vol.10 (21), p.3923
• Main Authors: Li, Qingxia, Gao, Dali, Zhang, Qieshi, Wei, Wenhong, Ren, Ziliang
• Format: Article
• Language: English
• Subjects: action recognition; Activity recognition; Artificial neural networks; Convolutional codes; convolutional neural network; Datasets; Design; Feature extraction; feature interactive learning; Food science; Image segmentation; Interactive learning; Learning; Machine learning; Mathematical research; Methods; Modal data; Modules; Multiplexers; Multiplexing; Neural networks; Optimization; rank pooling; Video compression
• ISSN: 2227-7390; 2227-7390
• DOI: 10.3390/math10213923

RGB and depth modalities contain more abundant and interactive information, and convolutional neural networks (ConvNets) based on multi-modal data have achieved successful progress in action recognition. Due to the limitation of a single stream, it is difficult to improve recognition performance by learning multi-modal interactive features. Inspired by the multi-stream learning mechanism and spatial-temporal information representation methods, we construct dynamic images by using the rank pooling method and design an interactive learning dual-ConvNet (ILD-ConvNet) with a multiplexer module to improve action recognition performance. Built on the rank pooling method, the constructed visual dynamic images can capture the spatial-temporal information from entire RGB videos. We extend this method to depth sequences to obtain more abundant multi-modal spatial-temporal information as the inputs of the ConvNets. In addition, we design a dual ILD-ConvNet with multiplexer modules to jointly learn the interactive features of two-stream from RGB and depth modalities. The proposed recognition framework has been tested on two benchmark multi-modal datasets—NTU RGB + D 120 and PKU-MMD. The proposed ILD-ConvNet with a temporal segmentation mechanism achieves an accuracy of 86.9% and 89.4% for Cross-Subject (C-Sub) and Cross-Setup (C-Set) on NTU RGB + D 120, 92.0% and 93.1% for Cross-Subject (C-Sub) and Cross-View (C-View) on PKU-MMD, which are comparable with the state of the art. The experimental results shown that our proposed ILD-ConvNet with a multiplexer module can extract interactive features from different modalities to enhance action recognition performance.