Skeleton-based action recognition with multi-stream, multi-scale dilated spatial-temporal graph convolution network

• Published in: Applied intelligence (Dordrecht, Netherlands) Netherlands), 2023-07, Vol.53 (14), p.17629-17643
• Main Authors: Zhang, Haiping, Liu, Xu, Yu, Dongjin, Guan, Liming, Wang, Dongjing, Ma, Conghao, Hu, Zepeng
• Format: Article
• Language: English
• Subjects: Artificial Intelligence; Artificial neural networks; Bones; Computer Science; Computer vision; Convolution; Datasets; Human activity recognition; Kernels; Machines; Manufacturing; Mechanical Engineering; Multilayers; Neural networks; Processes; Semantics; Spatial data; Streams
• ISSN: 0924-669X; 1573-7497
• DOI: 10.1007/s10489-022-04365-8

Action recognition techniques based on skeleton data are receiving more and more attention in the field of computer vision due to their ability to adapt to dynamic environments and complex backgrounds. Topologizing human skeleton data as spatial-temporal graphs and processing them using graph convolutional networks (GCNs) has been shown to produce good recognition results. However, with existing GCN methods, a fixed-size convolution kernel is often used to extract time-domain features, which may not be very suitable for multi-level model structures. Equal proportion fusion of different streams in a multi-stream network may ignore the difference in recognition ability of different streams, and these will affect the final recognition result. In this paper, we are proposing (1) a multi-scale dilated temporal graph convolution layer (MDTGCL) and (2) a multi-branch feature fusion (MFF) structure. The MDTGCL utilizes multiple convolution kernels and dilated convolution to better adapt to the multi-layer structure of the GCN model and to obtain longer periods of contextual spatial-temporal information, resulting in richer behavioural features. MFF entails weighted fusion based on the results of multi-stream outputs, and this is used to obtain the final recognition results. As higher-order skeleton data are highly discriminative and more conducive to human action recognition, we used spatial information on joints and bones and their multiple motion, as well as angle information pertaining to bones, to model together in this study. By combining the above, we designed a multi-stream, multi-scale dilated spatial-temporal graph convolutional network (2M-STGCN) model and conducted extensive experiments with two large datasets (NTU RGB+D 60 and Kinetics Skeleton 400), which showed that our model performs at SOTA level.