Sequential inter-hop graph convolution neural network (SIhGCN) for skeleton-based human action recognition

• Published in: Expert systems with applications 2022-06, Vol.195, p.116566, Article 116566
• Main Authors: Setiawan, Feri, Yahya, Bernardo Nugroho, Chun, Seok-Ju, Lee, Seok-Lyong
• Format: Article
• Language: English
• Subjects: Action recognition; Artificial neural networks; Attention mechanism; Color imagery; Feature aggregation; Graph convolutional neural network; Human activity recognition; Human motion; Invariants; Neural networks; Normalized Laplacian matrix
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2022.116566

•A graph convolution model for skeleton-based action recognition is proposed.•Normalized Laplacian Matrix is utilized to encode the graph information.•An attention-based feature aggregation is proposed to extract the salient features.•The proposed method achieves better results than the baseline models. Skeleton-based human action recognition has attracted a lot of attention due to its capability and potential to provide more information than just using the sequence of RGB images. The use of Graph Convolutional Neural Network (GCN) becomes more popular since it can model the human skeleton very well. However, the existing GCN architectures ignore the different levels of importance on each hop during the feature aggregation and use the final hop information for further calculation, resulting in considerable information loss. Besides, they use the standard Laplacian or adjacency matrix to encode the property of a graph into a set of vectors which has a limitation in terms of graph invariants. In this work, we propose a Sequential Inter-hop Graph Convolution Neural Network (SIhGCN) which can capture salient graph information from every single hop rather than the final hop only and our work utilizes the normalized Laplacian matrix which provides better representation since it relates well to graph invariants. The proposed method is validated on two large datasets, NTU-RBG + D and Kinetics, to demonstrate the superiority of our proposed method.