Graph2Net: Perceptually-Enriched Graph Learning for Skeleton-Based Action Recognition

• Published in: IEEE transactions on circuits and systems for video technology 2022-04, Vol.32 (4), p.2120-2132
• Main Authors: Wu, Cong, Wu, Xiao-Jun, Kittler, Josef
• Format: Article
• Language: English
• Subjects: Convolution; Feature extraction; graph learning; Hidden Markov models; Human activity recognition; Human motion; Innovations; Learning; Representations; Skeleton; Skeleton-based action recognition; Spatial data; Spatiotemporal phenomena; Task analysis; Technological innovation
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2021.3085959

Skeleton representation has attracted a great deal of attention recently as an extremely robust feature for human action recognition. However, its non-Euclidean structural characteristics raise new challenges for conventional solutions. Recent studies have shown that there is a native superiority in modeling spatiotemporal skeleton information with a Graph Convolutional Network (GCN). Nevertheless, the skeleton graph modeling normally focuses on the physical adjacency of the elements of the human skeleton sequence, which contrasts with the requirement to provide a perceptually meaningful representation. To address this problem, in this paper, we propose a perceptually-enriched graph learning method by introducing innovative features to spatial and temporal skeleton graph modeling. For the spatial information modeling, we incorporate a Local-Global Graph Convolutional Network (LG-GCN) that builds a multifaceted spatial perceptual representation. This helps to overcome the limitations caused by over-reliance on the spatial adjacency relationships in the skeleton. For temporal modeling, we present a Region-Aware Graph Convolutional Network (RA-GCN), which directly embeds the regional relationships conveyed by a skeleton sequence into a temporal graph model. This innovation mitigates the deficiency of the original skeleton graph models. In addition, we strengthened the ability of the proposed channel modeling methods to extract multi-scale representations. These innovations result in a lightweight graph convolutional model, referred to as Graph2Net, that simultaneously extends the spatial and temporal perceptual fields, and thus enhances the capacity of the graph model to represent skeleton sequences. We conduct extensive experiments on NTU-RGB+D 60&120, Northwestern-UCLA, and Kinetics-400 datasets to show that our results surpass the performance of several mainstream methods while limiting the model complexity and computational overhead.