Adaptive shift graph convolutional neural network for hand gesture recognition based on 3D skeletal similarity

• Published in: Signal, image and video processing image and video processing, 2024, Vol.18 (11), p.7583-7595
• Main Author: Bulugu, Isack
• Format: Article
• Language: English
• Subjects: Accuracy; Algorithms; Artificial neural networks; Computer Imaging; Computer Science; Constraints; Datasets; Gesture recognition; Graph neural networks; Human performance; Image Processing and Computer Vision; Multimedia Information Systems; Neural networks; Nodes; Original Paper; Pattern Recognition and Graphics; Signal,Image and Speech Processing; Topology; Vision
• ISSN: 1863-1703; 1863-1711
• DOI: 10.1007/s11760-024-03412-w

Graph convolutional neural networks (GCNs) have shown promising results in the field of hand gesture recognition based on 3D skeletal data. However, most existing GCN methods rely on manually crafted graph structures based on the physical structure of the human hand. During training, each graph node can only establish connections based on these manual settings, and is unable to perceive new relationships between skeleton nodes that arise during gesture execution. This limitation leads to inflexible and often suboptimal graph topologies. Shift graph convolutional networks improve flexibility in the receptive field by altering the graph network structure, particularly by achieving good results in global shift angles. To address the shortcomings of previous GCN methods, an adaptive shift graph convolutional neural network (AS-GCN) is proposed for hand gesture recognition. AS-GCN draws inspiration from shift graph convolutional networks and employs the characteristics of each human action to guide the graph neural network in performing shift operations, aiming to accurately select nodes that require an expanded receptive field. Experiments are conducted on the SHREC’17 dataset for general skeleton-based gesture recognition, both with and without physical constraints on skeletal relationships. Compared to existing state-of-the-art (SOTA) algorithms, the AS-GCN algorithm demonstrates average improvements of 5.13% and 8.33% in gesture recognition accuracy for 14 gestures and 28 gestures setting, respectively, under physical constraints. Without physical constraints, the AS-GCN achieves average improvements of 4% and 7.97% for the 14 gestures and 28 gesture settings, respectively. The AS-GCN-C algorithm boosts accuracy for gestures 14 and 28 in the DHG-14/28 dataset, outperforming several state-of-the-art (SOTA) methods by 1.6–13.2% and 6.8–18.4% respectively. Similarly, the AS-GCN-A algorithm improves accuracy for both gesture settings, surpassing SOTA by margins ranging from 4.6 to 10.2% for gestures 14, and from 8.1 to 15.2% for gestures 28.