An enhanced self-attention and A2J approach for 3D hand pose estimation

• Published in: Multimedia tools and applications 2022-12, Vol.81 (29), p.41661-41676
• Main Authors: Ng, Mei-Ying, Chng, Chin-Boon, Koh, Wai-Kin, Chui, Chee-Kong, Chua, Matthew Chin-Heng
• Format: Article
• Language: English
• Subjects: 1181: Multimedia-based Healthcare Systems using Computational Intelligence; Computer Communication Networks; Computer Science; Computer vision; Data Structures and Information Theory; Estimation; Human-computer interaction; Machine translation; Multimedia; Multimedia Information Systems; Natural language processing; Neural networks; Pose estimation; Special Purpose and Application-Based Systems; Three dimensional models; Virtual reality
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-021-11020-w

Three dimensional (3D) hand pose estimation is the task of estimating the 3D location of hand keypoints. In recent years, this task has received much research attention due to its diverse applications in human-computer interaction and virtual reality. To the best of our knowledge, there has been limited studies that model self-attention in 3D hand pose estimation despite its use in various computer vision tasks. Hence, we propose augmenting convolution with self-attention to capture long-range dependencies in a depth image. In addition, motivated by a recent work which uses anchor points set on a depth image, we extend anchor points to the depth dimension to regress 3D hand joint locations. Validation experiments using the proposed approaches are performed on various hand pose datasets, and we obtain performances that are comparable to other state-of-the-art methods. The results demonstrate the potential of these approaches in a hand-based recognition system.