A novel autonomous learning framework to enhance sEMG-based hand gesture recognition using depth information

• Published in: Biomedical signal processing and control 2021-04, Vol.66, p.102444, Article 102444
• Main Authors: Ovur, Salih Ertug, Zhou, Xuanyi, Qi, Wen, Zhang, Longbin, Hu, Yingbai, Su, Hang, Ferrigno, Giancarlo, De Momi, Elena
• Format: Article
• Language: English
• Subjects: Augmented reality; Augmented reality applications; Autonomous learning; Classification; classifier; Clustering; Computer interaction; Depth vision; electromyography; gesture; Gesture recognition; Hand-gesture recognition; Hands gesture recognition; human; Human computer interaction; Learning systems; Machine learning; Motion analysis techniques; Multilayer neural networks; Network layers; Palmprint recognition; Real time recognition; Stable performance; Surface electromyography; vision
• ISSN: 1746-8094; 1746-8108; 1746-8108
• DOI: 10.1016/j.bspc.2021.102444

•A novel autonomous learning framework is presented to integrate the benefits of both depth vision and EMG signals.•Combination of depth information and sEMG with HSOM and MNN adopted to achieve better accuracy for the designed VR application.•A hand gesture recognition demonstration is implemented to verify the effectiveness of the proposed framework. Hand gesture recognition using surface electromyography (sEMG) has been one of the most efficient motion analysis techniques in human–computer interaction in the last few decades. In particular, multichannel sEMG techniques have achieved stable performance in hand gesture recognition. However, the general solution of collecting and labeling large data manually leads to time-consuming implementation. A novel learning method is therefore needed to facilitate efficient data collection and preprocessing. In this paper, a novel autonomous learning framework is proposed to integrate the benefits of both depth vision and EMG signals, which automatically label the class of collected EMG data using depth information. It then utilizes a multiple layer neural network (MNN) classifier to achieve real-time recognition of the hand gestures using only the sEMG. The overall framework is demonstrated in an augmented reality application by the recognition of 10 hand gestures using the Myo armband and an HTC VIVE PRO. The results show prominent performance by introducing depth information for real-time data labeling.