Dynamic Hand Gesture Recognition Based on A-Mode Ultrasound Sensing: Proposing an Algorithm Based on the Long Short-Term Memory Framework

• Published in: IEEE systems, man, and cybernetics magazine man, and cybernetics magazine, 2023-10, Vol.9 (4), p.45-52
• Main Authors: Liu, Donghan, Zhang, Dinghuang, Zhang, Gongyue, Liu, Honghai
• Format: Magazinearticle
• Language: English
• Subjects: Deep learning; Gesture recognition; Heuristic algorithms; Human computer interaction; Multimodal sensors; Support vector machines; Ultrasonic imaging
• ISSN: 2380-1298; 2333-942X
• DOI: 10.1109/MSMC.2023.3299431

Hand gesture recognition plays a crucial role in the field of human-computer interaction (HCI). In terms of the multimodal sensing of hand gestures, the A-mode ultrasound (AUS) signal is far less investigated, especially for dynamic hand gestures, than its counterparts, such as surface electromyography (sEMG). In this article, we explore the recognition of dynamic hand gestures by proposing an AUS-based deep learning algorithm that codes time correlation in the long short-term memory (LSTM) framework. First, a dynamic handwritten numbers 0 through 9 dataset was created and recorded. Then, after preprocessing the data, we propose an algorithm based on the deep learning framework. Also, we designed two different strategies that used two different structures for comparison. Finally, through experiments, the accuracy of different deep learning structures [convolutional neural network (CNN) and LSTM] and traditional feature extraction [support vector machine (SVM)] on dynamic gesture recognition of ultrasonic (US) signals are compared, and we prove that LSTM has better performance. The experiment results prove that the proposed method achieves 89.5% accuracy, which outperforms its counterparts. It paves the way for potential HCI applications involving dynamic hand gestures. It is anticipated that more uses of dynamic gesture recognition will be discussed in the future to bring the research into real-life applications.