Human In-Hand Motion Recognition Based on Multi-Modal Perception Information Fusion

• Published in: IEEE sensors journal 2022-04, Vol.22 (7), p.6793-6805
• Main Authors: Xue, Yaxu, Yu, Yadong, Yin, Kaiyang, Li, Pengfei, Xie, Shuangxi, Ju, Zhaojie
• Format: Article
• Language: English
• Subjects: Character recognition; Contact force; Data acquisition; Data analysis; Data integration; Empirical analysis; empirical mode decomposition; Feature extraction; Force; Gesture recognition; human in-hand motion; Human motion; Liapunov exponents; Machine learning; maximum Lyapunov exponent; Modal data; Motion perception; Multi-modal information; random forest; Recognition; Segmentation; Sensor phenomena and characterization; Sensors; Wavelet transforms
• ISSN: 1530-437X; 1558-1748
• DOI: 10.1109/JSEN.2022.3148992

A human in-hand motion (HIM) recognition system based on multi-modal perception information fusion is proposed in this paper, which can observe the state information between the object and the hand by using customized ten kinds of HIM manipulation in order to recognize the complex HIMs. First, combined with the characteristics of HIM capture, ten kinds of HIM sets are designed, and finger trajectory, contact force and electromyographic signal data are acquired synchronously through the multi-modal data acquisition platform; second, motion segmentation is realized through the threshold segmentation method, the multi-modal signal preprocessing is realized by Empirical Mode Decomposition (EMD), and multi-modal signal feature extraction is realized by Maximum Lyapunov Exponent (MLE); then, a detailed non-linear data analysis is carried out. A detailed analysis and discussion are presented from the results of the Random Forest (RF) recognizing HIMs, the comparison results of motion recognition rates of different subjects, the comparison results of motion recognition rates of different perceptrons, and the comparison results of the motion recognition rates of different machine learning methods. The experimental results show that the multi-modal perception information based HIM recognition system proposed in this paper can effectively recognize ten different HIMs, with an accuracy rate of 93.72%.