A Novel PPG-FMG-ACC Wristband for Hand Gesture Recognition

Wrist-based hand gesture recognition has the potential to unlock naturalistic human-computer interaction for a vast array of virtual and augmented reality applications. Photoplethysmography (PPG), force myography (FMG), and accelerometry (ACC) have generally been proposed as isolated single sensing...

Full description

Saved in:
Bibliographic Details
Published in:IEEE journal of biomedical and health informatics 2022-10, Vol.26 (10), p.5097-5108
Main Authors: Wang, Hong, Kang, Peiqi, Gao, Qinghua, Jiang, Shuo, Shull, Peter B.
Format: Article
Language:English
Subjects:
Accelerometers
accelerometry (ACC)
Artificial neural networks
attention mechanism
Augmented reality
Convolutional neural networks
Feature extraction
force myography (FMG)
Gesture recognition
Hand
Hand (anatomy)
Muscles
Neural networks
Photoplethysmography (PPG)
sensor fusion
Sensor phenomena and characterization
Sensors
Virtual reality
wearable sensors
Wrist
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Wrist-based hand gesture recognition has the potential to unlock naturalistic human-computer interaction for a vast array of virtual and augmented reality applications. Photoplethysmography (PPG), force myography (FMG), and accelerometry (ACC) have generally been proposed as isolated single sensing modalities for gesture recognition, but any of these alone is inherently limited in the amount of biological information it can collect during finger and hand movements. We thus propose a novel, wrist-based, PPG-FMG-ACC combined sensing approach based on a multi-head attention mechanism fusion convolutional neural network (CNN-AF) for gesture recognition. Nine subjects performed twelve hand gestures involving various wrist and finger postures. Experimental results showed that multi-modal fusion improved classification performance significantly (p < 0.01) compared to any single sensing modality, and the F1-score of the combined PPG-FMG-ACC approach was 40.1% higher than PPG alone, 27.4% higher than ACC alone, and 11.9% higher than FMG alone. To the best of our knowledge, this paper is the first to combine wrist-based PPG, FMG, and ACC signals for hand gesture recognition. These results could serve to inform wrist-based gesture recognition design (e.g., via a smartwatch) and thus expand the capabilities of intuitive and ubiquitous human-machine interaction.
ISSN:2168-2194
2168-2208
DOI:10.1109/JBHI.2022.3194017