Self-Calibrated Multi-Sensor Wearable for Hand Tracking and Modeling

We present a multi-sensor system for consistent 3D hand pose tracking and modeling that leverages the advantages of both wearable and optical sensors. Specifically, we employ a stretch-sensing soft glove and three IMUs in combination with an RGB-D camera. Different sensor modalities are fused based...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on visualization and computer graphics 2023-03, Vol.29 (3), p.1769-1784
Main Authors: Gosala, Nikhil, Wang, Fangjinhua, Cui, Zhaopeng, Liang, Hanxue, Glauser, Oliver, Wu, Shihao, Sorkine-Hornung, Olga
Format: Article
Language:English
Subjects:
Calibration
Cameras
Computational modeling
Computer Graphics
Hand
Hand tracking
Heating systems
Modelling
Occlusion
Optical measuring instruments
Solid modeling
Three-dimensional displays
Tracking
Wearable Electronic Devices
Wearable sensors
Wearable technology
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We present a multi-sensor system for consistent 3D hand pose tracking and modeling that leverages the advantages of both wearable and optical sensors. Specifically, we employ a stretch-sensing soft glove and three IMUs in combination with an RGB-D camera. Different sensor modalities are fused based on the availability and confidence estimation, enabling seamless hand tracking in challenging environments with partial or even complete occlusion. To maximize the accuracy while maintaining high ease-of-use, we propose an automated user calibration that uses the RGB-D camera data to refine both the glove mapping model and the multi-IMU system parameters. Extensive experiments show that our setup outperforms the wearable-only approaches when the hand is in the field-of-view and outplays the camera-only methods when the hand is occluded.
ISSN:1077-2626
1941-0506
DOI:10.1109/TVCG.2021.3131230