Action recognition and tracking via deep representation extraction and motion bases learning

• Published in: Multimedia tools and applications 2022-04, Vol.81 (9), p.11845-11864
• Main Authors: Li, Hao-Ting, Liu, Yung-Pin, Chang, Yun-Kai, Chiang, Chen-Kuo
• Format: Article
• Language: English
• Subjects: 1195: Deep Learning for Multimedia Signal Processing and Applications; Activity recognition; Centroids; Clustering; Coders; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Degrees of freedom; Feature extraction; Learning; Multimedia Information Systems; Representations; Rotation; Sensors; Special Purpose and Application-Based Systems; Tracking; Virtual reality
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-021-11888-8

Action recognition and positional tracking are critical issues in many applications in Virtual Reality (VR). In this paper, a novel feature representation method is proposed to recognize actions based on sensor signals. The feature extraction is achieved by jointly learning Convolutional Auto-Encoder (CAE) and the representation of motion bases via clustering, which is called the Sequence of Cluster Centroids (SoCC). Then, the learned features are used to train the action recognition classifier. We have collected new dataset of actions of limbs by sensor signals. In addition, a novel action tracking method is proposed for the VR environment. It extends the sensor signals from three Degrees of Freedom (DoF) of rotation to 6DoF of position plus rotation. Experimental results demonstrate that CAE-SoCC feature is effective for action recognition and accurate prediction of position displacement.