A Cuboid CNN Model With an Attention Mechanism for Skeleton-Based Action Recognition

The introduction of depth sensors such as Microsoft Kinect have driven research in human action recognition. Human skeletal data collected from depth sensors convey a significant amount of information for action recognition. While there has been considerable progress in action recognition, most exis...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on multimedia 2020-11, Vol.22 (11), p.2977-2989
Main Authors: Zhu, Kaijun, Wang, Ruxin, Zhao, Qingsong, Cheng, Jun, Tao, Dapeng
Format: Article
Language:English
Subjects:
action recognition
attention mechanism
Body parts
CNN
Datasets
feature cuboid
Feature extraction
Hidden Markov models
Human activity recognition
Human motion
Joints (anatomy)
Neural networks
Representations
Sensors
Skeleton
Spatiotemporal phenomena
Three-dimensional displays
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The introduction of depth sensors such as Microsoft Kinect have driven research in human action recognition. Human skeletal data collected from depth sensors convey a significant amount of information for action recognition. While there has been considerable progress in action recognition, most existing skeleton-based approaches neglect the fact that not all human body parts move during many actions, and they fail to consider the ordinal positions of body joints. Here, and motivated by the fact that an action's category is determined by local joint movements, we propose a cuboid model for skeleton-based action recognition. Specifically, a cuboid arranging strategy is developed to organize the pairwise displacements between all body joints to obtain a cuboid action representation. Such a representation is well structured and allows deep CNN models to focus analyses on actions. Moreover, an attention mechanism is exploited in the deep model, such that the most relevant features are extracted. Extensive experiments on our new Yunnan University-Chinese Academy of Sciences-Multimodal Human Action Dataset (CAS-YNU MHAD), the NTU RGB+D dataset, the UTD-MHAD dataset, and the UTKinect-Action3D dataset demonstrate the effectiveness of our method compared to the current state-of-the-art.
ISSN:1520-9210
1941-0077
DOI:10.1109/TMM.2019.2962304