Robust Human Activity Recognition Using Multimodal Feature-Level Fusion

• Published in: IEEE access 2019, Vol.7, p.60736-60751
• Main Authors: Ehatisham-Ul-Haq, Muhammad, Javed, Ali, Azam, Muhammad Awais, Malik, Hafiz M. A., Irtaza, Aun, Lee, Ik Hyun, Mahmood, Muhammad Tariq
• Format: Article
• Language: English
• Subjects: Cameras; Clutter; Computer vision; Dense HOG; depth sensor; Feature extraction; Feature recognition; feature-level fusion; Field of view; Histograms; human action recognition; Human activity recognition; Inertial sensing devices; inertial sensor; Moving object recognition; Occupancy; RGB camera; Robotics; Robustness; Sensor fusion; Sensor phenomena and characterization; Sensors; Spacetime; Support vector machines; Three-dimensional displays; Video; Wearable sensors; Wearable technology
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2019.2913393

Automated recognition of human activities or actions has great significance as it incorporates wide-ranging applications, including surveillance, robotics, and personal health monitoring. Over the past few years, many computer vision-based methods have been developed for recognizing human actions from RGB and depth camera videos. These methods include space-time trajectory, motion encoding, key poses extraction, space-time occupancy patterns, depth motion maps, and skeleton joints. However, these camera-based approaches are affected by background clutter and illumination changes and applicable to a limited field of view only. Wearable inertial sensors provide a viable solution to these challenges but are subject to several limitations such as location and orientation sensitivity. Due to the complementary trait of the data obtained from the camera and inertial sensors, the utilization of multiple sensing modalities for accurate recognition of human actions is gradually increasing. This paper presents a viable multimodal feature-level fusion approach for robust human action recognition, which utilizes data from multiple sensors, including RGB camera, depth sensor, and wearable inertial sensors. We extracted the computationally efficient features from the data obtained from RGB-D video camera and inertial body sensors. These features include densely extracted histogram of oriented gradient (HOG) features from RGB/depth videos and statistical signal attributes from wearable sensors data. The proposed human action recognition (HAR) framework is tested on a publicly available multimodal human action dataset UTD-MHAD consisting of 27 different human actions. K-nearest neighbor and support vector machine classifiers are used for training and testing the proposed fusion model for HAR. The experimental results indicate that the proposed scheme achieves better recognition results as compared to the state of the art. The feature-level fusion of RGB and inertial sensors provides the overall best performance for the proposed system, with an accuracy rate of 97.6%.