Switching Structured Prediction for Simple and Complex Human Activity Recognition

• Published in: IEEE transactions on cybernetics 2021-12, Vol.51 (12), p.5859-5870
• Main Authors: Arzani, Mohammad M., Fathy, Mahmood, Azirani, Ahmad A., Adeli, Ehsan
• Format: Article
• Language: English
• Subjects: Activity recognition; Distributed structured prediction (DSP); Graphical models; hidden state conditional random fields (CRFs); Human Activities; Human activity recognition; human activity recognition (HAR); Human factors; Humans; latent structured support vector machines; Manifolds; Models, Statistical; Moving object recognition; Optimization; Parameterization; Predictive models; probabilistic graphical models (PGMs); RGBD; Robot vision; Robots; Skeleton; Subspaces; Support vector machines; Switches; Switching; Task analysis
• ISSN: 2168-2267; 2168-2275
• DOI: 10.1109/TCYB.2019.2960481

Automatic human activity recognition is an integral part of any interactive application involving humans (e.g., human-robot interaction systems). One of the main challenges for activity recognition is the diversity in the way individuals often perform activities. Furthermore, changes in any of the environment factors (i.e., illumination, complex background, human body shapes, viewpoint, etc.) intensify this challenge. In addition, there are different types of activities that robots need to interpret for seamless interaction with humans. Some activities are short, quick, and simple (e.g., sitting), while others may be detailed/complex, and spread throughout a long span of time (e.g., washing mouth). In this article, we recognize the activities within the context of graphical models in a sequence-labeling framework based on skeleton data. We propose a new structured prediction strategy based on probabilistic graphical models (PGMs) to recognize both types of activities (i.e., complex and simple). These activity types are often spanned in very diverse subspaces in the space of all possible activities, which would require different model parameterizations. In order to deal with these parameterization and structural breaks across models, a category-switching scheme is proposed to switch over the models based on the activity types. For parameter optimization, we utilize a distributed structured prediction technique to implement our model in a distributed setting. The method is tested on three widely used datasets (CAD-60, UT-Kinect, and Florence 3-D) that cover both activity types. The results illustrate that our proposed method is able to recognize simple and complex activities while the previous work concentrated on only one of these two main types.