Spatio-Temporal Representation Matching-Based Open-Set Action Recognition by Joint Learning of Motion and Appearance

• Published in: IEEE access 2019, Vol.7, p.165997-166010
• Main Authors: Yoon, Yongsang, Yu, Jongmin, Jeon, Moongu
• Format: Article
• Language: English
• Subjects: Action recognition; Adaptive optics; Convolution; Feature extraction; joint learning of motion and appearance; Learning; Matching; open-set recognition; Recognition; Representations; spatio-temporal representation; Training; Trajectory; Two dimensional displays; Visualization
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2019.2953455

In this paper, we propose the spatio-temporal representation matching (STRM) for video-based action recognition under the open-set condition. Open-set action recognition is a more challenging problem than closed-set action recognition since samples of the untrained action class need to be recognized and most of the conventional frameworks are likely to give a false prediction. To handle the untrained action classes, we propose STRM, which involves jointly learning both motion and appearance. STRM extracts spatio-temporal representations from video clips through a joint learning pipeline with both motion and appearance information. Then, STRM computes the similarities between the ST-representations to find the one with highest similarity. We set the experimental protocol for open-set action recognition and carried out experiments on UCF101 and HMDB51 to evaluate STRM. We first investigated the effects of different hyper-parameter settings on STRM, and then compared its performance with existing state-of-the-art methods. The experimental results showed that the proposed method not only outperformed existing methods under the open-set condition, but also provided comparable performance to the state-of-the-art methods under the closed-set condition.