Exploiting inter-frame regional correlation for efficient action recognition

• Published in: Expert systems with applications 2021-09, Vol.178, p.114829, Article 114829
• Main Authors: Xu, Yuecong, Yang, Jianfei, Mao, Kezhi, Yin, Jianxiong, See, Simon
• Format: Article
• Language: English
• Subjects: Action recognition; Correlation; Feature extraction; Feature recognition; Frames (data processing); Inter-frame correlation; Optical flow (image analysis); Pixels
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2021.114829

•We proposed a novel temporal feature extraction method for action recognition.•The novel method explores inter-frame correlation on the regional level.•Our method achieves state-of-the-art performance on benchmark datasets. Temporal feature extraction is an important issue in video-based action recognition. Optical flow is a popular method to extract temporal feature, which produces excellent performance thanks to its capacity of capturing pixel-level correlation information between consecutive frames. However, such a pixel-level correlation is extracted at the cost of high computational complexity and large storage resource. In this paper, we propose a novel temporal feature extraction method, Attentive Correlated Temporal Feature (ACTF), by exploring inter-frame correlation within a certain region. The proposed ACTF exploits both bilinear and linear correlations between successive frames on the regional level. Our method has the advantage of achieving performance comparable to or better than optical flow-based methods while avoiding the introduction of optical flow. Experimental results demonstrate our proposed method achieves the competitive performances of 96.3% on UCF101 and 76.3% on HMDB51 benchmark datasets.