Continuous frame motion sensitive self-supervised collaborative network for video representation learning

• Published in: Advanced engineering informatics 2023-04, Vol.56, p.101941, Article 101941
• Main Authors: Bi, Shuai, Hu, Zhengping, Zhao, Mengyao, Zhang, Hehao, Di, Jirui, Sun, Zhe
• Format: Article
• Language: English
• Subjects: Action recognition; Global–local contrastive learning; Pretext task; Self-supervised representation learning; Video retrieval
• ISSN: 1474-0346; 1873-5320
• DOI: 10.1016/j.aei.2023.101941

Motion, as a feature of video that changes in temporal sequences, is crucial to visual understanding. The powerful video representation and extraction models are typically able to focus attention on motion features in challenging dynamic environments to complete more complex video understanding tasks. However, previous approaches discriminate mainly based on similar features in the spatial or temporal domain, ignoring the interdependence of consecutive video frames. In this paper, we propose the motion sensitive self-supervised collaborative network, a video representation learning framework that exploits a pretext task to assist feature comparison and strengthen the spatiotemporal discrimination power of the model. Specifically, we first propose the motion-aware module, which extracts consecutive motion features from the spatial regions by frame difference. The global–local contrastive module is then introduced, with context and enhanced video snippets being defined as appropriate positive samples for a broader feature similarity comparison. Finally, we introduce the snippet operation prediction module, which further assists contrastive learning to obtain more reliable global semantics by sensing changes in continuous frame features. Experimental results demonstrate that our work can effectively extract robust motion features and achieve competitive performance compared with other state-of-the-art self-supervised methods on downstream action recognition and video retrieval tasks. •Accurate extraction of continuous motion features in complex environments.•Acquire global and local spatio-temporal features via correlation of sequences.•Obtain supervised information for feature discrimination in continuous clips.•Achieve competitive performance with limited data pre-training.