Discriminative Segment Focus Network for Fine-grained Video Action Recognition

• Published in: ACM transactions on multimedia computing communications and applications 2024-05, Vol.20 (7), p.1-20, Article 218
• Main Authors: Sun, Baoli, Ye, Xinchen, Yan, Tiantian, Wang, Zhihui, Li, Haojie, Wang, Zhiyong
• Format: Article
• Language: English
• Subjects: Activity recognition and understanding; Computing methodologies
• ISSN: 1551-6857; 1551-6865
• DOI: 10.1145/3654671

Fine-grained video action recognition aims at identifying minor and discriminative variations among fine categories of actions. While many recent action recognition methods have been proposed to better model spatio-temporal representations, how to model the interactions among discriminative atomic actions to effectively characterize inter-class and intra-class variations has been neglected, which is vital for understanding fine-grained actions. In this work, we devise a Discriminative Segment Focus Network (DSFNet) to mine the discriminability of segment correlations and localize discriminative action-relevant segments for fine-grained video action recognition. Firstly, we propose a hierarchic correlation reasoning (HCR) module which explicitly establishes correlations between different segments at multiple temporal scales and enhances each segment by exploiting the correlations with other segments. Secondly, a discriminative segment focus (DSF) module is devised to localize the most action-relevant segments from the enhanced representations of HCR by enforcing the consistency between the discriminability and the classification confidence of a given segment with a consistency constraint. Finally, these localized segment representations are combined with the global action representation of the whole video for boosting final recognition. Extensive experimental results on two fine-grained action recognition datasets, i.e., FineGym and Diving48, and two action recognition datasets, i.e., Kinetics400 and Something-Something, demonstrate the effectiveness of our approach compared with the state-of-the-art methods.