JARViS: Detecting actions in video using unified actor-scene context relation modeling

• Published in: Neurocomputing (Amsterdam) 2024-12, Vol.610, p.128616, Article 128616
• Main Authors: Lee, Seok Hwan, Son, Taein, Seo, Soo Won, Kim, Jisong, Choi, Jun Won
• Format: Article
• Language: English
• Subjects: Action detection; Deep learning; Spatio-temporal context; Unified transformer; Video action detection
• ISSN: 0925-2312
• DOI: 10.1016/j.neucom.2024.128616

Video action detection (VAD) is a formidable vision task that involves the localization and classification of actions within the spatial and temporal dimensions of a video clip. Among the myriad VAD architectures, two-stage VAD methods utilize a pre-trained person detector to extract the region of interest features, subsequently employing these features for action detection. However, the performance of two-stage VAD methods has been limited as they depend solely on localized actor features to infer action semantics. In this study, we propose a new two-stage VAD framework called Joint Actor-scene context Relation modeling based on Visual Semantics (JARViS), which effectively consolidates cross-modal action semantics distributed globally across spatial and temporal dimensions using Transformer attention. JARViS employs a person detector to produce densely sampled actor features from a keyframe. Concurrently, it uses a video backbone to create spatio-temporal scene features from a video clip. Finally, the fine-grained interactions between actors and scenes are modeled through a Unified Action-Scene Context Transformer to directly output the final set of actions in parallel. Our experimental results demonstrate that JARViS outperforms existing methods by significant margins and achieves state-of-the-art performance on three popular VAD datasets, including AVA, UCF101-24, and JHMDB51-21.