Motion-Driven Visual Tempo Learning for Video-Based Action Recognition

Action visual tempo characterizes the dynamics and the temporal scale of an action, which is helpful to distinguish human actions that share high similarities in visual dynamics and appearance. Previous methods capture the visual tempo either by sampling raw videos with multiple rates, which require...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on image processing 2022-01, Vol.31, p.4104-4116
Main Authors: Liu, Yuanzhong, Yuan, Junsong, Tu, Zhigang
Format: Article
Language:English
Subjects:
Action recognition
Activity recognition
Backbone
Correlation
Dynamics
Feature extraction
Modules
Monolayers
multi-scale temporal structure
Multilayers
Sampling
Semantics
Source code
Spatiotemporal phenomena
temporal correlation module
Three-dimensional displays
visual tempo
Visualization
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Action visual tempo characterizes the dynamics and the temporal scale of an action, which is helpful to distinguish human actions that share high similarities in visual dynamics and appearance. Previous methods capture the visual tempo either by sampling raw videos with multiple rates, which require a costly multi-layer network to handle each rate, or by hierarchically sampling backbone features, which rely heavily on high-level features that miss fine-grained temporal dynamics. In this work, we propose a Temporal Correlation Module (TCM), which can be easily embedded into the current action recognition backbones in a plug-in-and-play manner, to extract action visual tempo from low-level backbone features at single-layer remarkably. Specifically, our TCM contains two main components: a Multi-scale Temporal Dynamics Module (MTDM) and a Temporal Attention Module (TAM). MTDM applies a correlation operation to learn pixel-wise fine-grained temporal dynamics for both fast-tempo and slow-tempo. TAM adaptively emphasizes expressive features and suppresses inessential ones via analyzing the global information across various tempos. Extensive experiments conducted on several action recognition benchmarks, e.g. Something-Something V1&V2, Kinetics-400, UCF-101, and HMDB-51, have demonstrated that the proposed TCM is effective to promote the performance of the existing video-based action recognition models for a large margin. The source code is publicly released at https://github.com/zphyix/TCM .
ISSN:1057-7149
1941-0042
DOI:10.1109/TIP.2022.3180585