DANet: Temporal Action Localization with Double Attention

Temporal action localization (TAL) aims to predict action instance categories in videos and identify their start and end times. However, existing Transformer-based backbones focus only on global or local features, resulting in the loss of information. In addition, both global and local self-attentio...

Full description

Saved in:
Bibliographic Details
Published in:Applied sciences 2023-06, Vol.13 (12), p.7176
Main Authors: Sun, Jianing, Wu, Xuan, Xiao, Yubin, Wu, Chunguo, Liang, Yanchun, Liang, Yi, Wang, Liupu, Zhou, You
Format: Article
Language:English
Subjects:
Ablation
Analysis
artificial intelligence
attention mechanism
Computational linguistics
computer vision
Datasets
Language processing
Localization
Natural language interfaces
Proposals
Semantics
temporal action localization
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Temporal action localization (TAL) aims to predict action instance categories in videos and identify their start and end times. However, existing Transformer-based backbones focus only on global or local features, resulting in the loss of information. In addition, both global and local self-attention mechanisms tend to average embeddings, thereby reducing the preservation of critical features. To solve these two problems better, we propose two kinds of attention mechanisms, namely multi-headed local self-attention (MLSA) and max-average pooling attention (MA) to extract simultaneously local and global features. In MA, max-pooling is used to select the most critical information from local clip embeddings instead of averaging embeddings, and average-pooling is used to aggregate global features. We use MLSA for modeling local temporal context. In addition, to enhance collaboration between MA and MLSA, we propose the double attention block (DABlock), comprising MA and MLSA. Finally, we propose the final network double attention network (DANet), composed of DABlocks and other advanced blocks. To evaluate DANet’s performance, we conduct extensive experiments for the TAL task. Experimental results demonstrate that DANet outperforms the other state-of-the-art models on all datasets. Finally, ablation studies demonstrate the effectiveness of the proposed MLSA and MA. Compared with structures using backbone with convolution and global Transformer, DABlock consisting of MLSA and MA has a superior performance, achieving an 8% and 0.5% improvement on overall average mAP, respectively.
ISSN:2076-3417
2076-3417
DOI:10.3390/app13127176