Online video visual relation detection with hierarchical multi-modal fusion

With the development of artificial intelligence technology, visual scene understanding has become a hot research topic. Online visual relation detection plays an important role in dynamic visual scene understanding. However, the complete modeling of dynamic relations and how to utilize a large amoun...

Full description

Saved in:
Bibliographic Details
Published in:Multimedia tools and applications 2024-01, Vol.83 (24), p.65707-65727
Main Authors: He, Yuxuan, Gan, Ming-Gang, Ma, Qianzhao
Format: Article
Language:English
Subjects:
Ablation
Artificial intelligence
Computer Communication Networks
Computer Science
Data Structures and Information Theory
Datasets
Language
Linguistics
Methods
Modules
Multimedia
Multimedia Information Systems
Natural language
Neural networks
Robots
Scene analysis
Semantics
Special Purpose and Application-Based Systems
Track 6: Computer Vision for Multimedia Applications
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:With the development of artificial intelligence technology, visual scene understanding has become a hot research topic. Online visual relation detection plays an important role in dynamic visual scene understanding. However, the complete modeling of dynamic relations and how to utilize a large amount of video content to infer visual relations are two difficult problems needed to be solved. Therefore, we propose Hierarchical Multi-Modal Fusion network for online video visual relation detection. We propose ASE-GCN to model dynamic scenes from different perspectives in order to fully capture visual relations in dynamic scenes. Meanwhile, we use trajectory features and natural language features as additional auxiliary features to describe the visual scene together with high-level visual features constructed by ASE-GCN. In order to make full use of these information to infer the visual relation, we design Hierarchical Fusion module before the relation predictor, which fuses the multi-role and multi-modal features using the methods based on attention and trilinear pooling. Comparative experiments on the ImageNet-VidVRD dataset demonstrate that our network outperforms other methods, while ablation studies verify the proposed modules are effective.
ISSN:1573-7721
1380-7501
1573-7721
DOI:10.1007/s11042-023-15310-3