DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering

Video question answering is a challenging task, which requires agents to be able to understand rich video contents and perform spatial-temporal reasoning. However, existing graph-based methods fail to perform multi-step reasoning well, neglecting two properties of VideoQA: (1) Even for the same vide...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on multimedia 2022, Vol.24, p.3369-3380
Main Authors: Wang, Jianyu, Bao, Bing-Kun, Xu, Changsheng
Format: Article
Language:English
Subjects:
Benchmarks
Cognition
Datasets
Dogs
Feature extraction
Fuses
graph neural network
Knowledge discovery
multi-modal
multi-step reasoning
Questions
Reasoning
Task analysis
Video question answering
Visual observation
Visualization
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Video question answering is a challenging task, which requires agents to be able to understand rich video contents and perform spatial-temporal reasoning. However, existing graph-based methods fail to perform multi-step reasoning well, neglecting two properties of VideoQA: (1) Even for the same video, different questions may require different amount of video clips or objects to infer the answer with relational reasoning; (2) During reasoning, appearance and motion features have complicated interdependence which are correlated and complementary to each other. Based on these observations, we propose a Dual-Visual Graph Reasoning Unit (DualVGR) which reasons over videos in an end-to-end fashion. The first contribution of our DualVGR is the design of an explainable Query Punishment Module, which can filter out irrelevant visual features through multiple cycles of reasoning. The second contribution is the proposed Video-based Multi-view Graph Attention Network, which captures the relations between appearance and motion features. Our DualVGR network achieves state-of-the-art performance on the benchmark MSVD-QA and SVQA datasets, and demonstrates competitive results on benchmark MSRVTT-QA datasets. Our code is available at https://github.com/MM-IR/DualVGR-VideoQA .
ISSN:1520-9210
1941-0077
DOI:10.1109/TMM.2021.3097171