Co-Attention Network With Question Type for Visual Question Answering

Visual Question Answering (VQA) is a challenging multi-modal learning task since it requires an understanding of both visual and textual modalities simultaneously. Therefore, the approaches used to represent the images and questions in a fine-grained manner play key roles in the performance. In orde...

Full description

Saved in:
Bibliographic Details
Published in:IEEE access 2019, Vol.7, p.40771-40781
Main Authors: Yang, Chao, Jiang, Mengqi, Jiang, Bin, Zhou, Weixin, Li, Keqin
Format: Article
Language:English
Subjects:
Co-attention
Cognition
Cognitive tasks
Computational modeling
Computer architecture
Feature extraction
Knowledge discovery
Learning
Natural language processing
question type
Questions
Representations
self-attention
Task analysis
visual question answering
Visualization
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Visual Question Answering (VQA) is a challenging multi-modal learning task since it requires an understanding of both visual and textual modalities simultaneously. Therefore, the approaches used to represent the images and questions in a fine-grained manner play key roles in the performance. In order to obtain the fine-grained image and question representations, we develop a co-attention mechanism using an end-to-end deep network architecture to jointly learn both the image and the question features. Specifically, textual attention implemented by a self-attention model will reduce unrelated information and extract more discriminative features for question-level representations, which is in turn used to guide visual attention. We also note that a lot of finished works use complex models to extract feature representations but neglect to use high-level information summary such as question types in learning. Hence, we introduce the question type in our work by directly concatenating it with the multi-modal joint representation to narrow down the candidate answer space. A new network architecture combining the proposed co-attention mechanism and question type provides a unified model for VQA. The extensive experiments on two public datasets demonstrate the effectiveness of our model as compared with several state-of-the-art approaches.
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2019.2908035