BERT Representations for Video Question Answering

Visual question answering (VQA) aims at answering questions about the visual content of an image or a video. Currently, most work on VQA is focused on image-based question answering, and less attention has been paid into answering questions about videos. However, VQA in video presents some unique ch...

Full description

Saved in:
Bibliographic Details
Main Authors: Yang, Zekun, Garcia, Noa, Chu, Chenhui, Otani, Mayu, Nakashima, Yuta, Takemura, Haruo
Format: Conference Proceeding
Language:English
Subjects:
Bit error rate
Feature extraction
Knowledge discovery
Semantics
Task analysis
Visualization
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Visual question answering (VQA) aims at answering questions about the visual content of an image or a video. Currently, most work on VQA is focused on image-based question answering, and less attention has been paid into answering questions about videos. However, VQA in video presents some unique challenges that are worth studying: it not only requires to model a sequence of visual features over time, but often it also needs to reason about associated subtitles. In this work, we propose to use BERT, a sequential modelling technique based on Transformers, to encode the complex semantics from video clips. Our proposed model jointly captures the visual and language information of a video scene by encoding not only the subtitles but also a sequence of visual concepts with a pretrained language-based Transformer. In our experiments, we exhaustively study the performance of our model by taking different input arrangements, showing outstanding improvements when compared against previous work on two well-known video VQA datasets: TVQA and Pororo.
ISSN:2642-9381
DOI:10.1109/WACV45572.2020.9093596