Deep Captioning with Attention-Based Visual Concept Transfer Mechanism for Enriching Description

In this paper, we propose a novel deep captioning framework called Attention-based multimodal recurrent neural network with Visual Concept Transfer Mechanism (A-VCTM). There are three advantages of the proposed A-VCTM. (1) A multimodal layer is used to integrate the visual representation and context...

Full description

Saved in:
Bibliographic Details
Published in:Neural processing letters 2019-10, Vol.50 (2), p.1891-1905
Main Authors: Zhang, Junxuan, Hu, Haifeng
Format: Article
Language:English
Subjects:
Artificial Intelligence
Bridge construction
Complex Systems
Computational Intelligence
Computer Science
Context
Probability distribution
Recurrent neural networks
Representations
Semantics
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we propose a novel deep captioning framework called Attention-based multimodal recurrent neural network with Visual Concept Transfer Mechanism (A-VCTM). There are three advantages of the proposed A-VCTM. (1) A multimodal layer is used to integrate the visual representation and context representation together, building a bridge that connects context information with visual information directly. (2) An attention mechanism is introduced to lead the model to focus on the regions corresponding to the next word to be generated (3) We propose a visual concept transfer mechanism to generate novel visual concepts and enrich the description sentences. Qualitative and quantitative results on two standard benchmarks, MSCOCO and Flickr30K show the effectiveness and practicability of the proposed A-VCTM framework.
ISSN:1370-4621
1573-773X
DOI:10.1007/s11063-019-09978-8