Caption TLSTMs: combining transformer with LSTMs for image captioning

Image to captions has attracted widespread attention over the years. Recurrent neural networks (RNN) and their corresponding variants have been the mainstream when it comes to dealing with image captioning task for a long time. However, transformer-based models have shown powerful and promising perf...

Full description

Saved in:
Bibliographic Details
Published in:International journal of multimedia information retrieval 2022-06, Vol.11 (2), p.111-121
Main Authors: Yan, Jie, Xie, Yuxiang, Luan, Xidao, Guo, Yanming, Gong, Quanzhi, Feng, Suru
Format: Article
Language:English
Subjects:
Artificial intelligence
Computer Science
Data Mining and Knowledge Discovery
Database Management
Deep learning
Graphs
Image Processing and Computer Vision
Image quality
Information Storage and Retrieval
Information Systems Applications (incl.Internet)
Language
Machine translation
Multimedia Information Systems
Natural language
Neural networks
Recurrent neural networks
Regular Paper
Representations
Semantics
Transformers
Visual tasks
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Image to captions has attracted widespread attention over the years. Recurrent neural networks (RNN) and their corresponding variants have been the mainstream when it comes to dealing with image captioning task for a long time. However, transformer-based models have shown powerful and promising performance on visual tasks contrary to classic neural networks. In order to extract richer and more robust multimodal intersection feature representation, we improve the original abstract scene graph to caption model and propose the Caption TLSTMs which is made up of two LSTMs with T ransformer blocks in the middle of them in this paper. Compared with the model before improvement, the architecture of our Caption TLSTMs enables the entire network to make the most of the long-term dependencies and feature representation ability of the LSTM, while encoding the multimodal textual, visual and graphic information with the transformer blocks as well. Finally, experiments on VisualGenome and MSCOCO datasets have shown good performance in improving the general image caption generation quality, demonstrating the effectiveness of the Caption TLSTMs model.
ISSN:2192-6611
2192-662X
DOI:10.1007/s13735-022-00228-7