Caption TLSTMs: combining transformer with LSTMs for image captioning

• Published in: International journal of multimedia information retrieval 2022-06, Vol.11 (2), p.111-121
• Main Authors: Yan, Jie, Xie, Yuxiang, Luan, Xidao, Guo, Yanming, Gong, Quanzhi, Feng, Suru
• Format: Article
• Language: English
• Subjects: Artificial intelligence; Computer Science; Data Mining and Knowledge Discovery; Database Management; Deep learning; Graphs; Image Processing and Computer Vision; Image quality; Information Storage and Retrieval; Information Systems Applications (incl.Internet); Language; Machine translation; Multimedia Information Systems; Natural language; Neural networks; Recurrent neural networks; Regular Paper; Representations; Semantics; Transformers; Visual tasks
• ISSN: 2192-6611; 2192-662X
• DOI: 10.1007/s13735-022-00228-7

Image to captions has attracted widespread attention over the years. Recurrent neural networks (RNN) and their corresponding variants have been the mainstream when it comes to dealing with image captioning task for a long time. However, transformer-based models have shown powerful and promising performance on visual tasks contrary to classic neural networks. In order to extract richer and more robust multimodal intersection feature representation, we improve the original abstract scene graph to caption model and propose the Caption TLSTMs which is made up of two LSTMs with T ransformer blocks in the middle of them in this paper. Compared with the model before improvement, the architecture of our Caption TLSTMs enables the entire network to make the most of the long-term dependencies and feature representation ability of the LSTM, while encoding the multimodal textual, visual and graphic information with the transformer blocks as well. Finally, experiments on VisualGenome and MSCOCO datasets have shown good performance in improving the general image caption generation quality, demonstrating the effectiveness of the Caption TLSTMs model.