Image captioning model using attention and object features to mimic human image understanding

Image captioning spans the fields of computer vision and natural language processing. The image captioning task generalizes object detection where the descriptions are a single word. Recently, most research on image captioning has focused on deep learning techniques, especially Encoder-Decoder model...

Full description

Saved in:
Bibliographic Details
Published in:Journal of big data 2022-02, Vol.9 (1), p.1-16, Article 20
Main Authors: Al-Malla, Muhammad Abdelhadie, Jafar, Assef, Ghneim, Nada
Format: Article
Language:English
Subjects:
Accuracy
Artificial neural networks
Big Data
Boxes
Cider
Coders
Communications Engineering
Computational Science and Engineering
Computer Science
Computer vision
Convolutional neural network
Data Mining and Knowledge Discovery
Database Management
Deep learning
Design of experiments
Encoders-Decoders
Feature extraction
Image captioning
Information Storage and Retrieval
Language
Machine learning
Mathematical Applications in Computer Science
Natural language
Natural language processing
Networks
Neural networks
Object features
Object recognition
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Image captioning spans the fields of computer vision and natural language processing. The image captioning task generalizes object detection where the descriptions are a single word. Recently, most research on image captioning has focused on deep learning techniques, especially Encoder-Decoder models with Convolutional Neural Network (CNN) feature extraction. However, few works have tried using object detection features to increase the quality of the generated captions. This paper presents an attention-based, Encoder-Decoder deep architecture that makes use of convolutional features extracted from a CNN model pre-trained on ImageNet (Xception), together with object features extracted from the YOLOv4 model, pre-trained on MS COCO. This paper also introduces a new positional encoding scheme for object features, the “importance factor”. Our model was tested on the MS COCO and Flickr30k datasets, and the performance is compared to performance in similar works. Our new feature extraction scheme raises the CIDEr score by 15.04%. The code is available at: https://github.com/abdelhadie-almalla/image_captioning
ISSN:2196-1115
2196-1115
DOI:10.1186/s40537-022-00571-w