CWI: A multimodal deep learning approach for named entity recognition from social media using character, word and image features

• Published in: Neural computing & applications 2022-02, Vol.34 (3), p.1905-1922
• Main Authors: Asgari-Chenaghlu, Meysam, Feizi-Derakhshi, M. Reza, Farzinvash, Leili, Balafar, M. A., Motamed, Cina
• Format: Article
• Language: English
• Subjects: Artificial Intelligence; Computational Biology/Bioinformatics; Computational Science and Engineering; Computer Science; Data Mining and Knowledge Discovery; Deep learning; Digital media; Feature extraction; Feature recognition; Image Processing and Computer Vision; Object recognition; Original Article; Probability and Statistics in Computer Science; Social networks; Transformers; User generated content
• ISSN: 0941-0643; 1433-3058
• DOI: 10.1007/s00521-021-06488-4

Named entity recognition (NER) from social media posts is a challenging task. User-generated content that forms the nature of social media is noisy and contains grammatical and linguistic errors. This noisy content makes tasks such as NER much harder. We propose two novel deep learning approaches utilizing multimodal deep learning and transformers. Both of our approaches use image features from short social media posts to provide better results on the NER task. On the first approach, we extract image features using InceptionV3 and use fusion to combine textual and image features. This approach presents more reliable name entity recognition when the images related to the entities are provided by the user. On the second approach, we use image features combined with text and feed it into a BERT-like transformer. The experimental results using precision, recall, and F1 score metrics show the superiority of our work compared to other state-of-the-art NER solutions.