LD-MAN: Layout-Driven Multimodal Attention Network for Online News Sentiment Recognition

The prevailing use of both images and text to express opinions on the web leads to the need for multimodal sentiment recognition. Some commonly used social media data containing short text and few images, such as tweets and product reviews, have been well studied. However, it is still challenging to...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on multimedia 2021, Vol.23, p.1785-1798
Main Authors: Guo, Wenya, Zhang, Ying, Cai, Xiangrui, Meng, Lei, Yang, Jufeng, Yuan, Xiaojie
Format: Article
Language:English
Subjects:
Analytical models
article layout
attention mechanism
Datasets
Feature extraction
Image recognition
Layout
Layouts
Multimodal sentiment recognition
Neural networks
News
Object recognition
online news
Product reviews
Sentiment analysis
Visualization
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The prevailing use of both images and text to express opinions on the web leads to the need for multimodal sentiment recognition. Some commonly used social media data containing short text and few images, such as tweets and product reviews, have been well studied. However, it is still challenging to predict the readers' sentiment after reading online news articles, since news articles often have more complicated structures, e.g., longer text and more images. To address this problem, we propose a layout-driven multimodal attention network (LD-MAN) to recognize news sentiment in an end-to-end manner. Rather than modeling text and images individually, LD-MAN uses the layout of online news to align images with the corresponding text. Specifically, it exploits a set of distance-based coefficients to model the image locations and measure the contextual relationship between images and text. LD-MAN then learns the affective representations of the articles from the aligned text and images using a multimodal attention mechanism. Considering the lack of relevant datasets in this field, we collect two multimodal online news datasets, containing a total of 14,566 articles with 56,260 images and 251,202 words. Experimental results demonstrate that the proposed method performs favorably compared with state-of-the-art approaches. We will release all the codes, models and datasets to the community.
ISSN:1520-9210
1941-0077
DOI:10.1109/TMM.2020.3003648