BCMF: A bidirectional cross-modal fusion model for fake news detection

• Published in: Information processing & management 2022-09, Vol.59 (5), p.103063, Article 103063
• Main Authors: Yu, Chuanming, Ma, Yinxue, An, Lu, Li, Gang
• Format: Article
• Language: English
• Subjects: Contextualized embedding; Cross-modal fusion; Deep learning; Fake news detection
• ISSN: 0306-4573; 1873-5371
• DOI: 10.1016/j.ipm.2022.103063

•We propose a novel model, namely BCMF, for fake news detection.•BCMF leverages both contextualized visual embeddings and bi-directional fusions.•We propose a bi-directional cross-modal aggregation mechanism to deeply fuse the visual and textual information.•The model outperforms most of the state-of-the-art methods on four datasets.•The research sheds light on the role of bidirectional cross-modal fusion. In recent years, fake news detection has been a significant task attracting much attention. However, most current approaches utilize the features from a single modality, such as text or image, while the comprehensive fusion between features of different modalities has been ignored. To deal with the above problem, we propose a novel model named Bidirectional Cross-Modal Fusion (BCMF), which comprehensively integrates the textual and visual representations in a bidirectional manner. Specifically, the proposed model is decomposed into four submodules, i.e., the input embedding, the image2text fusion, the text2image fusion, and the prediction module. We conduct intensive experiments on four real-world datasets, i.e., Weibo, Twitter, Politi, and Gossip. The results show 2.2, 2.5, 4.9, and 3.1 percentage points of improvements in classification accuracy compared to the state-of-the-art methods on Weibo, Twitter, Politi, and Gossip, respectively. The experimental results suggest that the proposed model could better capture integrated information of different modalities and has high generalizability among different datasets. Further experiments suggest that the bidirectional fusions, the number of multi-attention heads, and the aggregating function could impact the performance of the cross-modal fake news detection. The research sheds light on the role of bidirectional cross-modal fusion in leveraging multi-modal information to improve the effect of fake news detection.