Multimodal Fusion Method Based on Self-Attention Mechanism

Multimodal fusion is one of the popular research directions of multimodal research, and it is also an emerging research field of artificial intelligence. Multimodal fusion is aimed at taking advantage of the complementarity of heterogeneous data and providing reliable classification for the model. M...

Full description

Saved in:
Bibliographic Details
Published in:Wireless communications and mobile computing 2020, Vol.2020 (2020), p.1-8
Main Authors: Xu, Guoxia, Hua, Yingying, Shi, Yu, Wang, Ze, Zhu, Hu, Deng, Lizhen
Format: Article
Language:English
Subjects:
Artificial intelligence
Data integration
Decomposition
Mathematical analysis
Methods
Neural networks
Representations
Task complexity
Tensors
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Multimodal fusion is one of the popular research directions of multimodal research, and it is also an emerging research field of artificial intelligence. Multimodal fusion is aimed at taking advantage of the complementarity of heterogeneous data and providing reliable classification for the model. Multimodal data fusion is to transform data from multiple single-mode representations to a compact multimodal representation. In previous multimodal data fusion studies, most of the research in this field used multimodal representations of tensors. As the input is converted into a tensor, the dimensions and computational complexity increase exponentially. In this paper, we propose a low-rank tensor multimodal fusion method with an attention mechanism, which improves efficiency and reduces computational complexity. We evaluate our model through three multimodal fusion tasks, which are based on a public data set: CMU-MOSI, IEMOCAP, and POM. Our model achieves a good performance while flexibly capturing the global and local connections. Compared with other multimodal fusions represented by tensors, experiments show that our model can achieve better results steadily under a series of attention mechanisms.
ISSN:1530-8669
1530-8677
DOI:10.1155/2020/8843186