A Quantum-Like multimodal network framework for modeling interaction dynamics in multiparty conversational sentiment analysis

• Published in: Information fusion 2020-10, Vol.62, p.14-31
• Main Authors: Zhang, Yazhou, Song, Dawei, Li, Xiang, Zhang, Peng, Wang, Panpan, Rong, Lu, Yu, Guangliang, Wang, Bo
• Format: Article
• Language: English
• Subjects: Human conversation; Interactive dynamics; Long short-term memory (LSTM) network; Multimodal sentiment analysis; Quantum theory
• ISSN: 1566-2535; 1872-6305
• DOI: 10.1016/j.inffus.2020.04.003

•A quantum-like multimodal network framework for multimodal sentiment analysis in conversations is proposed.•A quantum interference-inspired multimodal decision fusion method is proposed.•A quantum measurement-inspired strong-weak influence model is proposed.•Extensive experiments have shown its effectiveness. Sentiment analysis in conversations is an emerging yet challenging artificial intelligence (AI) task. It aims to discover the affective states and emotional changes of speakers involved in a conversation on the basis of their opinions, which are carried by different modalities of information (e.g., a video associated with a transcript). There exists a wealth of intra- and inter-utterance interaction information that affects the emotions of speakers in a complex and dynamic way. How to accurately and comprehensively model complicated interactions is the key problem of the field. To fill this gap, in this paper, we propose a novel and comprehensive framework for multimodal sentiment analysis in conversations, called a quantum-like multimodal network (QMN), which leverages the mathematical formalism of quantum theory (QT) and a long short-term memory (LSTM) network. Specifically, the QMN framework consists of a multimodal decision fusion approach inspired by quantum interference theory to capture the interactions within each utterance (i.e., the correlations between different modalities) and a strong-weak influence model inspired by quantum measurement theory to model the interactions between adjacent utterances (i.e., how one speaker influences another). Extensive experiments are conducted on two widely used conversational sentiment datasets: the MELD and IEMOCAP datasets. The experimental results show that our approach significantly outperforms a wide range of baselines and state-of-the-art models.