An emoji-aware multitask framework for multimodal sarcasm detection

• Published in: Knowledge-based systems 2022-12, Vol.257, p.109924, Article 109924
• Main Authors: Chauhan, Dushyant Singh, Singh, Gopendra Vikram, Arora, Aseem, Ekbal, Asif, Bhattacharyya, Pushpak
• Format: Article
• Language: English
• Subjects: Deep learning; Emoji; Multimodal emotion; Multimodal sarcasm; Multimodal sentiment; MUStARD dataset
• ISSN: 0950-7051; 1872-7409
• DOI: 10.1016/j.knosys.2022.109924

Sarcasm is a case of implicit emotion and needs additional information like context and multimodality for better detection. But sometimes, this additional information also fails to help in sarcasm detection. For example, the utterance “Oh yes, you’ve been so helpful. Thank you so much for all your help”, said in a polite tone with a smiling face, can be understood easily as non-sarcastic because of its positive sentiment. But, if the above message is accompanied by a frustrated emoji ▪, the negative sentiment of the emoji becomes evident, and the intended sarcasm can be easily understood. Thus, in this paper, we propose the SEEmoji MUStARD, an extension of the multimodal MUStARD dataset. We annotate each utterance with relevant emoji, emoji’s sentiment, and emoji’s emotion. We propose an emoji-aware-multimodal multitask deep learning framework for sarcasm detection (i.e., primary task) and sentiment and emotion detection (i.e., secondary task) in a multimodal conversational scenario. Experimental results on the SEEmoji MUStARD show the efficacy of our proposed emoji-aware-multimodal approach for sarcasm detection over the existing models.