A Multimodal Classification of Noisy Hate Speech using Character Level Embedding and Attention

• Main Authors: Prasad, Nishchal, Saha, Sriparna, Bhattacharyya, Pushpak
• Format: Conference Proceeding
• Language: English
• Subjects: Bit error rate; Blogs; Neural networks; Social networking (online); Training; Transformers; Visualization
• ISSN: 2161-4407
• DOI: 10.1109/IJCNN52387.2021.9533371

Hate speech has become a critical problem in all social media, leading to many hate crimes alongside affecting the mental and emotional well-being of affected individuals. This calls out for methods to detect online hate speech more than ever. While numerous architectures exist for hate speech detection in unimodal setup (i.e., either textual or visual) we have targeted the problem in the context of both text and images inspired by the real-world raw data which involves several modalities. We propose a multimodal hate speech classifier, called as Character Text Image Classifier (CTIC), which builds upon Bidirectional Encoder Representations from Transformers (BERT), Capsule Network, and EfficientNet involving four modalities, namely word embeddings, character embeddings, sentence embeddings, and images. We report the experiments performed upon our proposed model and several other models which have been tested in the process. We train our model with different sampling techniques, and selective training, upon a Twitter dataset, called MMHS150K, consisting of both texts and associated images. Our proposed multimodal approach attains better performance than the previous models constructed upon the dataset.