Multimodal Emotion Recognition with Surgical and Fabric Masks

In this study, we investigate how different types of masks affect automatic emotion classification in different channels of audio, visual, and multimodal. We train emotion classification models for each modality with the original data without mask and the re-generated data with mask respectively, an...

Full description

Saved in:
Bibliographic Details
Main Authors: Yang, Ziqing, Nayan, Katherine, Fan, Zehao, Cao, Houwei
Format: Conference Proceeding
Language:English
Subjects:
cross-corpus evaluation
Emotion recognition
Face recognition
mask
muffled speech
multimodal emotion classification
occluded facial expressions
Predictive models
Signal processing
Speech recognition
Surgery
Visualization
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this study, we investigate how different types of masks affect automatic emotion classification in different channels of audio, visual, and multimodal. We train emotion classification models for each modality with the original data without mask and the re-generated data with mask respectively, and investigate how muffled speech and occluded facial expressions change the prediction of emotions. Moreover, we conduct the contribution analysis to study how muffled speech and occluded face interplay with each other and further investigate the individual contribution of audio, visual, and audio-visual modalities to the prediction of emotion with and without mask. Finally, we investigate the cross-corpus emotion recognition across clear speech and re-generated speech with different types of masks, and discuss the robustness of speech emotion recognition.
ISSN:2379-190X
DOI:10.1109/ICASSP43922.2022.9746414