Accuracy Enhancement Method for Speech Emotion Recognition From Spectrogram Using Temporal Frequency Correlation and Positional Information Learning Through Knowledge Transfer

• Published in: IEEE access 2024, Vol.12, p.128039-128048
• Main Authors: Kim, Jeong-Yoon, Lee, Seung-Ho
• Format: Article
• Language: English
• Subjects: Accuracy; Coding; Convolution; Correlation; Datasets; Emotion recognition; Emotions; Encoding; Error analysis; Feature maps; Image coding; Image transmission; Knowledge management; Matching; Position (location); positional encoding; Spectrogram; Spectrograms; Speech emotion recognition (SER); Speech recognition; Teachers; temporal frequency correlation; transfer learning; Transformers; vision transformer (ViT)
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2024.3447770

In this paper, we propose a method to improve the accuracy of speech emotion recognition (SER) by using vision transformer (ViT) to attend to the correlation of frequency (y-axis) with time (x-axis) in spectrogram and transferring positional information between ViT through knowledge transfer. The proposed method has the following originality i) We use vertically segmented patches of log-Mel spectrogram to analyze the correlation of frequencies over time. This type of patch allows us to correlate the most relevant frequencies for a particular emotion with the time they were uttered. ii) We propose the use of image coordinate encoding, an absolute positional encoding suitable for ViT. By normalizing the x, y coordinates of the image to −1 to 1 and concatenating them to the image, we can effectively provide valid absolute positional information for ViT. iii) Through feature map matching, the locality and location information of the teacher network is effectively transmitted to the student network. Teacher network is a ViT that contains locality of convolutional stem and absolute position information through image coordinate encoding, and student network is a structure that lacks positional encoding in the basic ViT structure. In feature map matching stage, we train through the mean absolute error (L1 loss) to minimize the difference between the feature maps of the two networks. To validate the proposed method, three emotion datasets (SAVEE, EmoDB, and CREMA-D) consisting of speech were converted into log-Mel spectrograms for comparison experiments. As a result of the experiment, the proposed method achieved 99.47%, 99.76%, and 95.24% accuracy, significantly exceeding the state-of-the-art methods, with much less computational complexity on three emotion datasets.