CNN-XGBoost fusion-based affective state recognition using EEG spectrogram image analysis

• Published in: Scientific reports 2022-08, Vol.12 (1), p.14122-14122, Article 14122
• Main Authors: Khan, Md. Sakib, Salsabil, Nishat, Alam, Md. Golam Rabiul, Dewan, M. Ali Akber, Uddin, Md. Zia
• Format: Article
• Language: English
• Subjects: 639/166/985; 639/705; Arousal; Dominance; EEG; Emotional behavior; Fourier analysis; Fourier transforms; Humanities and Social Sciences; Image processing; multidisciplinary; Neural networks; Science; Science (multidisciplinary); Support vector machines
• ISSN: 2045-2322; 2045-2322
• DOI: 10.1038/s41598-022-18257-x

Recognizing emotional state of human using brain signal is an active research domain with several open challenges. In this research, we propose a signal spectrogram image based CNN-XGBoost fusion method for recognising three dimensions of emotion, namely arousal (calm or excitement), valence (positive or negative feeling) and dominance (without control or empowered). We used a benchmark dataset called DREAMER where the EEG signals were collected from multiple stimulus along with self-evaluation ratings. In our proposed method, we first calculate the Short-Time Fourier Transform (STFT) of the EEG signals and convert them into RGB images to obtain the spectrograms. Then we use a two dimensional Convolutional Neural Network (CNN) in order to train the model on the spectrogram images and retrieve the features from the trained layer of the CNN using a dense layer of the neural network. We apply Extreme Gradient Boosting (XGBoost) classifier on extracted CNN features to classify the signals into arousal, valence and dominance of human emotion. We compare our results with the feature fusion-based state-of-the-art approaches of emotion recognition. To do this, we applied various feature extraction techniques on the signals which include Fast Fourier Transformation, Discrete Cosine Transformation, Poincare, Power Spectral Density, Hjorth parameters and some statistical features. Additionally, we use Chi-square and Recursive Feature Elimination techniques to select the discriminative features. We form the feature vectors by applying feature level fusion, and apply Support Vector Machine (SVM) and Extreme Gradient Boosting (XGBoost) classifiers on the fused features to classify different emotion levels. The performance study shows that the proposed spectrogram image based CNN-XGBoost fusion method outperforms the feature fusion-based SVM and XGBoost methods. The proposed method obtained the accuracy of 99.712% for arousal, 99.770% for valence and 99.770% for dominance in human emotion detection.