Hybrid deep convolutional model-based emotion recognition using multiple physiological signals

• Published in: Computer methods in biomechanics and biomedical engineering 2022-11, Vol.25 (15), p.1678-1690
• Main Author: Patlar Akbulut, Fatma
• Format: Article
• Language: English
• Subjects: Accuracy; affective computing; Algorithms; Artificial neural networks; Autoregressive models; Autoregressive processes; Classification; CNN; Deep learning; Electronic devices; Emotion recognition; Emotions; Feature extraction; Long short-term memory; Machine learning; Markov chains; Mathematical models; Neural networks; Physiological responses; Physiology; Recurrent neural networks; Sensors; Transfer learning
• ISSN: 1025-5842; 1476-8259
• DOI: 10.1080/10255842.2022.2032682

Emotion recognition has become increasingly utilized in the medical, advertising, and military domains. Recognizing the cues of emotion from human behaviors or physiological responses is encouraging for the research community. However, extracting true characteristics from sensor data to understand emotions can be challenging due to the complex nature of these signals. Therefore, advanced feature engineering techniques are required for accurate signal recognition. This study presents a hybrid affective model that employs a transfer learning approach for emotion classification using large-frame sensor signals which employ a genuine dataset of signal fusion gathered from 30 participants using wearable sensor systems interconnected with mobile devices. The proposed approach implements several learning algorithms such as Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), Recurrent Neural Network (RNN), and several other shallow methods on the sensor input to handle the requirements for the traditional feature extraction process. The findings reveal that the use of deep learning methods is satisfactory in affect recognition when a great number of frames is employed, and the proposed hybrid deep model outperforms traditional neural network (overall accuracy of 54%) and deep learning approaches (overall accuracy of 76%), with an average classification accuracy of 93%. This hybrid deep model also has a higher accuracy than our previously proposed statistical autoregressive hidden Markov model (AR-HMM) approach, with 88.6% accuracy. Accuracy assessment was performed by means of several statistics measures (accuracy, precision, recall, F-measure, and RMSE).