Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications

• Published in: International journal of human-computer studies 2014-10, Vol.72 (10-11), p.717-727
• Main Authors: Kukolja, Davor, Popović, Siniša, Horvat, Marko, Kovač, Bernard, Ćosić, Krešimir
• Format: Article
• Language: English
• Subjects: Affective computing; Affectivity. Emotion; Applied sciences; Artificial intelligence; Biological and medical sciences; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Connectionism. Neural networks; Data processing. List processing. Character string processing; Emotion estimation; Exact sciences and technology; Feature reduction; Fundamental and applied biological sciences. Psychology; Machine learning; Memory organisation. Data processing; Personality. Affectivity; Physiology; Psychology. Psychoanalysis. Psychiatry; Psychology. Psychophysiology; Software
• ISSN: 1071-5819; 1095-9300
• DOI: 10.1016/j.ijhcs.2014.05.006

In order to improve intelligent Human-Computer Interaction it is important to create a personalized adaptive emotion estimator that is able to learn over time emotional response idiosyncrasies of individual person and thus enhance estimation accuracy. This paper, with the aim of identifying preferable methods for such a concept, presents an experiment-based comparative study of seven feature reduction and seven machine learning methods commonly used for emotion estimation based on physiological signals. The analysis was performed on data obtained in an emotion elicitation experiment involving 14 participants. Specific discrete emotions were targeted with stimuli from the International Affective Picture System database. The experiment was necessary to achieve the uniformity in the various aspects of emotion elicitation, data processing, feature calculation, self-reporting procedures and estimation evaluation, in order to avoid inconsistency problems that arise when results from studies that use different emotion-related databases are mutually compared. The results of the performed experiment indicate that the combination of a multilayer perceptron (MLP) with sequential floating forward selection (SFFS) exhibited the highest accuracy in discrete emotion classification based on physiological features calculated from ECG, respiration, skin conductance and skin temperature. Using leave-one-session-out crossvalidation method, 60.3% accuracy in classification of 5 discrete emotions (sadness, disgust, fear, happiness and neutral) was obtained. In order to identify which methods may be the most suitable for real-time estimator adaptation, execution and learning times of emotion estimators were also comparatively analyzed. Based on this analysis, preferred feature reduction method for real-time estimator adaptation was minimum redundancy – maximum relevance (mRMR), which was the fastest approach in terms of combined execution and learning time, as well as the second best in accuracy, after SFFS. In combination with mRMR, highest accuracies were achieved by k-nearest neighbor (kNN) and MLP with negligible difference (50.33% versus 50.54%); however, mRMR+kNN is preferable option for real-time estimator adaptation due to considerably lower combined execution and learning time of kNN versus MLP. •We compared accuracy and learning/execution times of emotion estimation methods.•Feature selection methods had more impact on accuracy than machine learning methods.•Combinatio