Multimodal emotion detection from multiple data streams for improved decision making

Recent neurological studies shows that emotions are tightly connected to the thinking and cognitive actions, being part of the decision-making process. Considering this, having a way to help decision making processes based on current emotion of the user or to consider the potential emotional impact...

Full description

Saved in:
Bibliographic Details
Published in:Procedia computer science 2022, Vol.214, p.1082-1089
Main Authors: Mihai, Neghina, Alexandru, Matei, Bala-Constantin, Zamfirescu
Format: Article
Language:English
Subjects:
affect
emotion detection
multimodal
sensor fusion
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Recent neurological studies shows that emotions are tightly connected to the thinking and cognitive actions, being part of the decision-making process. Considering this, having a way to help decision making processes based on current emotion of the user or to consider the potential emotional impact if a decision is made, would be beneficial. This paper introduces a novel method for fusing multiple emotional signals, using a weighted average, where each weight value adapts to real time conditions, based on signal type, presence, and quality. In the context of a training station for manual operation, we implemented and tested separately several emotion detection methods, each based on a different signal acquired from audio, video, and galvanic skin response data streams. The final goal is to include the proposed method together with state of the art emotion detection machine learning algorithms as part of the digital twin training station for manual operation.
ISSN:1877-0509
1877-0509
DOI:10.1016/j.procs.2022.11.281