Toward an affect-sensitive multimodal human-computer interaction

• Published in: Proceedings of the IEEE 2003-09, Vol.91 (9), p.1370-1390
• Main Authors: Pantic, M., Rothkrantz, L.J.M.
• Format: Article
• Language: English
• Subjects: Animation; Anthropometry; Cognitive science; Design engineering; Emotion recognition; Emotional intelligence; Emotions; Face detection; Facial; Feedback; Human; Human computer interaction; Intelligence; Interactive; Knowledge based systems; Movements; Neuroscience; Psychology; Recognition; Social interaction; Tasks
• ISSN: 0018-9219; 1558-2256
• DOI: 10.1109/JPROC.2003.817122

The ability to recognize affective states of a person we are communicating with is the core of emotional intelligence. Emotional intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for successful interpersonal social interaction. This paper argues that next-generation human-computer interaction (HCI) designs need to include the essence of emotional intelligence - the ability to recognize a user's affective states-in order to become more human-like, more effective, and more efficient. Affective arousal modulates all nonverbal communicative cues (facial expressions, body movements, and vocal and physiological reactions). In a face-to-face interaction, humans detect and interpret those interactive signals of their communicator with little or no effort. Yet design and development of an automated system that accomplishes these tasks is rather difficult. This paper surveys the past work in solving these problems by a computer and provides a set of recommendations for developing the first part of an intelligent multimodal HCI-an automatic personalized analyzer of a user's nonverbal affective feedback.