Problem detection in human–machine interactions based on facial expressions of users

• Published in: Speech communication 2005-03, Vol.45 (3), p.343-359
• Main Authors: Barkhuysen, Pashiera, Krahmer, Emiel, Swerts, Marc
• Format: Article
• Language: English
• Subjects: Applied sciences; Exact sciences and technology; Information, signal and communications theory; Signal and communications theory; Signal processing; Signal representation. Spectral analysis; Signal, noise; Speech processing; Telecommunications and information theory
• ISSN: 0167-6393; 1872-7182
• DOI: 10.1016/j.specom.2004.10.004

This paper describes research into audiovisual cues to communication problems in interactions between users and a spoken dialogue system. The study consists of two parts. First, we describe a series of three perception experiments in which subjects are offered film fragments (without any dialogue context) of speakers interacting with a spoken dialogue system. In half of these fragments, the speaker is or becomes aware of a communication problem. Subjects have to determine by forced choice which are the problematic fragments. In all three tests, subjects are capable of performing this task to some extent, but with varying levels of correct classifications. Second, we report results of an observational analysis in which we first attempt to relate the perceptual results to visual features of the stimuli presented to subjects, and second to find out which visual features actually are potential cues for error detection. Our major finding is that more problematic contexts lead to more dynamic facial expressions, in line with earlier claims that communication errors lead to marked speaker behaviour. We conclude that visual information from a user’s face is potentially beneficial for problem detection.