In your face? Exploring multimodal response patterns involving facial responses to verbal and gestural stance-taking expressions

• Published in: Journal of pragmatics 2022-03, Vol.190, p.6-17
• Main Authors: Feyaerts, Kurt, Rominger, Christian, Lackner, Helmut Karl, Brône, Geert, Jehoul, Annelies, Oben, Bert, Papousek, Ilona
• Format: Article
• Language: English
• Subjects: Conversation; Dyadic interaction; Facial expression; Facial expressions; Gesture analysis; Interaction analysis; Linguistics; Multimodal co-occurrence patterns; Multimodality; Psychology; Spontaneous speech; Stance-taking
• ISSN: 0378-2166; 1879-1387
• DOI: 10.1016/j.pragma.2022.01.002

In the present study, informed by insights from cognitive and interactional linguistics, we set out to explore how facial expressions systematically occur as responses in interactional sequences. More specifically, we use FACS-analyses (Facial Action Coding System) to study which Action Units (AU) on the part of the listener co-occur with multimodal stance-taking acts by speakers. Based on a data set of 24 dyadic interactions, we show that different types of stance acts (e.g. marking obviousness vs. using expressive amplifiers) reveal different patterns of facial responses. In addition, also within one type of stance act, there is systematic variation in facial responses. For example, listeners displayed significantly different AU-patterns in reactions to verbal obviousness markers, compared to non-verbal obviousness markers. Together, these observations highlight that, analogous to verbal responses in interactional sequences, also facial motor responses appear to be systematic, and highly dependent on conversational context. As such, the AU's under scrutiny serve as intersubjectively aligned response turns completing a situationally designed stance-taking act. With this interdisciplinary study, combining linguistics with psychology and physiology, we aim for a better understanding of the multimodal complexity that constitutes the process of meaning making in spontaneous conversation. •Including facial expression in the analysis of interactional meaning making.•Facial expressions serve as aligned response turns in dyadic sequences.•Four facial Action Units serve as response turn to various stance-taking acts.•Facial Action Units identified in multimodal co-occurrence pattern involving stance.•Statistical evidence: Facial Action Units are not automated, incidental reactions.