Gesture in the eye of the beholder: An eye-tracking study on factors determining the attention for gestures produced by people with aphasia

• Published in: Neuropsychologia 2022-09, Vol.174, p.108315-108315, Article 108315
• Main Authors: van Nispen, Karin, Sekine, Kazuki, van der Meulen, Ineke, Preisig, Basil C.
• Format: Article
• Language: English
• Subjects: Aphasia; Attention; Eye movements; Gestures; Multimodal communication; Speech perception
• ISSN: 0028-3932; 1873-3514
• DOI: 10.1016/j.neuropsychologia.2022.108315

Co-speech hand gestures are an ubiquitous form of nonverbal communication, which can express additional information that is not present in speech. Hand gestures may become more relevant when verbal production is impaired, as in speakers with post-stroke aphasia. In fact, speakers with aphasia produce more gestures than non-brain damaged speakers. Furthermore, there is evidence to suggest that speakers with aphasia produce gestures that convey information essential to understand their communication. In the present study, we addressed the question whether these gestures catch the attention of their addressees. Healthy volunteers (observers) watched short video clips while their eye movements were recorded. These video clips featured speakers with aphasia and non-brain damaged speakers describing two different scenarios (buying a sweater or having witnessed an accident). Our results show that hand gestures produced by speakers with aphasia are on average attended to longer than gestures produced by non-brain damaged speakers. This effect was significant even when we controlled for the longer duration of the gestural movements in speakers with aphasia. Further, the amount of information in speech was also correlated with gesture attention. That is gestures produced by speakers with less informative speech were attended to more frequently. In conclusion, our findings suggest that listeners reallocate their attention and focus more strongly on non-verbal information from co-speech gestures if speech comprehension becomes challenging due to the speaker's verbal production deficits. These findings support a communicative function of co-speech gestures and advocate for instructing people with aphasia to communicate things in the form of gestures that cannot be expressed verbally because interlocutors take notice of these gestures. •Healthy observers watched video clips while their eye movements were recorded.•These video clips featured speakers with aphasia and non-brain damaged speakers.•Speakers in the video were using gestures to describe a vivid scenario.•Hand gestures produced by speakers with aphasia were longer attended to than gestures produced by non-brain damaged speakers.•Gestures produced by speakers with less informative speech were attended to more frequently.