Look at me when I'm talking to you: Selective attention at a multisensory cocktail party can be decoded using stimulus reconstruction and alpha power modulations

• Published in: The European journal of neuroscience 2019-10, Vol.50 (8), p.3282-3295
• Main Authors: O'Sullivan, Aisling E., Lim, Chantelle Y., Lalor, Edmund C.
• Format: Article
• Language: English
• Subjects: Attention; audiovisual speech; Cortex; EEG; electroencephalography; envelope reconstruction; Scalp; Sensory integration; Speech; visual attention; Visual perception; visual speech
• ISSN: 0953-816X; 1460-9568
• DOI: 10.1111/ejn.14425

Recent work using electroencephalography has applied stimulus reconstruction techniques to identify the attended speaker in a cocktail party environment. The success of these approaches has been primarily based on the ability to detect cortical tracking of the acoustic envelope at the scalp level. However, most studies have ignored the effects of visual input, which is almost always present in naturalistic scenarios. In this study, we investigated the effects of visual input on envelope‐based cocktail party decoding in two multisensory cocktail party situations: (a) Congruent AV—facing the attended speaker while ignoring another speaker represented by the audio‐only stream and (b) Incongruent AV (eavesdropping)—attending the audio‐only speaker while looking at the unattended speaker. We trained and tested decoders for each condition separately and found that we can successfully decode attention to congruent audiovisual speech and can also decode attention when listeners were eavesdropping, i.e., looking at the face of the unattended talker. In addition to this, we found alpha power to be a reliable measure of attention to the visual speech. Using parieto‐occipital alpha power, we found that we can distinguish whether subjects are attending or ignoring the speaker's face. Considering the practical applications of these methods, we demonstrate that with only six near‐ear electrodes we can successfully determine the attended speech. This work extends the current framework for decoding attention to speech to more naturalistic scenarios, and in doing so provides additional neural measures which may be incorporated to improve decoding accuracy. Decoding selective attention at a multisensory cocktail party is optimal when separate decoders are used for attending audiovisual speech and for eavesdropping. Alpha power is a reliable measure at determining attention to the speaker's face, and thus informs which decoder to use. These findings shed light on attentional decoding in more naturalistic environments and provides additional neural measures for decoding selective attention to natural speech.