The influence of static eye and head position on the ventriloquist effect

• Published in: The European journal of neuroscience 2013-05, Vol.37 (9), p.1501-1510
• Main Authors: Van Barneveld, Denise C. P. B. M., Van Wanrooij, Marc M.
• Format: Article
• Language: English
• Subjects: Acoustic Stimulation; Adult; Biological and medical sciences; Eye and associated structures. Visual pathways and centers. Vision; Eye Movements - physiology; Female; Fundamental and applied biological sciences. Psychology; Head Movements - physiology; human; Humans; Male; Middle Aged; multisensory integration; Photic Stimulation; Psychomotor Performance; reference frame; Retina - physiology; sound localisation; Sound Localization - physiology; Space Perception - physiology; spatial behaviour; Vertebrates: nervous system and sense organs
• ISSN: 0953-816X; 1460-9568
• DOI: 10.1111/ejn.12176

Orienting responses to audiovisual events have shorter reaction times and better accuracy and precision when images and sounds in the environment are aligned in space and time. How the brain constructs an integrated audiovisual percept is a computational puzzle because the auditory and visual senses are represented in different reference frames: the retina encodes visual locations with respect to the eyes; whereas the sound localisation cues are referenced to the head. In the well‐known ventriloquist effect, the auditory spatial percept of the ventriloquist's voice is attracted toward the synchronous visual image of the dummy, but does this visual bias on sound localisation operate in a common reference frame by correctly taking into account eye and head position? Here we studied this question by independently varying initial eye and head orientations, and the amount of audiovisual spatial mismatch. Human subjects pointed head and/or gaze to auditory targets in elevation, and were instructed to ignore co‐occurring visual distracters. Results demonstrate that different initial head and eye orientations are accurately and appropriately incorporated into an audiovisual response. Effectively, sounds and images are perceptually fused according to their physical locations in space independent of an observer's point of view. Implications for neurophysiological findings and modelling efforts that aim to reconcile sensory and motor signals for goal‐directed behaviour are discussed. Does the ventriloquist effect operate in a common reference frame by correctly taking into account eye and head position? Here we demonstrate that different initial head and eye orientations are accurately and appropriately incorporated into an audiovisual response. Effectively, sounds and images are perceptually fused according to their physical locations in space independent of an observer's point of view.