Multimodal Coordination of Sound and Movement in Music and Speech

Speech and music emerge from a spectrum of nested motor and perceptual coordination patterns across timescales of brief movements to actions. Intuitively, this nested clustering in movements should be reflected in sound. We examined similarities and differences in multimodal, multiscale coordination...

Full description

Saved in:
Bibliographic Details
Published in:Discourse processes 2020-09, Vol.57 (8), p.682-702
Main Authors: Alviar, Camila, Dale, Rick, Dewitt, Akeiylah, Kello, Christopher
Format: Article
Language:English
Subjects:
Acoustics
Amplitude (Acoustics)
Correlation
Interpersonal Communication
Motion
Music
Musical Instruments
Singing
Speech Communication
Speech perception
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Speech and music emerge from a spectrum of nested motor and perceptual coordination patterns across timescales of brief movements to actions. Intuitively, this nested clustering in movements should be reflected in sound. We examined similarities and differences in multimodal, multiscale coordination of speech and music using two complementary measures: We computed spectra for envelopes of acoustic amplitudes and motion amplitudes and correlated spectral powers across modalities as a function of frequency. We also correlated smoothed envelopes and examined peaks in their cross-correlation functions. YouTube videos of five different modes of speaking and five different types of music were analyzed. Speech performances yielded stronger, more reliable relationships between sound and movement compared with music. Interestingly, a cappella singing patterned more with music, and improvisational jazz piano patterned more with speech. Results suggest that nested temporal structures in sound and movement are coordinated as a function of communicative aspects of performance.
ISSN:0163-853X
1532-6950
DOI:10.1080/0163853X.2020.1768500