The Role of Sound Source Perception in Gestural Sound Description

• Published in: ACM transactions on applied perception 2014-04, Vol.11 (1), p.1-19
• Main Authors: Caramiaux, B., Bevilacqua, F., Bianco, T., Schnell, N., Houix, O., Susini, P.
• Format: Article
• Language: English
• Subjects: Artificial Intelligence; Computer Aided Engineering; Computer Science; Computer Vision and Pattern Recognition; Engineering Sciences; Human-Computer Interaction; Humanities and Social Sciences; Machine Learning; Multimedia; Musicology and performing arts; Other; Signal and Image processing; Sound
• ISSN: 1544-3558; 1544-3965
• DOI: 10.1145/2536811

We investigated gesture description of sound stimuli performed during a listening task. Our hypothesis is that the strategies in gestural responses depend on the level of identification of the sound source and specifically on the identification of the action causing the sound. To validate our hypothesis, we conducted two experiments. In the first experiment, we built two corpora of sounds. The first corpus contains sounds with identifiable causal actions. The second contains sounds for which no causal actions could be identified. These corpora properties were validated through a listening test. In the second experiment, participants performed arm and hand gestures synchronously while listening to sounds taken from these corpora. Afterward, we conducted interviews asking participants to verbalize their experience while watching their own video recordings. They were questioned on their perception of the listened sounds and on their gestural strategies. We showed that for the sounds where causal action can be identified, participants mainly mimic the action that has produced the sound. In the other case, when no action can be associated with the sound, participants trace contours related to sound acoustic features. We also found that the interparticipants’ gesture variability is higher for causal sounds compared to noncausal sounds. Variability demonstrates that, in the first case, participants have several ways of producing the same action, whereas in the second case, the sound features tend to make the gesture responses consistent.