Flowgrad: Using Motion for Visual Sound Source Localization

Most recent work in visual sound source localization relies on semantic audio-visual representations learned in a self-supervised manner and, by design, excludes temporal information present in videos. While it proves to be effective for widely used benchmark datasets, the method falls short for cha...

Full description

Saved in:
Bibliographic Details
Main Authors: Singh, Rajsuryan, Zinemanas, Pablo, Serra, Xavier, Bello, Juan Pablo, Fuentes, Magdalena
Format: Conference Proceeding
Language:English
Subjects:
audio-visual urban scene understanding
Benchmark testing
Codes
explainability
Location awareness
Semantics
Signal processing
Sound source localization
Speech coding
Visualization
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Most recent work in visual sound source localization relies on semantic audio-visual representations learned in a self-supervised manner and, by design, excludes temporal information present in videos. While it proves to be effective for widely used benchmark datasets, the method falls short for challenging scenarios like urban traffic. This work introduces temporal context into the state-of-the-art methods for sound source localization in urban scenes using optical flow to encode motion information. An analysis of the strengths and weaknesses of our methods helps us better understand the problem of visual sound source localization and sheds light on open challenges for audio-visual scene understanding. The code and pretrained models are publicly available at https://github.com/rrrajjjj/flowgrad
ISSN:2379-190X
DOI:10.1109/ICASSP49357.2023.10094965