Hear The Flow: Optical Flow-Based Self-Supervised Visual Sound Source Localization

Learning to localize the sound source in videos without explicit annotations is a novel area of audio-visual research. Existing work in this area focuses on creating attention maps to capture the correlation between the two modalities to localize the source of the sound. In a video, oftentimes, the...

Full description

Saved in:
Bibliographic Details
Main Authors: Fedorishin, Dennis, Dayal Mohan, Deen, Jawade, Bhavin, Setlur, Srirangaraj, Govindaraju, Venu
Format: Conference Proceeding
Language:English
Subjects:
Algorithms: Vision + language and/or other modalities
and algorithms (including transfer
and un-supervised learning
Computer vision
formulations
Image databases
Image motion analysis
Location awareness
low-shot
Machine learning architectures
Multimedia Web sites
self
Self-supervised learning
semi
Visualization
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Learning to localize the sound source in videos without explicit annotations is a novel area of audio-visual research. Existing work in this area focuses on creating attention maps to capture the correlation between the two modalities to localize the source of the sound. In a video, oftentimes, the objects exhibiting movement are the ones generating the sound. In this work, we capture this characteristic by modeling the optical flow in a video as a prior to better aid in localizing the sound source. We further demonstrate that the addition of flow-based attention substantially improves visual sound source localization. Finally, we benchmark our method on standard sound source localization datasets and achieve state-of-the-art performance on the Soundnet Flickr and VGG Sound Source datasets. Code: https://github.com/denfed/heartheflow.
ISSN:2642-9381
DOI:10.1109/WACV56688.2023.00231