SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound

Generating combined visual and auditory sensory experiences is critical for the consumption of immersive content. Recent advances in neural generative models have enabled the creation of high-resolution content across multiple modalities such as images, text, speech, and videos. Despite these succes...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-06
Main Authors: Dagli, Rishit, Prakash, Shivesh, Wu, Robert, Khosravani, Houman
Format: Article
Language:English
Subjects:
Auditory perception
Image quality
Speech
Video
Visual tasks
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by
cites
container_end_page
container_issue
container_start_page
container_title arXiv.org
container_volume
creator Dagli, Rishit
Prakash, Shivesh
Wu, Robert
Khosravani, Houman
description Generating combined visual and auditory sensory experiences is critical for the consumption of immersive content. Recent advances in neural generative models have enabled the creation of high-resolution content across multiple modalities such as images, text, speech, and videos. Despite these successes, there remains a significant gap in the generation of high-quality spatial audio that complements generated visual content. Furthermore, current audio generation models excel in either generating natural audio or speech or music but fall short in integrating spatial audio cues necessary for immersive experiences. In this work, we introduce SEE-2-SOUND, a zero-shot approach that decomposes the task into (1) identifying visual regions of interest; (2) locating these elements in 3D space; (3) generating mono-audio for each; and (4) integrating them into spatial audio. Using our framework, we demonstrate compelling results for generating spatial audio for high-quality videos, images, and dynamic images from the internet, as well as media generated by learned approaches.
format article
fullrecord <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_3067014115</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>3067014115</sourcerecordid><originalsourceid>FETCH-proquest_journals_30670141153</originalsourceid><addsrcrecordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mSwDnZ11TXSDfYP9XOxUohKLcrXDc7IL1EILkgsyUzMUXDNK8ssys_LTc0r0S0BykGFg_NL81J4GFjTEnOKU3mhNDeDsptriLOHbkFRfmFpanFJfFZ-aVEeUCre2MDM3MDQxNDQ1Jg4VQAXZzWG</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3067014115</pqid></control><display><type>article</type><title>SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound</title><source>Publicly Available Content Database (Proquest) (PQ_SDU_P3)</source><creator>Dagli, Rishit ; Prakash, Shivesh ; Wu, Robert ; Khosravani, Houman</creator><creatorcontrib>Dagli, Rishit ; Prakash, Shivesh ; Wu, Robert ; Khosravani, Houman</creatorcontrib><description>Generating combined visual and auditory sensory experiences is critical for the consumption of immersive content. Recent advances in neural generative models have enabled the creation of high-resolution content across multiple modalities such as images, text, speech, and videos. Despite these successes, there remains a significant gap in the generation of high-quality spatial audio that complements generated visual content. Furthermore, current audio generation models excel in either generating natural audio or speech or music but fall short in integrating spatial audio cues necessary for immersive experiences. In this work, we introduce SEE-2-SOUND, a zero-shot approach that decomposes the task into (1) identifying visual regions of interest; (2) locating these elements in 3D space; (3) generating mono-audio for each; and (4) integrating them into spatial audio. Using our framework, we demonstrate compelling results for generating spatial audio for high-quality videos, images, and dynamic images from the internet, as well as media generated by learned approaches.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Auditory perception ; Image quality ; Speech ; Video ; Visual tasks</subject><ispartof>arXiv.org, 2024-06</ispartof><rights>2024. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://www.proquest.com/docview/3067014115?pq-origsite=primo$$EHTML$$P50$$Gproquest$$Hfree_for_read</linktohtml><link.rule.ids>780,784,25752,37011,44589</link.rule.ids></links><search><creatorcontrib>Dagli, Rishit</creatorcontrib><creatorcontrib>Prakash, Shivesh</creatorcontrib><creatorcontrib>Wu, Robert</creatorcontrib><creatorcontrib>Khosravani, Houman</creatorcontrib><title>SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound</title><title>arXiv.org</title><description>Generating combined visual and auditory sensory experiences is critical for the consumption of immersive content. Recent advances in neural generative models have enabled the creation of high-resolution content across multiple modalities such as images, text, speech, and videos. Despite these successes, there remains a significant gap in the generation of high-quality spatial audio that complements generated visual content. Furthermore, current audio generation models excel in either generating natural audio or speech or music but fall short in integrating spatial audio cues necessary for immersive experiences. In this work, we introduce SEE-2-SOUND, a zero-shot approach that decomposes the task into (1) identifying visual regions of interest; (2) locating these elements in 3D space; (3) generating mono-audio for each; and (4) integrating them into spatial audio. Using our framework, we demonstrate compelling results for generating spatial audio for high-quality videos, images, and dynamic images from the internet, as well as media generated by learned approaches.</description><subject>Auditory perception</subject><subject>Image quality</subject><subject>Speech</subject><subject>Video</subject><subject>Visual tasks</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>PIMPY</sourceid><recordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mSwDnZ11TXSDfYP9XOxUohKLcrXDc7IL1EILkgsyUzMUXDNK8ssys_LTc0r0S0BykGFg_NL81J4GFjTEnOKU3mhNDeDsptriLOHbkFRfmFpanFJfFZ-aVEeUCre2MDM3MDQxNDQ1Jg4VQAXZzWG</recordid><startdate>20240606</startdate><enddate>20240606</enddate><creator>Dagli, Rishit</creator><creator>Prakash, Shivesh</creator><creator>Wu, Robert</creator><creator>Khosravani, Houman</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20240606</creationdate><title>SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound</title><author>Dagli, Rishit ; Prakash, Shivesh ; Wu, Robert ; Khosravani, Houman</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_30670141153</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Auditory perception</topic><topic>Image quality</topic><topic>Speech</topic><topic>Video</topic><topic>Visual tasks</topic><toplevel>online_resources</toplevel><creatorcontrib>Dagli, Rishit</creatorcontrib><creatorcontrib>Prakash, Shivesh</creatorcontrib><creatorcontrib>Wu, Robert</creatorcontrib><creatorcontrib>Khosravani, Houman</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science &amp; Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection (Proquest) (PQ_SDU_P3)</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database (Proquest) (PQ_SDU_P3)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Dagli, Rishit</au><au>Prakash, Shivesh</au><au>Wu, Robert</au><au>Khosravani, Houman</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound</atitle><jtitle>arXiv.org</jtitle><date>2024-06-06</date><risdate>2024</risdate><eissn>2331-8422</eissn><abstract>Generating combined visual and auditory sensory experiences is critical for the consumption of immersive content. Recent advances in neural generative models have enabled the creation of high-resolution content across multiple modalities such as images, text, speech, and videos. Despite these successes, there remains a significant gap in the generation of high-quality spatial audio that complements generated visual content. Furthermore, current audio generation models excel in either generating natural audio or speech or music but fall short in integrating spatial audio cues necessary for immersive experiences. In this work, we introduce SEE-2-SOUND, a zero-shot approach that decomposes the task into (1) identifying visual regions of interest; (2) locating these elements in 3D space; (3) generating mono-audio for each; and (4) integrating them into spatial audio. Using our framework, we demonstrate compelling results for generating spatial audio for high-quality videos, images, and dynamic images from the internet, as well as media generated by learned approaches.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier EISSN: 2331-8422
ispartof arXiv.org, 2024-06
issn 2331-8422
language eng
recordid cdi_proquest_journals_3067014115
source Publicly Available Content Database (Proquest) (PQ_SDU_P3)
subjects Auditory perception
Image quality
Speech
Video
Visual tasks
title SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-08T07%3A57%3A51IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=SEE-2-SOUND:%20Zero-Shot%20Spatial%20Environment-to-Spatial%20Sound&rft.jtitle=arXiv.org&rft.au=Dagli,%20Rishit&rft.date=2024-06-06&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E3067014115%3C/proquest%3E%3Cgrp_id%3Ecdi_FETCH-proquest_journals_30670141153%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=3067014115&rft_id=info:pmid/&rfr_iscdi=true