LDM3D: Latent Diffusion Model for 3D

This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that generates both image and depth map data from a given text prompt, allowing users to generate RGBD images from text prompts. The LDM3D model is fine-tuned on a dataset of tuples containing an RGB image, depth map and caption, a...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-05
Main Authors: Gabriela Ben Melech Stan, Wofk, Diana, Fox, Scottie, Redden, Alex, Saxton, Will, Yu, Jean, Aflalo, Estelle, Shao-Yen Tseng, Nonato, Fabio, Muller, Matthias, Lal, Vasudev
Format: Article
Language:English
Subjects:
Color imagery
Computer vision
Three dimensional models
Video data
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by
cites
container_end_page
container_issue
container_start_page
container_title arXiv.org
container_volume
creator Gabriela Ben Melech Stan
Wofk, Diana
Fox, Scottie
Redden, Alex
Saxton, Will
Yu, Jean
Aflalo, Estelle
Shao-Yen Tseng
Nonato, Fabio
Muller, Matthias
Lal, Vasudev
description This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that generates both image and depth map data from a given text prompt, allowing users to generate RGBD images from text prompts. The LDM3D model is fine-tuned on a dataset of tuples containing an RGB image, depth map and caption, and validated through extensive experiments. We also develop an application called DepthFusion, which uses the generated RGB images and depth maps to create immersive and interactive 360-degree-view experiences using TouchDesigner. This technology has the potential to transform a wide range of industries, from entertainment and gaming to architecture and design. Overall, this paper presents a significant contribution to the field of generative AI and computer vision, and showcases the potential of LDM3D and DepthFusion to revolutionize content creation and digital experiences. A short video summarizing the approach can be found at https://t.ly/tdi2.
format article
fullrecord <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_2815836634</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2815836634</sourcerecordid><originalsourceid>FETCH-proquest_journals_28158366343</originalsourceid><addsrcrecordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mRQ8XHxNXaxUvBJLEnNK1FwyUxLKy3OzM9T8M1PSc1RSMsvUjB24WFgTUvMKU7lhdLcDMpuriHOHroFRfmFpanFJfFZ-aVFeUCpeCMLQ1MLYzMzYxNj4lQBAA2mLPE</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2815836634</pqid></control><display><type>article</type><title>LDM3D: Latent Diffusion Model for 3D</title><source>Publicly Available Content (ProQuest)</source><creator>Gabriela Ben Melech Stan ; Wofk, Diana ; Fox, Scottie ; Redden, Alex ; Saxton, Will ; Yu, Jean ; Aflalo, Estelle ; Shao-Yen Tseng ; Nonato, Fabio ; Muller, Matthias ; Lal, Vasudev</creator><creatorcontrib>Gabriela Ben Melech Stan ; Wofk, Diana ; Fox, Scottie ; Redden, Alex ; Saxton, Will ; Yu, Jean ; Aflalo, Estelle ; Shao-Yen Tseng ; Nonato, Fabio ; Muller, Matthias ; Lal, Vasudev</creatorcontrib><description>This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that generates both image and depth map data from a given text prompt, allowing users to generate RGBD images from text prompts. The LDM3D model is fine-tuned on a dataset of tuples containing an RGB image, depth map and caption, and validated through extensive experiments. We also develop an application called DepthFusion, which uses the generated RGB images and depth maps to create immersive and interactive 360-degree-view experiences using TouchDesigner. This technology has the potential to transform a wide range of industries, from entertainment and gaming to architecture and design. Overall, this paper presents a significant contribution to the field of generative AI and computer vision, and showcases the potential of LDM3D and DepthFusion to revolutionize content creation and digital experiences. A short video summarizing the approach can be found at https://t.ly/tdi2.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Color imagery ; Computer vision ; Three dimensional models ; Video data</subject><ispartof>arXiv.org, 2023-05</ispartof><rights>2023. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://www.proquest.com/docview/2815836634?pq-origsite=primo$$EHTML$$P50$$Gproquest$$Hfree_for_read</linktohtml><link.rule.ids>780,784,25753,37012,44590</link.rule.ids></links><search><creatorcontrib>Gabriela Ben Melech Stan</creatorcontrib><creatorcontrib>Wofk, Diana</creatorcontrib><creatorcontrib>Fox, Scottie</creatorcontrib><creatorcontrib>Redden, Alex</creatorcontrib><creatorcontrib>Saxton, Will</creatorcontrib><creatorcontrib>Yu, Jean</creatorcontrib><creatorcontrib>Aflalo, Estelle</creatorcontrib><creatorcontrib>Shao-Yen Tseng</creatorcontrib><creatorcontrib>Nonato, Fabio</creatorcontrib><creatorcontrib>Muller, Matthias</creatorcontrib><creatorcontrib>Lal, Vasudev</creatorcontrib><title>LDM3D: Latent Diffusion Model for 3D</title><title>arXiv.org</title><description>This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that generates both image and depth map data from a given text prompt, allowing users to generate RGBD images from text prompts. The LDM3D model is fine-tuned on a dataset of tuples containing an RGB image, depth map and caption, and validated through extensive experiments. We also develop an application called DepthFusion, which uses the generated RGB images and depth maps to create immersive and interactive 360-degree-view experiences using TouchDesigner. This technology has the potential to transform a wide range of industries, from entertainment and gaming to architecture and design. Overall, this paper presents a significant contribution to the field of generative AI and computer vision, and showcases the potential of LDM3D and DepthFusion to revolutionize content creation and digital experiences. A short video summarizing the approach can be found at https://t.ly/tdi2.</description><subject>Color imagery</subject><subject>Computer vision</subject><subject>Three dimensional models</subject><subject>Video data</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>PIMPY</sourceid><recordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mRQ8XHxNXaxUvBJLEnNK1FwyUxLKy3OzM9T8M1PSc1RSMsvUjB24WFgTUvMKU7lhdLcDMpuriHOHroFRfmFpanFJfFZ-aVFeUCpeCMLQ1MLYzMzYxNj4lQBAA2mLPE</recordid><startdate>20230521</startdate><enddate>20230521</enddate><creator>Gabriela Ben Melech Stan</creator><creator>Wofk, Diana</creator><creator>Fox, Scottie</creator><creator>Redden, Alex</creator><creator>Saxton, Will</creator><creator>Yu, Jean</creator><creator>Aflalo, Estelle</creator><creator>Shao-Yen Tseng</creator><creator>Nonato, Fabio</creator><creator>Muller, Matthias</creator><creator>Lal, Vasudev</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20230521</creationdate><title>LDM3D: Latent Diffusion Model for 3D</title><author>Gabriela Ben Melech Stan ; Wofk, Diana ; Fox, Scottie ; Redden, Alex ; Saxton, Will ; Yu, Jean ; Aflalo, Estelle ; Shao-Yen Tseng ; Nonato, Fabio ; Muller, Matthias ; Lal, Vasudev</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_28158366343</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Color imagery</topic><topic>Computer vision</topic><topic>Three dimensional models</topic><topic>Video data</topic><toplevel>online_resources</toplevel><creatorcontrib>Gabriela Ben Melech Stan</creatorcontrib><creatorcontrib>Wofk, Diana</creatorcontrib><creatorcontrib>Fox, Scottie</creatorcontrib><creatorcontrib>Redden, Alex</creatorcontrib><creatorcontrib>Saxton, Will</creatorcontrib><creatorcontrib>Yu, Jean</creatorcontrib><creatorcontrib>Aflalo, Estelle</creatorcontrib><creatorcontrib>Shao-Yen Tseng</creatorcontrib><creatorcontrib>Nonato, Fabio</creatorcontrib><creatorcontrib>Muller, Matthias</creatorcontrib><creatorcontrib>Lal, Vasudev</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science &amp; Engineering Collection</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content (ProQuest)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Gabriela Ben Melech Stan</au><au>Wofk, Diana</au><au>Fox, Scottie</au><au>Redden, Alex</au><au>Saxton, Will</au><au>Yu, Jean</au><au>Aflalo, Estelle</au><au>Shao-Yen Tseng</au><au>Nonato, Fabio</au><au>Muller, Matthias</au><au>Lal, Vasudev</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>LDM3D: Latent Diffusion Model for 3D</atitle><jtitle>arXiv.org</jtitle><date>2023-05-21</date><risdate>2023</risdate><eissn>2331-8422</eissn><abstract>This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that generates both image and depth map data from a given text prompt, allowing users to generate RGBD images from text prompts. The LDM3D model is fine-tuned on a dataset of tuples containing an RGB image, depth map and caption, and validated through extensive experiments. We also develop an application called DepthFusion, which uses the generated RGB images and depth maps to create immersive and interactive 360-degree-view experiences using TouchDesigner. This technology has the potential to transform a wide range of industries, from entertainment and gaming to architecture and design. Overall, this paper presents a significant contribution to the field of generative AI and computer vision, and showcases the potential of LDM3D and DepthFusion to revolutionize content creation and digital experiences. A short video summarizing the approach can be found at https://t.ly/tdi2.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier EISSN: 2331-8422
ispartof arXiv.org, 2023-05
issn 2331-8422
language eng
recordid cdi_proquest_journals_2815836634
source Publicly Available Content (ProQuest)
subjects Color imagery
Computer vision
Three dimensional models
Video data
title LDM3D: Latent Diffusion Model for 3D
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-30T04%3A49%3A19IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=LDM3D:%20Latent%20Diffusion%20Model%20for%203D&rft.jtitle=arXiv.org&rft.au=Gabriela%20Ben%20Melech%20Stan&rft.date=2023-05-21&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E2815836634%3C/proquest%3E%3Cgrp_id%3Ecdi_FETCH-proquest_journals_28158366343%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2815836634&rft_id=info:pmid/&rfr_iscdi=true