4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion

We propose 4Real-Video, a novel framework for generating 4D videos, organized as a grid of video frames with both time and viewpoint axes. In this grid, each row contains frames sharing the same timestep, while each column contains frames from the same viewpoint. We propose a novel two-stream archit...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-12
Main Authors: Wang, Chaoyang, Zhuang, Peiye, Ngo, Tuan Duc, Menapace, Willi, Siarohin, Aliaksandr, Vasilkovsky, Michael, Skorokhodov, Ivan, Tulyakov, Sergey, Wonka, Peter, Hsin-Ying, Lee
Format: Article
Language:English
Subjects:
Diffusion layers
Diffusion rate
Frames (data processing)
Synchronism
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by
cites
container_end_page
container_issue
container_start_page
container_title arXiv.org
container_volume
creator Wang, Chaoyang
Zhuang, Peiye
Ngo, Tuan Duc
Menapace, Willi
Siarohin, Aliaksandr
Vasilkovsky, Michael
Skorokhodov, Ivan
Tulyakov, Sergey
Wonka, Peter
Hsin-Ying, Lee
description We propose 4Real-Video, a novel framework for generating 4D videos, organized as a grid of video frames with both time and viewpoint axes. In this grid, each row contains frames sharing the same timestep, while each column contains frames from the same viewpoint. We propose a novel two-stream architecture. One stream performs viewpoint updates on columns, and the other stream performs temporal updates on rows. After each diffusion transformer layer, a synchronization layer exchanges information between the two token streams. We propose two implementations of the synchronization layer, using either hard or soft synchronization. This feedforward architecture improves upon previous work in three ways: higher inference speed, enhanced visual quality (measured by FVD, CLIP, and VideoScore), and improved temporal and viewpoint consistency (measured by VideoScore and Dust3R-Confidence).
format article
fullrecord <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_3141681261</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>3141681261</sourcerecordid><originalsourceid>FETCH-proquest_journals_31416812613</originalsourceid><addsrcrecordid>eNqNi7sKwjAUQIMgWLT_EHAONI_G4mqtDg4i0rVEvdVbQqJJu_j1PvADnM5wzhmRREjJWaGEmJA0xi7LMqEXIs9lQip1AGNZjRfwS7oDExy6K92Ag2AsPs3JAt3ffO_ZJ8TY45mqkn4HWmLbDhG9m5Fxa2yE9McpmVfr42rL7sE_Boh90_khuLdqJFdcF1xoLv-rXjM8OjU</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3141681261</pqid></control><display><type>article</type><title>4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion</title><source>Publicly Available Content Database (Proquest) (PQ_SDU_P3)</source><creator>Wang, Chaoyang ; Zhuang, Peiye ; Ngo, Tuan Duc ; Menapace, Willi ; Siarohin, Aliaksandr ; Vasilkovsky, Michael ; Skorokhodov, Ivan ; Tulyakov, Sergey ; Wonka, Peter ; Hsin-Ying, Lee</creator><creatorcontrib>Wang, Chaoyang ; Zhuang, Peiye ; Ngo, Tuan Duc ; Menapace, Willi ; Siarohin, Aliaksandr ; Vasilkovsky, Michael ; Skorokhodov, Ivan ; Tulyakov, Sergey ; Wonka, Peter ; Hsin-Ying, Lee</creatorcontrib><description>We propose 4Real-Video, a novel framework for generating 4D videos, organized as a grid of video frames with both time and viewpoint axes. In this grid, each row contains frames sharing the same timestep, while each column contains frames from the same viewpoint. We propose a novel two-stream architecture. One stream performs viewpoint updates on columns, and the other stream performs temporal updates on rows. After each diffusion transformer layer, a synchronization layer exchanges information between the two token streams. We propose two implementations of the synchronization layer, using either hard or soft synchronization. This feedforward architecture improves upon previous work in three ways: higher inference speed, enhanced visual quality (measured by FVD, CLIP, and VideoScore), and improved temporal and viewpoint consistency (measured by VideoScore and Dust3R-Confidence).</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Diffusion layers ; Diffusion rate ; Frames (data processing) ; Synchronism</subject><ispartof>arXiv.org, 2024-12</ispartof><rights>2024. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://www.proquest.com/docview/3141681261?pq-origsite=primo$$EHTML$$P50$$Gproquest$$Hfree_for_read</linktohtml><link.rule.ids>780,784,25753,37012,44590</link.rule.ids></links><search><creatorcontrib>Wang, Chaoyang</creatorcontrib><creatorcontrib>Zhuang, Peiye</creatorcontrib><creatorcontrib>Ngo, Tuan Duc</creatorcontrib><creatorcontrib>Menapace, Willi</creatorcontrib><creatorcontrib>Siarohin, Aliaksandr</creatorcontrib><creatorcontrib>Vasilkovsky, Michael</creatorcontrib><creatorcontrib>Skorokhodov, Ivan</creatorcontrib><creatorcontrib>Tulyakov, Sergey</creatorcontrib><creatorcontrib>Wonka, Peter</creatorcontrib><creatorcontrib>Hsin-Ying, Lee</creatorcontrib><title>4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion</title><title>arXiv.org</title><description>We propose 4Real-Video, a novel framework for generating 4D videos, organized as a grid of video frames with both time and viewpoint axes. In this grid, each row contains frames sharing the same timestep, while each column contains frames from the same viewpoint. We propose a novel two-stream architecture. One stream performs viewpoint updates on columns, and the other stream performs temporal updates on rows. After each diffusion transformer layer, a synchronization layer exchanges information between the two token streams. We propose two implementations of the synchronization layer, using either hard or soft synchronization. This feedforward architecture improves upon previous work in three ways: higher inference speed, enhanced visual quality (measured by FVD, CLIP, and VideoScore), and improved temporal and viewpoint consistency (measured by VideoScore and Dust3R-Confidence).</description><subject>Diffusion layers</subject><subject>Diffusion rate</subject><subject>Frames (data processing)</subject><subject>Synchronism</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>PIMPY</sourceid><recordid>eNqNi7sKwjAUQIMgWLT_EHAONI_G4mqtDg4i0rVEvdVbQqJJu_j1PvADnM5wzhmRREjJWaGEmJA0xi7LMqEXIs9lQip1AGNZjRfwS7oDExy6K92Ag2AsPs3JAt3ffO_ZJ8TY45mqkn4HWmLbDhG9m5Fxa2yE9McpmVfr42rL7sE_Boh90_khuLdqJFdcF1xoLv-rXjM8OjU</recordid><startdate>20241205</startdate><enddate>20241205</enddate><creator>Wang, Chaoyang</creator><creator>Zhuang, Peiye</creator><creator>Ngo, Tuan Duc</creator><creator>Menapace, Willi</creator><creator>Siarohin, Aliaksandr</creator><creator>Vasilkovsky, Michael</creator><creator>Skorokhodov, Ivan</creator><creator>Tulyakov, Sergey</creator><creator>Wonka, Peter</creator><creator>Hsin-Ying, Lee</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20241205</creationdate><title>4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion</title><author>Wang, Chaoyang ; Zhuang, Peiye ; Ngo, Tuan Duc ; Menapace, Willi ; Siarohin, Aliaksandr ; Vasilkovsky, Michael ; Skorokhodov, Ivan ; Tulyakov, Sergey ; Wonka, Peter ; Hsin-Ying, Lee</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_31416812613</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Diffusion layers</topic><topic>Diffusion rate</topic><topic>Frames (data processing)</topic><topic>Synchronism</topic><toplevel>online_resources</toplevel><creatorcontrib>Wang, Chaoyang</creatorcontrib><creatorcontrib>Zhuang, Peiye</creatorcontrib><creatorcontrib>Ngo, Tuan Duc</creatorcontrib><creatorcontrib>Menapace, Willi</creatorcontrib><creatorcontrib>Siarohin, Aliaksandr</creatorcontrib><creatorcontrib>Vasilkovsky, Michael</creatorcontrib><creatorcontrib>Skorokhodov, Ivan</creatorcontrib><creatorcontrib>Tulyakov, Sergey</creatorcontrib><creatorcontrib>Wonka, Peter</creatorcontrib><creatorcontrib>Hsin-Ying, Lee</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science &amp; Engineering Collection</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>ProQuest Central Essentials</collection><collection>AUTh Library subscriptions: ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central</collection><collection>SciTech Premium Collection (Proquest) (PQ_SDU_P3)</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database (Proquest) (PQ_SDU_P3)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Wang, Chaoyang</au><au>Zhuang, Peiye</au><au>Ngo, Tuan Duc</au><au>Menapace, Willi</au><au>Siarohin, Aliaksandr</au><au>Vasilkovsky, Michael</au><au>Skorokhodov, Ivan</au><au>Tulyakov, Sergey</au><au>Wonka, Peter</au><au>Hsin-Ying, Lee</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion</atitle><jtitle>arXiv.org</jtitle><date>2024-12-05</date><risdate>2024</risdate><eissn>2331-8422</eissn><abstract>We propose 4Real-Video, a novel framework for generating 4D videos, organized as a grid of video frames with both time and viewpoint axes. In this grid, each row contains frames sharing the same timestep, while each column contains frames from the same viewpoint. We propose a novel two-stream architecture. One stream performs viewpoint updates on columns, and the other stream performs temporal updates on rows. After each diffusion transformer layer, a synchronization layer exchanges information between the two token streams. We propose two implementations of the synchronization layer, using either hard or soft synchronization. This feedforward architecture improves upon previous work in three ways: higher inference speed, enhanced visual quality (measured by FVD, CLIP, and VideoScore), and improved temporal and viewpoint consistency (measured by VideoScore and Dust3R-Confidence).</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier EISSN: 2331-8422
ispartof arXiv.org, 2024-12
issn 2331-8422
language eng
recordid cdi_proquest_journals_3141681261
source Publicly Available Content Database (Proquest) (PQ_SDU_P3)
subjects Diffusion layers
Diffusion rate
Frames (data processing)
Synchronism
title 4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-07T20%3A30%3A07IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=4Real-Video:%20Learning%20Generalizable%20Photo-Realistic%204D%20Video%20Diffusion&rft.jtitle=arXiv.org&rft.au=Wang,%20Chaoyang&rft.date=2024-12-05&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E3141681261%3C/proquest%3E%3Cgrp_id%3Ecdi_FETCH-proquest_journals_31416812613%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=3141681261&rft_id=info:pmid/&rfr_iscdi=true