Flexible Diffusion Modeling of Long Videos

We present a framework for video modeling based on denoising diffusion probabilistic models that produces long-duration video completions in a variety of realistic environments. We introduce a generative model that can at test-time sample any arbitrary subset of video frames conditioned on any other...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2022-12
Main Authors: Harvey, William, Naderiparizi, Saeid, Vaden Masrani, Weilbach, Christian, Wood, Frank
Format: Article
Language:English
Subjects:
Autonomous cars
Conditioning
Datasets
Frames (data processing)
Modelling
Probabilistic models
Video
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We present a framework for video modeling based on denoising diffusion probabilistic models that produces long-duration video completions in a variety of realistic environments. We introduce a generative model that can at test-time sample any arbitrary subset of video frames conditioned on any other subset and present an architecture adapted for this purpose. Doing so allows us to efficiently compare and optimize a variety of schedules for the order in which frames in a long video are sampled and use selective sparse and long-range conditioning on previously sampled frames. We demonstrate improved video modeling over prior work on a number of datasets and sample temporally coherent videos over 25 minutes in length. We additionally release a new video modeling dataset and semantically meaningful metrics based on videos generated in the CARLA autonomous driving simulator.
ISSN:2331-8422