DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space

This paper explores image modeling from the frequency space and introduces DCTdiff, an end-to-end diffusion generative paradigm that efficiently models images in the discrete cosine transform (DCT) space. We investigate the design space of DCTdiff and reveal the key design factors. Experiments on di...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-12
Main Authors: Mang Ning, Li, Mingxiao, Su, Jianlin, Jia, Haozhe, Liu, Lanmiao, Beneš, Martin, Albert Ali Salah, Itir Onal Ertugrul
Format: Article
Language:English
Subjects:
Autoregressive models
Design factors
Discrete cosine transform
Image quality
Image resolution
Modelling
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper explores image modeling from the frequency space and introduces DCTdiff, an end-to-end diffusion generative paradigm that efficiently models images in the discrete cosine transform (DCT) space. We investigate the design space of DCTdiff and reveal the key design factors. Experiments on different frameworks (UViT, DiT), generation tasks, and various diffusion samplers demonstrate that DCTdiff outperforms pixel-based diffusion models regarding generative quality and training efficiency. Remarkably, DCTdiff can seamlessly scale up to high-resolution generation without using the latent diffusion paradigm. Finally, we illustrate several intriguing properties of DCT image modeling. For example, we provide a theoretical proof of why `image diffusion can be seen as spectral autoregression', bridging the gap between diffusion and autoregressive models. The effectiveness of DCTdiff and the introduced properties suggest a promising direction for image modeling in the frequency space. The code is at \url{https://github.com/forever208/DCTdiff}.
ISSN:2331-8422