Loading…

Multimeasurement Generative Models

We formally map the problem of sampling from an unknown distribution with a density in \(\mathbb{R}^d\) to the problem of learning and sampling a smoother density in \(\mathbb{R}^{Md}\) obtained by convolution with a fixed factorial kernel: the new density is referred to as M-density and the kernel...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2022-06
Main Authors: Saremi, Saeed, Srivastava, Rupesh Kumar
Format: Article
Language:English
Subjects:
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We formally map the problem of sampling from an unknown distribution with a density in \(\mathbb{R}^d\) to the problem of learning and sampling a smoother density in \(\mathbb{R}^{Md}\) obtained by convolution with a fixed factorial kernel: the new density is referred to as M-density and the kernel as multimeasurement noise model (MNM). The M-density in \(\mathbb{R}^{Md}\) is smoother than the original density in \(\mathbb{R}^d\), easier to learn and sample from, yet for large \(M\) the two problems are mathematically equivalent since clean data can be estimated exactly given a multimeasurement noisy observation using the Bayes estimator. To formulate the problem, we derive the Bayes estimator for Poisson and Gaussian MNMs in closed form in terms of the unnormalized M-density. This leads to a simple least-squares objective for learning parametric energy and score functions. We present various parametrization schemes of interest including one in which studying Gaussian M-densities directly leads to multidenoising autoencoders--this is the first theoretical connection made between denoising autoencoders and empirical Bayes in the literature. Samples in \(\mathbb{R}^d\) are obtained by walk-jump sampling (Saremi & Hyvarinen, 2019) via underdamped Langevin MCMC (walk) to sample from M-density and the multimeasurement Bayes estimation (jump). We study permutation invariant Gaussian M-densities on MNIST, CIFAR-10, and FFHQ-256 datasets, and demonstrate the effectiveness of this framework for realizing fast-mixing stable Markov chains in high dimensions.
ISSN:2331-8422