Loading…
Multimeasurement Generative Models
We formally map the problem of sampling from an unknown distribution with a density in \(\mathbb{R}^d\) to the problem of learning and sampling a smoother density in \(\mathbb{R}^{Md}\) obtained by convolution with a fixed factorial kernel: the new density is referred to as M-density and the kernel...
Saved in:
Published in: | arXiv.org 2022-06 |
---|---|
Main Authors: | , |
Format: | Article |
Language: | English |
Subjects: | |
Online Access: | Get full text |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Summary: | We formally map the problem of sampling from an unknown distribution with a density in \(\mathbb{R}^d\) to the problem of learning and sampling a smoother density in \(\mathbb{R}^{Md}\) obtained by convolution with a fixed factorial kernel: the new density is referred to as M-density and the kernel as multimeasurement noise model (MNM). The M-density in \(\mathbb{R}^{Md}\) is smoother than the original density in \(\mathbb{R}^d\), easier to learn and sample from, yet for large \(M\) the two problems are mathematically equivalent since clean data can be estimated exactly given a multimeasurement noisy observation using the Bayes estimator. To formulate the problem, we derive the Bayes estimator for Poisson and Gaussian MNMs in closed form in terms of the unnormalized M-density. This leads to a simple least-squares objective for learning parametric energy and score functions. We present various parametrization schemes of interest including one in which studying Gaussian M-densities directly leads to multidenoising autoencoders--this is the first theoretical connection made between denoising autoencoders and empirical Bayes in the literature. Samples in \(\mathbb{R}^d\) are obtained by walk-jump sampling (Saremi & Hyvarinen, 2019) via underdamped Langevin MCMC (walk) to sample from M-density and the multimeasurement Bayes estimation (jump). We study permutation invariant Gaussian M-densities on MNIST, CIFAR-10, and FFHQ-256 datasets, and demonstrate the effectiveness of this framework for realizing fast-mixing stable Markov chains in high dimensions. |
---|---|
ISSN: | 2331-8422 |