Understanding the Local Geometry of Generative Model Manifolds

Deep generative models learn continuous representations of complex data manifolds using a finite number of samples during training. For a pre-trained generative model, the common way to evaluate the quality of the manifold representation learned, is by computing global metrics like Fréchet Inception...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-08
Main Authors: Ahmed Imtiaz Humayun, Ibtihel Amara, Schumann, Candice, Farnadi, Golnoosh, Rostamzadeh, Negar, Havaei, Mohammad
Format: Article
Language:English
Subjects:
Complexity
Geometry
Manifolds (mathematics)
Noise generation
Performance evaluation
Qualitative analysis
Representations
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Deep generative models learn continuous representations of complex data manifolds using a finite number of samples during training. For a pre-trained generative model, the common way to evaluate the quality of the manifold representation learned, is by computing global metrics like Fréchet Inception Distance using a large number of generated and real samples. However, generative model performance is not uniform across the learned manifold, e.g., for \textit{foundation models} like Stable Diffusion generation performance can vary significantly based on the conditioning or initial noise vector being denoised. In this paper we study the relationship between the \textit{local geometry of the learned manifold} and downstream generation. Based on the theory of continuous piecewise-linear (CPWL) generators, we use three geometric descriptors - scaling (\(\psi\)), rank (\(\nu\)), and complexity (\(\delta\)) - to characterize a pre-trained generative model manifold locally. We provide quantitative and qualitative evidence showing that for a given latent, the local descriptors are correlated with generation aesthetics, artifacts, uncertainty, and even memorization. Finally we demonstrate that training a \textit{reward model} on the local geometry can allow controlling the likelihood of a generated sample under the learned distribution.
ISSN:2331-8422