The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks

Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights. Recent work developing this class of methods has explored ever richer parameterizations of the approximate posterior in the hope of improving performance. In contra...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2020-07
Main Authors: Swiatkowski, Jakub, Roth, Kevin, Veeling, Bastiaan S, Tran, Linh, Dillon, Joshua V, Snoek, Jasper, Mandt, Stephan, Salimans, Tim, Jenatton, Rodolphe, Nowozin, Sebastian
Format: Article
Language:English
Subjects:
Approximation
Bayesian analysis
Convergence
Lower bounds
Neural networks
Normal distribution
Parameterization
Signal to noise ratio
Statistical inference
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights. Recent work developing this class of methods has explored ever richer parameterizations of the approximate posterior in the hope of improving performance. In contrast, here we share a curious experimental finding that suggests instead restricting the variational distribution to a more compact parameterization. For a variety of deep Bayesian neural networks trained using Gaussian mean-field variational inference, we find that the posterior standard deviations consistently exhibit strong low-rank structure after convergence. This means that by decomposing these variational parameters into a low-rank factorization, we can make our variational approximation more compact without decreasing the models' performance. Furthermore, we find that such factorized parameterizations improve the signal-to-noise ratio of stochastic gradient estimates of the variational lower bound, resulting in faster convergence.
ISSN:2331-8422