Subspace Inference for Bayesian Deep Learning

Bayesian inference was once a gold standard for learning with neural networks, providing accurate full predictive distributions and well calibrated uncertainty. However, scaling Bayesian inference techniques to deep neural networks is challenging due to the high dimensionality of the parameter space...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2019-07
Main Authors: Izmailov, Pavel, Maddox, Wesley J, Kirichenko, Polina, Garipov, Timur, Vetrov, Dmitry, Andrew Gordon Wilson
Format: Article
Language:English
Subjects:
Bayesian analysis
Deep learning
Image classification
Mathematical models
Neural networks
Parameters
Predictions
Statistical inference
Subspaces
Uncertainty
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Bayesian inference was once a gold standard for learning with neural networks, providing accurate full predictive distributions and well calibrated uncertainty. However, scaling Bayesian inference techniques to deep neural networks is challenging due to the high dimensionality of the parameter space. In this paper, we construct low-dimensional subspaces of parameter space, such as the first principal components of the stochastic gradient descent (SGD) trajectory, which contain diverse sets of high performing models. In these subspaces, we are able to apply elliptical slice sampling and variational inference, which struggle in the full parameter space. We show that Bayesian model averaging over the induced posterior in these subspaces produces accurate predictions and well calibrated predictive uncertainty for both regression and image classification.
ISSN:2331-8422