Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations

We perform scalable approximate inference in continuous-depth Bayesian neural networks. In this model class, uncertainty about separate weights in each layer gives hidden units that follow a stochastic differential equation. We demonstrate gradient-based stochastic variational inference in this infi...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2022-01
Main Authors: Xu, Winnie, Chen, Ricky T Q, Li, Xuechen, Duvenaud, David
Format: Article
Language:English
Subjects:
Bayesian analysis
Differential equations
Inference
Neural networks
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We perform scalable approximate inference in continuous-depth Bayesian neural networks. In this model class, uncertainty about separate weights in each layer gives hidden units that follow a stochastic differential equation. We demonstrate gradient-based stochastic variational inference in this infinite-parameter setting, producing arbitrarily-flexible approximate posteriors. We also derive a novel gradient estimator that approaches zero variance as the approximate posterior over weights approaches the true posterior. This approach brings continuous-depth Bayesian neural nets to a competitive comparison against discrete-depth alternatives, while inheriting the memory-efficient training and tunable precision of Neural ODEs.
ISSN:2331-8422