Semi-Parametric Inducing Point Networks and Neural Processes

We introduce semi-parametric inducing point networks (SPIN), a general-purpose architecture that can query the training set at inference time in a compute-efficient manner. Semi-parametric architectures are typically more compact than parametric models, but their computational complexity is often qu...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-03
Main Authors: Rastogi, Richa, Schiff, Yair, Hacohen, Alon, Li, Zhaozhi, Lee, Ian, Deng, Yuntian, Sabuncu, Mert R, Kuleshov, Volodymyr
Format: Article
Language:English
Subjects:
Artificial neural networks
Computational efficiency
Computer architecture
Computing costs
Inference
Machine learning
Neural networks
Training
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We introduce semi-parametric inducing point networks (SPIN), a general-purpose architecture that can query the training set at inference time in a compute-efficient manner. Semi-parametric architectures are typically more compact than parametric models, but their computational complexity is often quadratic. In contrast, SPIN attains linear complexity via a cross-attention mechanism between datapoints inspired by inducing point methods. Querying large training sets can be particularly useful in meta-learning, as it unlocks additional training signal, but often exceeds the scaling limits of existing models. We use SPIN as the basis of the Inducing Point Neural Process, a probabilistic model which supports large contexts in meta-learning and achieves high accuracy where existing models fail. In our experiments, SPIN reduces memory requirements, improves accuracy across a range of meta-learning tasks, and improves state-of-the-art performance on an important practical problem, genotype imputation.
ISSN:2331-8422