Memory-based Parameter Adaptation

Deep neural networks have excelled on a wide range of problems, from vision to language and game playing. Neural networks very gradually incorporate information into weights as they process data, requiring very low learning rates. If the training distribution shifts, the network is slow to adapt, an...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2018-02
Main Authors: Sprechmann, Pablo, Jayakumar, Siddhant M, Rae, Jack W, Pritzel, Alexander, Adrià Puigdomènech Badia, Uria, Benigno, Vinyals, Oriol, Hassabis, Demis, Pascanu, Razvan, Blundell, Charles
Format: Article
Language:English
Subjects:
Adaptation
Image classification
Learning
Neural networks
Parameters
Training
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Deep neural networks have excelled on a wide range of problems, from vision to language and game playing. Neural networks very gradually incorporate information into weights as they process data, requiring very low learning rates. If the training distribution shifts, the network is slow to adapt, and when it does adapt, it typically performs badly on the training distribution before the shift. Our method, Memory-based Parameter Adaptation, stores examples in memory and then uses a context-based lookup to directly modify the weights of a neural network. Much higher learning rates can be used for this local adaptation, reneging the need for many iterations over similar data before good predictions can be made. As our method is memory-based, it alleviates several shortcomings of neural networks, such as catastrophic forgetting, fast, stable acquisition of new knowledge, learning with an imbalanced class labels, and fast learning during evaluation. We demonstrate this on a range of supervised tasks: large-scale image classification and language modelling.
ISSN:2331-8422