Linear algebra with transformers

Transformers can learn to perform numerical computations from examples only. I study nine problems of linear algebra, from basic matrix operations to eigenvalue decomposition and inversion, and introduce and discuss four encoding schemes to represent real numbers. On all problems, transformers train...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2022-11
Main Author: Charton, François
Format: Article
Language:English
Subjects:
Accuracy
Datasets
Eigenvalues
Linear algebra
Matrices (mathematics)
Matrix algebra
Multiplication
Singular value decomposition
Training
Transformers
Vectors (mathematics)
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Transformers can learn to perform numerical computations from examples only. I study nine problems of linear algebra, from basic matrix operations to eigenvalue decomposition and inversion, and introduce and discuss four encoding schemes to represent real numbers. On all problems, transformers trained on sets of random matrices achieve high accuracies (over 90%). The models are robust to noise, and can generalize out of their training distribution. In particular, models trained to predict Laplace-distributed eigenvalues generalize to different classes of matrices: Wigner matrices or matrices with positive eigenvalues. The reverse is not true.
ISSN:2331-8422