Expressive Power and Loss Surfaces of Deep Learning Models

The goals of this paper are two-fold. The first goal is to serve as an expository tutorial on the working of deep learning models which emphasizes geometrical intuition about the reasons for success of deep learning. The second goal is to complement the current results on the expressive power of dee...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2021-11
Main Author: Dube, Simant
Format: Article
Language:English
Subjects:
Artificial neural networks
Deep learning
Machine learning
Multiplication
Polynomials
Spin glasses
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The goals of this paper are two-fold. The first goal is to serve as an expository tutorial on the working of deep learning models which emphasizes geometrical intuition about the reasons for success of deep learning. The second goal is to complement the current results on the expressive power of deep learning models and their loss surfaces with novel insights and results. In particular, we describe how deep neural networks carve out manifolds especially when the multiplication neurons are introduced. Multiplication is used in dot products and the attention mechanism and it is employed in capsule networks and self-attention based transformers. We also describe how random polynomial, random matrix, spin glass and computational complexity perspectives on the loss surfaces are interconnected.
ISSN:2331-8422