An analysis of training and generalization errors in shallow and deep networks

This paper is motivated by an open problem around deep networks, namely, the apparent absence of over-fitting despite large over-parametrization which allows perfect fitting of the training data. In this paper, we analyze this phenomenon in the case of regression problems when each unit evaluates a...

Full description

Saved in:
Bibliographic Details
Published in:Neural networks 2020-01, Vol.121, p.229-241
Main Authors: Mhaskar, H.N., Poggio, T.
Format: Article
Language:English
Subjects:
Deep learning
Generalization error
Humans
Interpolatory approximation
Machine Learning
Neural Networks, Computer
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper is motivated by an open problem around deep networks, namely, the apparent absence of over-fitting despite large over-parametrization which allows perfect fitting of the training data. In this paper, we analyze this phenomenon in the case of regression problems when each unit evaluates a periodic activation function. We argue that the minimal expected value of the square loss is inappropriate to measure the generalization error in approximation of compositional functions in order to take full advantage of the compositional structure. Instead, we measure the generalization error in the sense of maximum loss, and sometimes, as a pointwise error. We give estimates on exactly how many parameters ensure both zero training error as well as a good generalization error. We prove that a solution of a regularization problem is guaranteed to yield a good training error as well as a good generalization error and estimate how much error to expect at which test data.
ISSN:0893-6080
1879-2782
DOI:10.1016/j.neunet.2019.08.028