Learning from the Past Training Trajectories: Regularization by Validation

Deep model optimization methods discard the training weights which contain information about the validation loss landscape that can guide further model optimization. In this paper, we first show that a supervisor neural network can be used to predict the validation losses or accuracy of another deep...

Full description

Saved in:
Bibliographic Details
Published in:Journal of advanced computational intelligence and intelligent informatics 2024-01, Vol.28 (1), p.67-78
Main Authors: Zhang, Enzhi, Wahib, Mohamed, Zhong, Rui, Munetomo, Masaharu
Format: Article
Language:English
Subjects:
Multilayer perceptrons
Neural networks
Optimization
Regularization
Training
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Deep model optimization methods discard the training weights which contain information about the validation loss landscape that can guide further model optimization. In this paper, we first show that a supervisor neural network can be used to predict the validation losses or accuracy of another deep model (student) through its discarded training weights. Then based on this behavior, we propose a weight-loss (accuracy) pair-based training framework called regularization by validation to help decrease overfitting and increase the generalization performance of the student model by predicting the validation losses. We conduct our experiments on the MNIST, CIFAR-10, and CIFAR-100 datasets with the multilayer perceptron and ResNet-56 to show that we can improve the generalization performance with the past training trajectories.
ISSN:1343-0130
1883-8014
DOI:10.20965/jaciii.2024.p0067