Learning to Minimize the Remainder in Supervised Learning

The learning process of deep learning methods usually updates the model's parameters in multiple iterations. Each iteration can be viewed as the first-order approximation of Taylor's series expansion. The remainder, which consists of higher-order terms, is usually ignored in the learning p...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on multimedia 2023, Vol.25, p.1738-1748
Main Authors: Luo, Yan, Wong, Yongkang, Kankanhalli, Mohan S., Zhao, Qi
Format: Article
Language:English
Subjects:
Ablation
Approximation
Computational modeling
Data search
Deep learning
gradient adjustment
Image classification
Image retrieval
Iterative methods
Learning
Multimedia
Object detection
Object recognition
Optimization methods
Recommender systems
remainder
Semantics
Series expansion
Stochastic processes
Supervised learning
Task analysis
Training
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The learning process of deep learning methods usually updates the model's parameters in multiple iterations. Each iteration can be viewed as the first-order approximation of Taylor's series expansion. The remainder, which consists of higher-order terms, is usually ignored in the learning process for simplicity. This learning scheme empowers various multimedia-based applications, such as image retrieval, recommendation system, and video search. Generally, multimedia data ( e.g. images) are semantics-rich and high-dimensional, hence the remainders of approximations are possibly non-zero. In this work, we consider that the remainder is informative and study how it affects the learning process. To this end, we propose a new learning approach, namely gradient adjustment learning (GAL), to leverage the knowledge learned from the past training iterations to adjust vanilla gradients, such that the remainders are minimized and the approximations are improved. The proposed GAL is model- and optimizer-agnostic, and is easy to adapt to the standard learning framework. It is evaluated on three tasks, i.e. image classification, object detection, and regression, with state-of-the-art models and optimizers. The experiments show that the proposed GAL consistently enhances the evaluated models, whereas the ablation studies validate various aspects of the proposed GAL. The code is available at https://github.com/luoyan407/gradient_adjustment.git .
ISSN:1520-9210
1941-0077
DOI:10.1109/TMM.2022.3158066