Trainable Weights for Multitask Learning

The research on multi-task learning has been steadily increasing due to its advantages, such as preventing overfitting, averting catastrophic forgetting, solving multiple inseparable tasks, and coping with data shortage. Here, we question whether to incorporate different orderings of feature levels...

Full description

Saved in:
Bibliographic Details
Published in:IEEE access 2023, Vol.11, p.105633-105641
Main Authors: Ryu, Chaeeun, Lee, Changwoo, Choi, Hyuk Jin, Lee, Chang-Hyun, Jeon, Byoungjun, Chie, Eui Kyu, Kim, Young-Gon
Format: Article
Language:English
Subjects:
Auxiliary task learning
Data models
Feature extraction
Hospitals
incremental learning multitask learning
Knowledge representation
Learning
Multitasking
Parameter estimation
Predictive models
Questions
Task analysis
Topology
trainable parameters
Training
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The research on multi-task learning has been steadily increasing due to its advantages, such as preventing overfitting, averting catastrophic forgetting, solving multiple inseparable tasks, and coping with data shortage. Here, we question whether to incorporate different orderings of feature levels based on distinct characteristics of tasks and their interrelationships in multitask learning. While in many classification tasks, leveraging the features extracted from the last layer is common, we thought that given different characteristics of tasks there might be a need to encompass different representation levels, i.e., different orderings of feature levels. Hence, we utilized the knowledge of different representation levels by features extracted from the various blocks of the main module and applied trainable parameters as weights on the features. This indicates that we optimized the solution to the question by learning to weigh the features in a task-specific manner and solving tasks with a combination of newly weighted features. Our method SimPara presents a modular topology of multitask learning that is efficient in terms of memory and computation, effective, and easily applicable to diverse tasks or models. To show that our approach is task-agnostic and highly applicable, we demonstrate its effectiveness in auxiliary task learning, active learning, and multilabel learning settings. This work underscores that by simply learning weights to better order the features learned by a single backbone, we can incur better task-specific performance of the model.
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2023.3319072