Sequential Cross Attention Based Multi-Task Learning

In multi-task learning (MTL) for visual scene understanding, it is crucial to transfer useful information between multiple tasks with minimal interferences. In this paper, we propose a novel architecture that effectively transfers informative features by applying the attention mechanism to the multi...

Full description

Saved in:
Bibliographic Details
Main Authors: Kim, Sunkyung, Choi, Hyesong, Min, Dongbo
Format: Conference Proceeding
Language:English
Subjects:
Aggregates
Codes
cross attention
Estimation
Feature extraction
Image segmentation
monocular depth estimation
Multi-task learning
Multitasking
self-attention
semantic segmentation
Visualization
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In multi-task learning (MTL) for visual scene understanding, it is crucial to transfer useful information between multiple tasks with minimal interferences. In this paper, we propose a novel architecture that effectively transfers informative features by applying the attention mechanism to the multi-scale features of the tasks. Since applying the attention module directly to all possible features in terms of scale and task requires a high complexity, we propose to apply the attention module sequentially for the task and scale. The cross-task attention module (CTAM) is first applied to facilitate the exchange of relevant information between the multiple task features of the same scale. The cross-scale attention module (CSAM) then aggregates useful information from feature maps at different resolutions in the same task. Also, we attempt to capture long range dependencies through the self-attention module in the feature extraction network. Extensive experiments demonstrate that our method achieves state-of-the-art performance on the NYUD-v2 and PASCAL-Context dataset. Our code is available at https://github.com/kimsunkyung/SCA-MTL
ISSN:2381-8549
DOI:10.1109/ICIP46576.2022.9897871