A Unified Visual Information Preservation Framework for Self-supervised Pre-training in Medical Image Analysis

Recent advances in self-supervised learning (SSL) in computer vision are primarily comparative, whose goal is to preserve invariant and discriminative semantics in latent representations by comparing siamese image views. However, the preserved high-level semantics do not contain enough local informa...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on pattern analysis and machine intelligence 2023-07, Vol.45 (7), p.1-16
Main Authors: Zhou, Hong-Yu, Lu, Chixiang, Chen, Chaoqi, Yang, Sibei, Yu, Yizhou
Format: Article
Language:English
Subjects:
Algorithms
Annotations
Brain Neoplasms
Computer vision
Context restoration
Feature pyramid
Humans
Image analysis
Image Processing, Computer-Assisted
Image restoration
Image segmentation
Imaging, Three-Dimensional
Medical diagnostic imaging
Medical image analysis
Medical imaging
Optimization
Pixels
Restoration
Self-supervised learning
Semantics
Task analysis
Three-dimensional displays
Transfer Learning
Tumors
Visualization
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Recent advances in self-supervised learning (SSL) in computer vision are primarily comparative, whose goal is to preserve invariant and discriminative semantics in latent representations by comparing siamese image views. However, the preserved high-level semantics do not contain enough local information, which is vital in medical image analysis (e.g., image-based diagnosis and tumor segmentation). To mitigate the locality problem of comparative SSL, we propose to incorporate the task of pixel restoration for explicitly encoding more pixel-level information into high-level semantics. We also address the preservation of scale information, a powerful tool in aiding image understanding but has not drawn much attention in SSL. The resulting framework can be formulated as a multi-task optimization problem on the feature pyramid. Specifically, we conduct multi-scale pixel restoration and siamese feature comparison in the pyramid. In addition, we propose non-skip U-Net to build the feature pyramid and develop sub-crop to replace multi-crop in 3D medical imaging. The proposed unified SSL framework (PCRLv2) surpasses its self-supervised counterparts on various tasks, including brain tumor segmentation (BraTS 2018), chest pathology identification (ChestX-ray, CheXpert), pulmonary nodule detection (LUNA), and abdominal organ segmentation (LiTS), sometimes outperforming them by large margins with limited annotations. Codes and models are available at https://github.com/RL4M/PCRLv2.
ISSN:0162-8828
1939-3539
2160-9292
DOI:10.1109/TPAMI.2023.3234002