Learning Cross-Modality Representations From Multi-Modal Images

Machine learning algorithms can have difficulties adapting to data from different sources, for example from different imaging modalities. We present and analyze three techniques for unsupervised cross-modality feature learning, using a shared auto-encoder-like convolutional network that learns a com...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on medical imaging 2019-02, Vol.38 (2), p.638-648
Main Authors: van Tulder, Gijs, de Bruijne, Marleen
Format: Article
Language:English
Subjects:
Accuracy
Artificial neural networks
autoencoders
Biocompatibility
Biomedical imaging
Biomedical materials
Brain
Brain tumors
Classification
Computed tomography
Data processing
Decoding
Deep Learning
Encoding
Humans
Image processing
Image Processing, Computer-Assisted - methods
Image reconstruction
Image segmentation
Knee
Knee Joint - diagnostic imaging
Learning algorithms
Machine learning
Magnetic Resonance Imaging
Medical imaging
Modal data
Model accuracy
Multimodal Imaging - methods
Neuroimaging
Osteoarthritis
Representation learning
Representations
Training
transfer learning
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Machine learning algorithms can have difficulties adapting to data from different sources, for example from different imaging modalities. We present and analyze three techniques for unsupervised cross-modality feature learning, using a shared auto-encoder-like convolutional network that learns a common representation from multi-modal data. We investigate a form of feature normalization, a learning objective that minimizes cross-modality differences, and modality dropout, in which the network is trained with varying subsets of modalities. We measure the same-modality and cross-modality classification accuracies and explore whether the models learn modality-specific or shared features. This paper presents experiments on two public data sets, with knee images from two MRI modalities, provided by the Osteoarthritis Initiative, and brain tumor segmentation on four MRI modalities from the BRATS challenge. All three approaches improved the cross-modality classification accuracy, with modality dropout and per-feature normalization giving the largest improvement. We observed that the networks tend to learn a combination of cross-modality and modality-specific features. Overall, a combination of all three methods produced the most cross-modality features and the highest cross-modality classification accuracy, while maintaining most of the same-modality accuracy.
ISSN:0278-0062
1558-254X
DOI:10.1109/TMI.2018.2868977