X-ModalNet: A semi-supervised deep cross-modal network for classification of remote sensing data

• Published in: ISPRS journal of photogrammetry and remote sensing 2020-09, Vol.167, p.12-23
• Main Authors: Hong, Danfeng, Yokoya, Naoto, Xia, Gui-Song, Chanussot, Jocelyn, Zhu, Xiao Xiang
• Format: Article
• Language: English
• Subjects: Adversarial; Cross-modality; Deep learning; Deep neural network; Engineering Sciences; Fusion; Hyperspectral; Label propagation; Multispectral; Mutual learning; Remote sensing; Semi-supervised; Signal and Image processing; Synthetic aperture radar
• ISSN: 0924-2716; 1872-8235
• DOI: 10.1016/j.isprsjprs.2020.06.014

This paper addresses the problem of semi-supervised transfer learning with limited cross-modality data in remote sensing. A large amount of multi-modal earth observation images, such as multispectral imagery (MSI) or synthetic aperture radar (SAR) data, are openly available on a global scale, enabling parsing global urban scenes through remote sensing imagery. However, their ability in identifying materials (pixel-wise classification) remains limited, due to the noisy collection environment and poor discriminative information as well as limited number of well-annotated training images. To this end, we propose a novel cross-modal deep-learning framework, called X-ModalNet, with three well-designed modules: self-adversarial module, interactive learning module, and label propagation module, by learning to transfer more discriminative information from a small-scale hyperspectral image (HSI) into the classification task using a large-scale MSI or SAR data. Significantly, X-ModalNet generalizes well, owing to propagating labels on an updatable graph constructed by high-level features on the top of the network, yielding semi-supervised cross-modality learning. We evaluate X-ModalNet on two multi-modal remote sensing datasets (HSI-MSI and HSI-SAR) and achieve a significant improvement in comparison with several state-of-the-art methods.