Dense Adaptive Grouping Distillation Network for Multimodal Land Cover Classification With Privileged Modality

• Published in: IEEE transactions on geoscience and remote sensing 2022, Vol.60, p.1-14
• Main Authors: Li, Xiao, Lei, Lin, Zhang, Caiguang, Kuang, Gangyao
• Format: Article
• Language: English
• Subjects: Adaptive optics; Classification; Decoupling; Deep convolutional neural network (DCNN); Distillation; Distilling; Feature extraction; feature grouping; Immunoglobulins; knowledge distillation; Land cover; Machine learning; multimodal land cover classification; Optical imaging; Optical scattering; Optical sensors; privileged information; Remote sensing; Semantics; Synthetic aperture radar
• ISSN: 0196-2892; 1558-0644
• DOI: 10.1109/TGRS.2022.3176936

Multimodal land cover classification (MLCC) is a fundamental problem in remote sensing interpretation, which can obtain excellent performance on account of the complementary information between the optical and SAR modalities. However, it is usually impossible to obtain multimodal data at the same time, due to the restriction of imaging conditions. When one of the modalities data is completely missing during the test phase, classical multimodal learning methods might not be able to handle the MLCC task with privileged modality. In this article, we propose an efficient dense adaptive grouping distillation network (DAGDNet), which learns privileged information from available modalities in the train sets and improves the classification performance in the test sets when one of the modality data is scarce. More specifically, to relieve the heterogeneous gaps between different modalities and then transfer the privileged information, we propose an interactive gated-based feature grouping module (IG-FGM), which decomposes multimodal features into modality-shared and modality-specific components to realize the decoupling of multimodal features and grouping distillation. Furthermore, the IG-FGM is inserted into different layers of the "teacher" network to implement progressive blending of multimodalities. Then, to adaptively highlight the importance of hierarchical features distillation and grouping distillation, we propose a multistage adaptive distillation learning (MS-ADL) strategy so that the weights of different distillation losses are required to change continuously along with the training process. Finally, we evaluate the superior performances of our model on representative coregistered optical and SAR datasets.