Joint Classification of Hyperspectral and LiDAR Data Using Height Information Guided Hierarchical Fusion-and-Separation Network

Hyperspectral image (HSI) and light detection and ranging (LiDAR) data are complementary to each other, which can be combined to improve the classification performance. However, existing deep network models do not sufficiently consider their complementarity to design the network structure and loss f...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on geoscience and remote sensing 2024, Vol.62, p.1-15
Main Authors: Song, Tiecheng, Zeng, Zheng, Gao, Chenqiang, Chen, Haonan, Li, Jun
Format: Article
Language:English
Subjects:
Artificial neural networks
Classification
Complementarity
Convolution
Formability
Height
Hyperspectral imaging
Lidar
Modules
Neural networks
Separation
Spatial data
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Hyperspectral image (HSI) and light detection and ranging (LiDAR) data are complementary to each other, which can be combined to improve the classification performance. However, existing deep network models do not sufficiently consider their complementarity to design the network structure and loss functions. Moreover, there lacks a hierarchical mutual-assistance learning mechanism that leverages the modality-shared features to enhance the modality-specific ones and vice versa. In view of these, we propose a novel height information guided hierarchical fusion-and-separation network (HFSNet) for joint classification of HSI and LiDAR data. HFSNet consists of three major components, i.e., dual-structure feature encoders (DSFEs), feature fusion-and-separation blocks (F2SBs), and an edge decoder (ED). Specifically, the transformer and convolutional neural network (CNN) are introduced in DSFEs to encode the spectral and spatial information of HSI and LiDAR data, respectively. In F2SBs, the deformable convolution-based height information guided fusion module (HIGFM) and the modality separation refinement module (MSRM) are proposed to sequentially extract modality-shared and modality-specific features. Additionally, the ED is incorporated into our model to predict the LiDAR edge map from the HSI feature to improve the model’s generalization ability. As such, the learned features from HSI and LiDAR data are deeply fused and mutually enhanced. Experiments on three benchmark datasets show the superiority of HFSNet to the state-of-the-art methods for jointly classifying HSI and LiDAR data with limited training samples.
ISSN:0196-2892
1558-0644
DOI:10.1109/TGRS.2024.3353775