A task decoupled framework for enhancing the deep learning-based spatiotemporal fusion method

• Published in: International journal of remote sensing 2023-07, Vol.44 (13), p.4163-4189
• Main Authors: Guo, Dizhou, Shi, Wenzhong
• Format: Article
• Language: English
• Subjects: Accuracy; Deep learning; Frameworks; Image reconstruction; Land cover; loss function; Mathematical analysis; Mathematical models; Modelling; Root-mean-square errors; Spatial data; Spatiotemporal fusion; task-decoupled framework
• ISSN: 0143-1161; 1366-5901
• DOI: 10.1080/01431161.2023.2232548

Spatiotemporal fusion (STF) is a cost-effective way to reconstruct time-series images. In recent years, deep learning-based (DL-based) STF methods have received substantial attention. However, two limitations of DL-based STF methods still remain: (1) existing methods require simultaneous learning of both the multi-source images correction model and the STF model, which complicates the training task. The high complexity poses a challenge for the network to accurately learn the underlying mathematical principles of STF, thereby reducing the method's reliability and generalization ability; (2) existing methods tend to generate blurry predictions. To address these limitations, this study proposes a task decoupled (TD) framework that offers a simple yet effective solution for enhancing DL-based STF method. The framework consists of a correction model and a STF model, which are trained using actual and simulated image pairs, respectively, to learn the multi-source images correction model and STF mathematical model. The loss of edge feature is added in the loss function of the STF model to ameliorate its detailed information preservation ability. The proposed framework is evaluated on three DL-based STF methods in five different sites using root-mean-square error (RMSE) and Robert's edge (Edge) to assess spectral and spatial accuracy. The experimental results indicate that the proposed framework can significantly improve the models' ability to predict spectral information (average increase rate = 5.3% in spectral accuracy), preserve spatial information (average increase rate = 16.2% in spatial accuracy), retrieve land cover change, and generalize to new data. These findings demonstrate the effectiveness of the proposed framework in addressing the limitations of existing DL-based STF methods and its potential for advancing STF applications.