Bidirectional Layout-Semantic-Pixel Joint Decoupling and Embedding Network for Remote Sensing Colorization

• Published in: IEEE transactions on geoscience and remote sensing 2024, Vol.62, p.1-14
• Main Authors: Nie, Jie, Wang, Jingyu, Jing, Niantai, Zuo, Zijie, Chen, Shuguo, Liang, Xinyue
• Format: Article
• Language: English
• Subjects: Atmospheric conditions; Color; Color imagery; Colorization; Colour; Consistency; Decoupling; Embedding; Feature extraction; Gray-scale; Image color analysis; Image processing; Layout; Layouts; Modules; multiscale; Pixels; Remote sensing; Remote sensors; Salience; Saturation; Saturation (color); Semantic segmentation; Semantics; Spatial distribution
• ISSN: 0196-2892; 1558-0644
• DOI: 10.1109/TGRS.2023.3342449

In recent years, there has been a growing demand for the colorization of remote sensing images due to their inherent limitations caused by remote sensors, such as hazy or noisy atmospheric conditions. These factors result in the captured images needing to be clarified. Compared to ordinary images, remote sensing images present unique challenges in color recovery due to their imbalanced spatial distribution of objects. In this article, we propose a novel bidirectional layout-semantic-pixel joint decoupling and embedding network (BDEnet) following the idea of human painting to generate highly saturated color images with strong spatial consistency and object salience. The proposed BDEnet model emulates the process of human painting through a step-by-step approach. It begins by determining the overall tone of a large macroscopic region and progressively refining the local color based on this initial assessment. Specifically, BDEnet incorporates finer-grained semantics and pixel color information into a colored layout that represents a wide range of continuous areas, thereby accomplishing the colorization task. The BDEnet model operates at three scales, namely the layout (macro), semantic (medium), and pixel (micro) scales. It comprises three key modules: the multiscale feature decoupling (MFD) module, the layout-semantic-pixel multigranularity learning (MGL) module, and the semantic-pixel embedding (SPE) module. MFD module effectively reduces redundant noise from the semantic and layout scales by employing scale decoupling. This process ensures the extraction of efficient features essential for MGL. In the MGL module, three branches with different scales are employed to achieve layout division, semantic segmentation, and pixel coloring. To address the issue of insufficient category label guidance in layouts, we propose a novel approach called similar semantic merging (SSM) using a weakly supervised scheme to accomplish layout division. Finally, the SPE module incorporates stable semantic and pixel information into the layout features. This integration results in the generation of color images that exhibit strong spatial consistency, emphasize object salience, and possess high color saturation.