Bridging Remote Sensors with Multisensor Geospatial Foundation Models

• Main Authors: Han, Boran, Zhang, Shuai, Shi, Xingjian, Reichstein, Markus
• Format: Conference Proceeding
• Language: English
• Subjects: Adaptation models; Data models; Geospatial analysis; Optical imaging; Predictive models; Representation learning; Scene classification
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52733.2024.02631

In the realm of geospatial analysis, the diversity of remote sensors, encompassing both optical and microwave technologies, offers a wealth of distinct observational capabilities. Recognizing this, we present msGFM, a multisensor geospatial foundation model that effectively unifies data from four key sensor modalities. This integration spans an expansive dataset of two million multisensor images. ms-GFM is uniquely adept at handling both paired and unpaired sensor data. For data originating from identical geolocations, our model employs an innovative cross-sensor pretraining approach in masked image modeling, enabling the synthesis of joint representations from diverse sensors. ms-GFM, incorporating four remote sensors, upholds strong performance, forming a comprehensive model adaptable to various sensor types. msGFM has demonstrated enhanced proficiency in a range of both single-sensor and multisensor downstream tasks. These include scene classification, segmentation, cloud removal, and pan-sharpening. A key discovery of our research is that representations derived from natural images are not always compatible with the distinct characteristics of geospatial remote sensors, under-scoring the limitations of existing representations in this field. Our work can serve as a guide for developing multisensor geospatial pretraining models, paving the way for more advanced geospatial capabilities. Code can be found at https://github.com/boranhan/Geospatial_Foundation_Models