RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model

Leveraging the extensive training data from SA-1B, the segment anything model (SAM) demonstrates remarkable generalization and zero-shot capabilities. However, as a category-agnostic instance segmentation method, SAM heavily relies on prior manual guidance, including points, boxes, and coarse-graine...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on geoscience and remote sensing 2024, Vol.62, p.1-17
Main Authors: Chen, Keyan, Liu, Chenyang, Chen, Hao, Zhang, Haotian, Li, Wenyuan, Zou, Zhengxia, Shi, Zhenwei
Format: Article
Language:English
Subjects:
Datasets
Illustrations
Image processing
Image segmentation
Instance segmentation
Learning
Remote sensing
Ships
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Leveraging the extensive training data from SA-1B, the segment anything model (SAM) demonstrates remarkable generalization and zero-shot capabilities. However, as a category-agnostic instance segmentation method, SAM heavily relies on prior manual guidance, including points, boxes, and coarse-grained masks. Furthermore, its performance in remote sensing image segmentation tasks remains largely unexplored and unproven. In this article, we aim to develop an automated instance segmentation approach for remote sensing images based on the foundational SAM model and incorporating semantic category information. Drawing inspiration from prompt learning, we propose a method to learn the generation of appropriate prompts for SAM. This enables SAM to produce semantically discernible segmentation results for remote sensing images, a concept that we have termed RSPrompter. We also propose several ongoing derivatives for instance segmentation tasks, drawing on recent advancements within the SAM community, and compare their performance with RSPrompter. Extensive experimental results, derived from the WHU building dataset, the NWPU VHR-10 dataset, and the SAR Ship Detection Dataset (SSDD) dataset, validate the effectiveness of our proposed method. The code for our method is publicly available at https://kychen.me/RSPrompter .
ISSN:0196-2892
1558-0644
DOI:10.1109/TGRS.2024.3356074