Cross-Domain Semantic Segmentation via Domain-Invariant Interactive Relation Transfer

• Main Authors: Lv, Fengmao, Liang, Tao, Chen, Xiang, Lin, Guosheng
• Format: Conference Proceeding
• Language: English
• Subjects: Adaptation models; Biological system modeling; Image reconstruction; Image segmentation; Neural networks; Semantics; Training
• ISSN: 2575-7075
• DOI: 10.1109/CVPR42600.2020.00439

Exploiting photo-realistic synthetic data to train semantic segmentation models has received increasing attention over the past years. However, the domain mismatch between synthetic and real images will cause a significant performance drop when the model trained with synthetic images is directly applied to real-world scenarios. In this paper, we propose a new domain adaptation approach, called Pivot Interaction Transfer (PIT). Our method mainly focuses on constructing pivot information that is common knowledge shared across domains as a bridge to promote the adaptation of semantic segmentation model from synthetic domains to real-world domains. Specifically, we first infer the image-level category information about the target images, which is then utilized to facilitate pixel-level transfer for semantic segmentation, with the assumption that the interactive relation between the image-level category information and the pixel-level semantic information is invariant across domains. To this end, we propose a novel multi-level region expansion mechanism that aligns both the image-level and pixel-level information. Comprehensive experiments on the adaptation from both GTAV and SYNTHIA to Cityscapes clearly demonstrate the superiority of our method.