Decoupling foreground and background with Siamese ViT networks for weakly-supervised semantic segmentation

• Published in: Neurocomputing (Amsterdam) 2024-12, Vol.610, p.128540, Article 128540
• Main Authors: Lin, Meiling, Li, Gongyan, Xu, Shaoyun, Hao, Yuexing, Zhang, Shu
• Format: Article
• Language: English
• Subjects: Class activation map; Multiple confidence regions; Semantic segmentation; Weakly supervised
• ISSN: 0925-2312
• DOI: 10.1016/j.neucom.2024.128540

Due to the coarse granularity of information extraction in image-level annotation-based weakly supervised semantic segmentation algorithms, there exists a significant gap between the generated pseudo-labels and the real pixel-level labels. In this paper, we propose the DeFB-SV framework, which consists of a dual-branch Siamese network structure. This framework separates the foreground and background of images by generating unified resolution and mixed resolution class activation maps, which are then fused to obtain pseudo-labels. The mixed-resolution class activation maps are produced by a new mixed-resolution patch partition method, where we introduce a semantically heuristic patch scorer to divide the image into patches of different sizes based on semantics. Additionally, a novel multi-confidence region division mechanism is proposed to enable the adaptive extraction of the effective parts of pseudo-labels, further enhancing the accuracy of weakly supervised semantic segmentation algorithms. The proposed semantic segmentation framework, DeFB-SV, is evaluated on the PASCAL VOC 2012 and MS COCO 2014 datasets, demonstrating comparable segmentation performance with state-of-the-art methods. •A novel weakly supervised semantic segmentation framework named DeFB-SV.•A Siamese network consisting of two ViT branches yielding fine-grained pseudo-labels.•A semantically heuristic patch scorer generating mixed-resolution image patches.•A multi-confidence-region strategy achieving finer segmentation results adaptively.