Part-Based Semantic Transform for Few-Shot Semantic Segmentation

Few-shot semantic segmentation remains an open problem for the lack of an effective method to handle the semantic misalignment between objects. In this article, we propose part-based semantic transform (PST) and target at aligning object semantics in support images with those in query images by sema...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transaction on neural networks and learning systems 2022-12, Vol.33 (12), p.7141-7152
Main Authors: Yang, Boyu, Wan, Fang, Liu, Chang, Li, Bohao, Ji, Xiangyang, Ye, Qixiang
Format: Article
Language:English
Subjects:
Algorithms
Decomposition
Feature extraction
Few-shot segmentation
Image segmentation
Measurement
Misalignment
Mixture models
prototype mixture models (PMMs)
Prototypes
semantic match
Semantic segmentation
semantic transform
Semantics
Transforms
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Few-shot semantic segmentation remains an open problem for the lack of an effective method to handle the semantic misalignment between objects. In this article, we propose part-based semantic transform (PST) and target at aligning object semantics in support images with those in query images by semantic decomposition-and-match. The semantic decomposition process is implemented with prototype mixture models (PMMs), which use an expectation-maximization (EM) algorithm to decompose object semantics into multiple prototypes corresponding to object parts. The semantic match between prototypes is performed with a min-cost flow module, which encourages correct correspondence while depressing mismatches between object parts. With semantic decomposition-and-match, PST enforces the network's tolerance to objects' appearance and/or pose variation and facilities channelwise and spatial semantic activation of objects in query images. Extensive experiments on Pascal VOC and MS-COCO datasets show that PST significantly improves upon state-of-the-arts. In particular, on MS-COCO, it improves the performance of five-shot semantic segmentation by up to 7.79% with a moderate cost of inference speed and model size. Code for PST is released at https://github.com/Yang-Bob/PST .
ISSN:2162-237X
2162-2388
DOI:10.1109/TNNLS.2021.3084252