FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondence

We present a descriptor, called fully convolutional self-similarity (FCSS), for dense semantic correspondence. Unlike traditional dense correspondence approaches for estimating depth or optical flow, semantic correspondence estimation poses additional challenges due to intra-class appearance and sha...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on pattern analysis and machine intelligence 2019-03, Vol.41 (3), p.581-595
Main Authors: Kim, Seungryong, Min, Dongbo, Ham, Bumsub, Lin, Stephen, Sohn, Kwanghoon
Format: Article
Language:English
Subjects:
Affine transformations
Convolutional codes
convolutional neural networks
Dense semantic correspondence
Estimation
Microsoft Windows
Optical flow (image analysis)
Robustness
Sampling
Self-similarity
Semantics
Shape
Similarity
Training
weakly-supervised learning
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We present a descriptor, called fully convolutional self-similarity (FCSS), for dense semantic correspondence. Unlike traditional dense correspondence approaches for estimating depth or optical flow, semantic correspondence estimation poses additional challenges due to intra-class appearance and shape variations among different instances within the same object or scene category. To robustly match points across semantically similar images, we formulate FCSS using local self-similarity (LSS), which is inherently insensitive to intra-class appearance variations. LSS is incorporated through a proposed convolutional self-similarity (CSS) layer, where the sampling patterns and the self-similarity measure are jointly learned in an end-to-end and multi-scale manner. Furthermore, to address shape variations among different object instances, we propose a convolutional affine transformer (CAT) layer that estimates explicit affine transformation fields at each pixel to transform the sampling patterns and corresponding receptive fields. As training data for semantic correspondence is rather limited, we propose to leverage object candidate priors provided in most existing datasets and also correspondence consistency between object pairs to enable weakly-supervised learning. Experiments demonstrate that FCSS significantly outperforms conventional handcrafted descriptors and CNN-based descriptors on various benchmarks.
ISSN:0162-8828
1939-3539
2160-9292
DOI:10.1109/TPAMI.2018.2803169