SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

Sketch-based image retrieval (SBIR) is a long-standing research topic in computer vision. Existing methods mainly focus on category-level or instance-level image retrieval. This paper investigates the fine-grained scene-level SBIR problem where a free-hand sketch depicting a scene is used to retriev...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on image processing 2022-01, Vol.31, p.3737-3751
Main Authors: Liu, Fang, Deng, Xiaoming, Zou, Changqing, Lai, Yu-Kun, Chen, Keqi, Zuo, Ran, Ma, Cuixia, Liu, Yong-Jin, Wang, Hongan
Format: Article
Language:English
Subjects:
Adaptation models
Computer vision
Domains
Electronic mail
fine-grained image retrieval
graph convolutional network
Image management
Image retrieval
Layout
Queries
scene sketch
Semantics
Sketch input
Sketch-based image retrieval
Task analysis
Training
Visualization
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Sketch-based image retrieval (SBIR) is a long-standing research topic in computer vision. Existing methods mainly focus on category-level or instance-level image retrieval. This paper investigates the fine-grained scene-level SBIR problem where a free-hand sketch depicting a scene is used to retrieve desired images. This problem is useful yet challenging mainly because of two entangled facts: 1) achieving an effective representation of the input query data and scene-level images is difficult as it requires to model the information across multiple modalities such as object layout, relative size and visual appearances, and 2) there is a great domain gap between the query sketch input and target images. We present SceneSketcher-v2, a Graph Convolutional Network (GCN) based architecture to address these challenges. SceneSketcher-v2 employs a carefully designed graph convolution network to fuse the multi-modality information in the query sketch and target images and uses a triplet training process and end-to-end training manner to alleviate the domain gap. Extensive experiments demonstrate SceneSketcher-v2 outperforms state-of-the-art scene-level SBIR models with a significant margin.
ISSN:1057-7149
1941-0042
DOI:10.1109/TIP.2022.3175403