Toward Deep Universal Sketch Perceptual Grouper

• Published in: IEEE transactions on image processing 2019-07, Vol.28 (7), p.3219-3231
• Main Authors: Ke Li, Kaiyue Pang, Yi-Zhe Song, Tao Xiang, Hospedales, Timothy M., Honggang Zhang
• Format: Article
• Language: English
• Subjects: Analytical models; Data models; dataset; deep grouping model; Image annotation; Image management; Image retrieval; Image segmentation; Semantics; Sketch perceptual grouping; Sketches; Task analysis; Training; universal grouper; Visualization
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2019.2895155

Human free-hand sketches provide the useful data for studying human perceptual grouping, where the grouping principles such as the Gestalt laws of grouping are naturally in play during both the perception and sketching stages. In this paper, we make the first attempt to develop a universal sketch perceptual grouper. That is, a grouper that can be applied to sketches of any category created with any drawing style and ability, to group constituent strokes/segments into semantically meaningful object parts. The first obstacle to achieving this goal is the lack of large-scale datasets with grouping annotation. To overcome this, we contribute the largest sketch perceptual grouping dataset to date, consisting of 20 000 unique sketches evenly distributed over 25 object categories. Furthermore, we propose a novel deep perceptual grouping model learned with both generative and discriminative losses. The generative loss improves the generalization ability of the model, while the discriminative loss guarantees both local and global grouping consistency. Extensive experiments demonstrate that the proposed grouper significantly outperforms the state-of-the-art competitors. In addition, we show that our grouper is useful for a number of sketch analysis tasks, including sketch semantic segmentation, synthesis, and fine-grained sketch-based image retrieval.