Semi-supervised Cycle-GAN for face photo-sketch translation in the wild

• Published in: Computer vision and image understanding 2023-10, Vol.235, p.103775, Article 103775
• Main Authors: Chen, Chaofeng, Liu, Wei, Tan, Xiao, Wong, Kwan-Yee K.
• Format: Article
• Language: English
• Subjects: Cycle-GAN; Face photo-sketch translation; Semi-supervised; Steganography
• ISSN: 1077-3142; 1090-235X
• DOI: 10.1016/j.cviu.2023.103775

The performance of face photo-sketch translation has improved a lot thanks to deep neural networks. GAN based methods trained on paired images can produce high-quality results under laboratory settings. Such paired datasets are, however, often very small and lack diversity. Meanwhile, Cycle-GANs trained with unpaired photo-sketch datasets suffer from the steganography phenomenon, which makes them not effective to face photos in the wild. In this paper, we introduce a semi-supervised approach with a noise-injection strategy, named Semi-Cycle-GAN (SCG), to tackle these problems. For the first problem, we propose a pseudo sketch feature representation for each input photo composed from a small reference set of photo-sketch pairs, and use the resulting pseudo pairs to supervise a photo-to-sketch generator Gp2s. The outputs of Gp2s can in turn help to train a sketch-to-photo generator Gs2p in a self-supervised manner. This allows us to train Gp2s and Gs2p using a small reference set of photo-sketch pairs together with a large face photo dataset (without ground-truth sketches). For the second problem, we show that the simple noise-injection strategy works well to alleviate the steganography effect in SCG and helps to produce more reasonable sketch-to-photo results with less overfitting than fully supervised approaches. Experiments show that SCG achieves competitive performance on public benchmarks and superior results on photos in the wild. •A semi-supervised learning framework based on Cycle-GAN for face sketch translation.•The network can be trained with lots of photos without GT, improving generalization.•A simple noise-injection strategy to solve invisible steganography in Cycle-GAN.