MaskFaceGAN: High-Resolution Face Editing With Masked GAN Latent Code Optimization

• Published in: IEEE transactions on image processing 2023, Vol.32, p.5893-5908
• Main Authors: Pernus, Martin, Struc, Vitomir, Dobrisek, Simon
• Format: Article
• Language: English
• Subjects: Codes; Computational modeling; Computer vision; Editing; Entanglement; Faces; Facial attribute editing; Facial features; GAN inversion; Generative adversarial networks; Image processing; Image quality; Image resolution; latent code optimization; Optimization; Semantics; Source code
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2023.3326675

Face editing represents a popular research topic within the computer vision and image processing communities. While significant progress has been made recently in this area, existing solutions: (i) are still largely focused on low-resolution images, (ii) often generate editing results with visual artefacts, or (iii) lack fine-grained control over the editing procedure and alter multiple (entangled) attributes simultaneously, when trying to generate the desired facial semantics. In this paper, we aim to address these issues through a novel editing approach, called MaskFaceGAN that focuses on local attribute editing. The proposed approach is based on an optimization procedure that directly optimizes the latent code of a pre-trained (state-of-the-art) Generative Adversarial Network (i.e., StyleGAN2) with respect to several constraints that ensure: (i) preservation of relevant image content, (ii) generation of the targeted facial attributes, and (iii) spatially-selective treatment of local image regions. The constraints are enforced with the help of an (differentiable) attribute classifier and face parser that provide the necessary reference information for the optimization procedure. MaskFaceGAN is evaluated in extensive experiments on the FRGC, SiblingsDB-HQf, and XM2VTS datasets and in comparison with several state-of-the-art techniques from the literature. Our experimental results show that the proposed approach is able to edit face images with respect to several local facial attributes with unprecedented image quality and at high-resolutions ( 1024\times 1024 ), while exhibiting considerably less problems with attribute entanglement than competing solutions. The source code is publicly available from: https://github.com/MartinPernus/MaskFaceGAN .