More Control for Free! Image Synthesis with Semantic Diffusion Guidance

• Main Authors: Liu, Xihui, Park, Dong Huk, Azadi, Samaneh, Zhang, Gong, Chopikyan, Arman, Hu, Yuxiao, Shi, Humphrey, Rohrbach, Anna, Darrell, Trevor
• Format: Conference Proceeding
• Language: English
• Subjects: Algorithms: Computational photography; Annotations; Computer vision; image and video synthesis; Image matching; Image synthesis; Noise reduction; Probabilistic logic; Semantics; Vision + language and/or other modalities
• ISSN: 2642-9381
• DOI: 10.1109/WACV56688.2023.00037

Controllable image synthesis models allow creation of diverse images based on text instructions or guidance from a reference image. Recently, denoising diffusion probabilistic models have been shown to generate more realistic imagery than prior methods, and have been successfully demonstrated in unconditional and class-conditional settings. We investigate fine-grained, continuous control of this model class, and introduce a novel unified framework for semantic diffusion guidance, which allows either language or image guidance, or both. Guidance is injected into a pretrained unconditional diffusion model using the gradient of image-text or image matching scores, without re-training the diffusion model. We explore CLIP-based language guidance as well as both content and style-based image guidance in a unified framework. Our text-guided synthesis approach can be applied to datasets without associated text annotations. We conduct experiments on FFHQ and LSUN datasets, and show results on fine-grained text-guided image synthesis, synthesis of images related to a style or content reference image, and examples with both textual and image guidance. 1