Learning to generate images with perceptual similarity metrics

• Main Authors: Snell, Jake, Ridgeway, Karl, Liao, Renjie, Roads, Brett D., Mozer, Michael C., Zemel, Richard S.
• Format: Conference Proceeding
• Language: English
• Subjects: Deep Learning; Image coding; Image generation; Image reconstruction; Measurement; Perceptual Losses; Training; Visualization
• ISSN: 2381-8549
• DOI: 10.1109/ICIP.2017.8297089

Deep networks are increasingly being applied to problems involving image synthesis, e.g., generating images from textual descriptions and reconstructing an input image from a compact representation. Supervised training of image-synthesis networks typically uses a pixel-wise loss (PL) to indicate the mismatch between a generated image and its corresponding target image. We propose instead to use a loss function that is better calibrated to human perceptual judgments of image quality: the multiscale structural-similarity score (MS-SSIM) [1]. Because MS-SSIM is differentiable, it is easily incorporated into gradient-descent learning. We compare the consequences of using MS-SSIM versus PL loss on training autoencoders. Human observers reliably prefer images synthesized by MS-SSIM-optimized models over those synthesized by PL-optimized models, for two distinct PL measures (L 1 and L 2 distances). We also explore the effect of training objective on image encoding and analyze conditions under which perceptually-optimized representations yield better performance on image classification. Finally, we demonstrate the superiority of perceptually-optimized networks for super-resolution imaging. We argue that significant advances can be made in modeling images through the use of training objectives that are well aligned to characteristics of human perception.