Style spectroscope: improve interpretability and controllability through Fourier analysis

• Published in: Machine learning 2024-06, Vol.113 (6), p.3485-3503
• Main Authors: Jin, Zhiyu, Shen, Xuli, Li, Bin, Xue, Xiangyang
• Format: Article
• Language: English
• Subjects: Algorithms; Amplitudes; Artificial Intelligence; Computer Science; Control; Controllability; Datasets; Equivalence; Feature maps; Fourier analysis; Image reconstruction; Machine Learning; Mechatronics; Natural Language Processing (NLP); Robotics; Simulation and Modeling; Special Issue of the ACML 2023
• ISSN: 0885-6125; 1573-0565
• DOI: 10.1007/s10994-023-06435-5

Universal style transfer (UST) infuses styles from arbitrary reference images into content images. Existing methods, while enjoying many practical successes, are unable of explaining experimental observations, including different performances of UST algorithms in preserving the spatial structure of content images. In addition, methods are limited to cumbersome global controls on stylization, so that they require additional spatial masks for desired stylization. In this work, we first provide a systematic Fourier analysis on a general framework for UST. We present an equivalent form of the framework in the frequency domain. The form implies that existing algorithms treat all frequency components and pixels of feature maps equally, except for the zero-frequency component. We connect Fourier amplitude and phase with a widely used style loss and a well-known content reconstruction loss in style transfer, respectively. Based on such equivalence and connections, we can thus interpret different structure preservation behaviors between algorithms with Fourier phase. Given the interpretations, we propose two plug-and-play manipulations upon style transfer methods for better structure preservation and desired stylization. Both qualitative and quantitative experiments demonstrate the improved performance of our manipulations upon mainstreaming methods without any additional training. Specifically, the metrics are improved by 6% in average on the content images from MS-COCO dataset and the style images from WikiArt dataset. We also conduct experiments to demonstrate (1) the abovementioned equivalence, (2) the interpretability based on Fourier amplitude and phase and (3) the controllability associated with frequency components.