Differentiable Histogram Loss Functions for Intensity-based Image-to-Image Translation

• Published in: IEEE transactions on pattern analysis and machine intelligence 2023-10, Vol.45 (10), p.11642-11653
• Main Authors: Avi-Aharon, Mor, Arbelle, Assaf, Raviv, Tammy Riklin
• Format: Article
• Language: English
• Subjects: Color; Deep learning; earth movers distance; Generators; histogram layers; Histograms; Image color analysis; Image edge detection; Image processors; image-to-image translation; Intensity histogram loss functions; mutual information loss; Neural networks; Semantics; Similarity; Task analysis
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2023.3278287

We introduce the HueNet - a novel deep learning framework for a differentiable construction of intensity (1D) and joint (2D) histograms and present its applicability to paired and unpaired image-to-image translation problems. The key idea is an innovative technique for augmenting a generative neural network by histogram layers appended to the image generator. These histogram layers allow us to define two new histogram-based loss functions for constraining the structural appearance of the synthesized output image and its color distribution. Specifically, the color similarity loss is defined by the Earth Mover's Distance between the intensity histograms of the network output and a color reference image. The structural similarity loss is determined by the mutual information between the output and a content reference image based on their joint histogram. Although the HueNet can be applied to a variety of image-to-image translation problems, we chose to demonstrate its strength on the tasks of color transfer, exemplar-based image colorization, and edges \to → photo, where the colors of the output image are predefined. The code is available at https://github.com/mor-avi-aharon-bgu/HueNet.git .