Fine-tuning Convolutional Neural Networks for fine art classification

• Published in: Expert systems with applications 2018-12, Vol.114, p.107-118
• Main Authors: Cetinic, Eva, Lipic, Tomislav, Grgic, Sonja
• Format: Article
• Language: English
• Subjects: Artificial neural networks; Classification; Computer vision; Convolutional codes; Convolutional Neural Networks; Deep learning; Digitization; Expert systems; Fine arts; Fine-tuning strategies; Image classification; Neural networks; Object recognition; On-line systems; Painting classification; Tuning; Weight
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2018.07.026

•We achieve state-of-the-art results for five fine art-related classification tasks.•Different convolutional neural network fine-tuning strategies are explored.•Impact of various source domain-dependent weight initialization is studied.•Networks pre-trained for scene and sentiment recognition perform best for art tasks.•Fine-tuned models can be used to retrieve images similar in style or content. The increasing availability of large digitized fine art collections opens new research perspectives in the intersection of artificial intelligence and art history. Motivated by the successful performance of Convolutional Neural Networks (CNN) for a wide variety of computer vision tasks, in this paper we explore their applicability for art-related image classification tasks. We perform extensive CNN fine-tuning experiments and consolidate in one place the results for five different art-related classification tasks on three large fine art datasets. Along with addressing the previously explored tasks of artist, genre, style and time period classification, we introduce a novel task of classifying artworks based on their association with a specific national artistic context. We present state-of-the-art classification results of the addressed tasks, signifying the impact of our method on computational analysis of art, as well as other image classification related research areas. Furthermore, in order to question transferability of deep representations across various source and target domains, we systematically compare the effects of domain-specific weight initialization by evaluating networks pre-trained for different tasks, varying from object and scene recognition to sentiment and memorability labelling. We show that fine-tuning networks pre-trained for scene recognition and sentiment prediction yields better results than fine-tuning networks pre-trained for object recognition. This novel outcome of our work suggests that the semantic correlation between different domains could be inherent in the CNN weights. Additionally, we address the practical applicability of our results by analysing different aspects of image similarity. We show that features derived from fine-tuned networks can be employed to retrieve images similar in either style or content, which can be used to enhance capabilities of search systems in different online art collections.