LG-CNN: From local parts to global discrimination for fine-grained recognition

• Published in: Pattern recognition 2017-11, Vol.71, p.118-131
• Main Authors: Xie, Guo-Sen, Zhang, Xu-Yao, Yang, Wenhan, Xu, Mingliang, Yan, Shuicheng, Liu, Cheng-Lin
• Format: Article
• Language: English
• Subjects: Bilinear pooling; Convolutional neural networks; Fine-grained recognition; Global discrimination; Local parts
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2017.06.002

•This paper presents a fine-grained recognition system, without using bounding box and part information in both training and testing phase.•We propose procedures for unsupervised part localization and global object discovery, and the localized part candidates and discovered approximate objects are taken as inputs of two-stream CNN.•The two-stream CNN architecture is proposed to model both the Local part information and the Global discriminative information in a joint framework.•We report superior or competitive results on public datasets CUB-200, Flower-102, Pets-37 and Caltech-101. Fine-grained recognition is one of the most difficult topics in visual recognition, which aims at distinguishing confusing categories such as bird species within a genus. The information of part and bounding boxes in fine-grained images is very important for improving the performance. However, in real applications, the part and/or bounding box annotations may not exist. This makes fine-grained recognition a challenging problem. In this paper, we propose a jointly trained Convolutional Neural Network (CNN) architecture to solve the fine-grained recognition problem without using part and bounding box information. In this framework, we first detect part candidates by calculating the gradients of feature maps of a trained CNN model w.r.t. the input image and then filter out unnecessary ones by fusing two saliency detection methods. Meanwhile, two groups of global object locations are obtained based on the saliency detection methods and a segmentation method. With the filtered part candidates and approximate object locations as inputs, we construct the CNN architecture with local parts and global discrimination (LG-CNN) which consists of two CNN networks with shared weights. The upper stream of LG-CNN is focused on the part information of the input image, the bottom stream of LG-CNN is focused on the global input image. LG-CNN is jointly trained by two stream loss functions to guide the updating of the shared weights. Experiments on three popular fine-grained datasets well validate the effectiveness of our proposed LG-CNN architecture. Applying our LG-CNN architecture to generic object recognition datasets also yields superior performance over the directly fine-tuned CNN architecture with a large margin.