Fine-grained recognition: Multi-granularity labels and category similarity matrix

• Published in: Knowledge-based systems 2023-08, Vol.273, p.110599, Article 110599
• Main Authors: Shu, Xin, Zhang, Lei, Wang, Zizhou, Wang, Lituan, Yi, Zhang
• Format: Article
• Language: English
• Subjects: Category similarity; Fine-grained recognition; Multi-granularity classification
• ISSN: 0950-7051; 1872-7409
• DOI: 10.1016/j.knosys.2023.110599

The core of fine-grained recognition is to distinguish different subcategories within a same broad category through subtle differences in images. Yet two important factors are less explored: Firstly, the degree of similarity between categories, with the more similar categories being more likely to be confused. Secondly, different fine-grained definitions under divergent levels of expertise, with one coarse label corresponding to multiple similar fine categories (e.g., a brand of car has multiple models). In this paper, we discover that multi-granularity label can function as an intermediary for turning inter-category similarity relations into fine-grained recognition performance. Specifically, we first explore the association between label hierarchies in multi-granularity prediction: both coarse and fine label predictions require coarse information, but fine label prediction additionally requires subtle information. Then, a multi-granularity classification framework is proposed, where the extracted feature is decoupled and re-coupled into groups for predicting labels at different granularities, respectively. The key is to involve coarse-grained features in the prediction of finer-level labels. The realization way is to train these features with a joint probability-based loss that we design to reduce inter-task interference through inter-granularity probability relations. Lastly, for tasks without coarse labels, a category similarity matrix is suggested for measuring inter-class similarity and a further non-parametric aggregation method is devised for clustering fine labels into coarse labels. The evaluation is carried out on several fine-grained classification benchmark datasets. Results show that the proposed approach achieves state-of-the-art in multi-granularity classification and exhibits the potential to enhance existing fine-grained recognition models.