Selective interactive networks with knowledge graphs for image classification

• Published in: Knowledge-based systems 2023-10, Vol.278, p.110889, Article 110889
• Main Authors: Tang, Wenqiang, Yang, Zhouwang, Song, Yanzhi
• Format: Article
• Language: English
• Subjects: Image classification; Integration effect; Knowledge graph; Knowledge selection
• ISSN: 0950-7051; 1872-7409
• DOI: 10.1016/j.knosys.2023.110889

The combination of networks and knowledge graphs (KGs) improves the accuracy and interpretability of networks in image classification. Previous methods fuse the entire KG with each sample. However, there is limited knowledge related to specific samples for the entire KG. The fusion process of the previous method includes some useless redundant knowledge, resulting in suboptimal fusion performance. In this paper, we propose a selective interactive network based on a knowledge graph, which builds a bidirectional communication bridge for knowledge and data. Specifically, we design an attention-based knowledge selection module that utilizes data to select effective knowledge from the KG, filtering out some useless knowledge involved in the subsequent fusion process. Meanwhile, we introduce knowledge selection loss that measures the consistency between knowledge and image features to achieve the effective selection of knowledge through data. Then, we employ a bilinear fusion method to combine the selected knowledge and image features to improve the recognition performance of the network. Extensive experiments conducted on the IDRiD and CUB-200-2011 datasets demonstrate the state-of-the-art performance of the proposed method, especially on the small dataset IDRiD, where the proposed method outperforms the baseline by 6.79% in accuracy. The knowledge selection mechanism establishes the association between predicted classes and the KG, and thus provides attribute descriptions for predicted classes, which has stronger interpretability. The visualization results demonstrate that our model captures more stable image features. •Our method builds a bidirectional bridge between the knowledge graph and network to enhance the network’s performance.•The designed knowledge selection module selects valid knowledge from the knowledge graph and offers attribute descriptions.•Accurate and effective knowledge guides networks to capture more stable image features.•Experimental results show that the proposed method achieves state-of-the-art performance on both small and large datasets.