Composite Object Relation Modeling for Few-Shot Scene Recognition

• Published in: IEEE transactions on image processing 2023, Vol.32, p.5678-5691
• Main Authors: Song, Xinhang, Liu, Chenlong, Zeng, Haitao, Zhu, Yaohui, Chen, Gongwei, Qin, Xiaorong, Jiang, Shuqiang
• Format: Article
• Language: English
• Subjects: Adaptation models; Artificial neural networks; Convolutional neural networks; Feature extraction; few-shot learning; generalization ability; graph modeling; Graph neural networks; Graphical representations; Image enhancement; Image recognition; Learning; Metalearning; Modelling; Neural networks; Scene recognition; Task analysis; Task complexity; Training
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2023.3321475

The goal of few-shot image recognition is to classify different categories with only one or a few training samples. Previous works of few-shot learning mainly focus on simple images, such as object or character images. Those works usually use a convolutional neural network (CNN) to learn the global image representations from training tasks, which are then adapted to novel tasks. However, there are many more abstract and complex images in real world, such as scene images, consisting of many object entities with flexible spatial relations among them. In such cases, global features can hardly obtain satisfactory generalization ability due to the large diversity of object relations in the scenes, which may hinder the adaptability to novel scenes. This paper proposes a composite object relation modeling method for few-shot scene recognition, capturing the spatial structural characteristic of scene images to enhance adaptability on novel scenes, considering that objects commonly co- occurred in different scenes. In different few-shot scene recognition tasks, the objects in the same images usually play different roles. Thus we propose a task-aware region selection module (TRSM) to further select the detected regions in different few-shot tasks. In addition to detecting object regions, we mainly focus on exploiting the relations between objects, which are more consistent to the scenes and can be used to cleave apart different scenes. Objects and relations are used to construct a graph in each image, which is then modeled with graph convolutional neural network. The graph modeling is jointly optimized with few-shot recognition, where the loss of few-shot learning is also capable of adjusting graph based representations. Typically, the proposed graph based representations can be plugged in different types of few-shot architectures, such as metric-based and meta-learning methods. Experimental results of few-shot scene recognition show the effectiveness of the proposed method.