Towards Open-Set Scene Graph Generation With Unknown Objects

Scene graph generation (SGG) aims to detect objects and their relationships in an image, thereby enabling a detailed understanding of a complex scene for various real-world applications. In SGG applications such as robot vision, it is important to correctly detect all objects without recognizing any...

Full description

Saved in:
Bibliographic Details
Published in:IEEE access 2022, Vol.10, p.11574-11583
Main Authors: Sonogashira, Motoharu, Iiyama, Masaaki, Kawanishi, Yasutomo
Format: Article
Language:English
Subjects:
Image recognition
Machine vision
Object detection
Object recognition
open-set
Predictive models
Protocols
Robots
Scene graph generation
Training
Training data
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Scene graph generation (SGG) aims to detect objects and their relationships in an image, thereby enabling a detailed understanding of a complex scene for various real-world applications. In SGG applications such as robot vision, it is important to correctly detect all objects without recognizing any object as another kind of object or ignoring it. However, previous studies on SGG do not consider unknown objects whose classes are unseen in training. Consequently, current SGG methods wrongly classify them as known object classes or overlook them. In this paper, we propose a new problem named "open-set SGG" with unknown objects, focusing on detecting even unknown objects and their relationships. Specifically, we formally define this new problem and propose an evaluation protocol, including an extended dataset with unknown objects and novel evaluation metrics designed for the open-set setting. We also build baseline methods by employing and extending existing SGG methods and compare them through experiments to establish the current baseline performance of open-set SGG. Finally, we discuss the limitations of the current SGG methodology in the open-set setting and point out future research directions.
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2022.3145465