Learning 3D Semantic Scene Graphs with Instance Embeddings

A 3D scene is more than the geometry and classes of the objects it comprises. An essential aspect beyond object-level perception is the scene context, described as a dense semantic network of interconnected nodes. Scene graphs have become a common representation to encode the semantic richness of im...

Full description

Saved in:
Bibliographic Details
Published in:International journal of computer vision 2022-03, Vol.130 (3), p.630-651
Main Authors: Wald, Johanna, Navab, Nassir, Tombari, Federico
Format: Article
Language:English
Subjects:
Artificial Intelligence
Computer architecture
Computer Imaging
Computer Science
Graph theory
Graphs
Image processing
Image Processing and Computer Vision
Image segmentation
Neural networks
Nodes
Pattern Recognition
Pattern Recognition and Graphics
Perception
Representations
Semantics
Special Issue on 3D Computer Vision
Vision
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:A 3D scene is more than the geometry and classes of the objects it comprises. An essential aspect beyond object-level perception is the scene context, described as a dense semantic network of interconnected nodes. Scene graphs have become a common representation to encode the semantic richness of images, where nodes in the graph are object entities connected by edges, so-called relationships. Such graphs have been shown to be useful in achieving state-of-the-art performance in image captioning, visual question answering and image generation or editing. While scene graph prediction methods so far focused on images, we propose instead a novel neural network architecture for 3D data, where the aim is to learn to regress semantic graphs from a given 3D scene. With this work, we go beyond object-level perception, by exploring relations between object entities. Our method learns instance embeddings alongside a scene segmentation and is able to predict semantics for object nodes and edges. We leverage 3DSSG , a large scale dataset based on 3RScan that features scene graphs of changing 3D scenes. Finally, we show the effectiveness of graphs as an intermediate representation on a retrieval task.
ISSN:0920-5691
1573-1405
DOI:10.1007/s11263-021-01546-9