RDBN: Visual relationship detection with inaccurate RGB-D images

Traditional visual relationship detection methods only use RGB information to train the semantic network, which do not match human habits that we combine RGB information with Depth information to perceive the world, thus, there is not enough generalization ability (zero-shot performance) to extract...

Full description

Saved in:
Bibliographic Details
Published in:Knowledge-based systems 2020-09, Vol.204, p.106142, Article 106142
Main Authors: Liu, Xiaozhou, Gan, Ming-Gang
Format: Article
Language:English
Subjects:
Ablation
Annotations
Color imagery
Datasets
Deep neural network
Image management
Image retrieval
RGB-D image
Semantics
Visual relationship detection
Visual scene understanding
Visual tasks
Zero-shot learning
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Traditional visual relationship detection methods only use RGB information to train the semantic network, which do not match human habits that we combine RGB information with Depth information to perceive the world, thus, there is not enough generalization ability (zero-shot performance) to extract the visual relationships in practical scenes. To solve this problem, a novel visual relationship detection framework based on RGB-D images is proposed in this paper. Since it is difficult to get accurate depth maps from complex scenes, we propose a fuzzy strategy based method to represent Depth features of inaccurate depth maps which are independent of manual depth annotations. In particular, we formulate the RGB-Depth-Balanced-Network (RDBN) which can simultaneously process RGB features and the corresponding estimated depth maps to counter the inaccuracy of depth maps and extract semantic information by the only input of monocular RGB images. In experiments, we conduct ablation experiments to analyze functions of different visual components to demonstrate the effectiveness of our RDBN. Furthermore, we show that RDBN outperforms state-of-the-art visual relationship detection methods on Visual Relationship Dataset (VRD) and UnRel Dataset when tackling the visual relationship detection task of zero-shot learning in specific depth conditions, and the task of image retrieval among unusual relationships.
ISSN:0950-7051
1872-7409
DOI:10.1016/j.knosys.2020.106142