EF-Net: A novel enhancement and fusion network for RGB-D saliency detection

• Published in: Pattern recognition 2021-04, Vol.112, p.107740, Article 107740
• Main Authors: Chen, Qian, Fu, Keren, Liu, Ze, Chen, Geng, Du, Hongwei, Qiu, Bensheng, Shao, Ling
• Format: Article
• Language: English
• Subjects: Depth enhancement; Feature fusion; RGB-D image; Salient object detection
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2020.107740

•We propose a novel enhancement-and-fusion framework for effective RGB-D saliency detection.•We propose a novel depth enhancement model to improve the quality of depth maps and an effective layer-wise aggregation module to fuse the features extracted from RGB images and enhanced depth maps.•Extensive experiments on five RGB-D benchmark datasets demonstrate that our method outperforms 12 state-of-the-art saliency detection methods by a large margin. Salient object detection (SOD) has gained tremendous attention in the field of computer vision. Multi-modal SOD based on the complementary information from RGB images and depth maps has shown remarkable success, making RGB-D saliency detection an active research topic. In this paper, we propose a novel multi-modal enhancement and fusion network (EF-Net) for effective RGB-D saliency detection. Specifically, we first utilize a color hint map module with RGB images to predict a hint map, which encodes the coarse information of salient objects. The resulting hint map is then utilized to enhance the depth map with our depth enhancement module, which suppresses the noise and sharpens the object boundary. Finally, we propose an effective layer-wise aggregation module to fuse the features extracted from the enhanced depth maps and RGB images for the accurate detection of salient objects. Our EF-Net utilizes an enhancement-and-fusion framework for saliency detection, which makes full use of the information from RGB images and depth maps. In addition, our depth enhancement module effectively resolves the low-quality issue of depth maps, which boosts the saliency detection performance remarkably. Extensive experiments on five widely-used benchmark datasets demonstrate that our method outperforms 12 state-of-the-art RGB-D saliency detection approaches in terms of five key evaluation metrics.