M2RNet: Multi-modal and multi-scale refined network for RGB-D salient object detection

• Published in: Pattern recognition 2023-03, Vol.135, Article 109139
• Main Authors: Fang, Xian, Jiang, Mingfeng, Zhu, Jinchao, Shao, Xiuli, Wang, Hongpeng
• Format: Article
• Language: English
• Subjects: Deep learning; Loss function; Multi-modal feature; Multi-scale feature; Saliency detection
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2022.109139

•A nested dual attention module (NDAM) is proposed to explicitly exploit the combined features of RGB and depth flows.•An adjacent interactive aggregation module (AIAM) is proposed to gradually integrate the neighbor features of high, middle and low levels.•A joint hybrid optimization loss (JHOL) is proposed to make the predictions have a prominent outline.•A novel multi-modal and multi-scale refined network (M2RNet) is proposed for salient object detection.•Extensive experiments demonstrate that our method achieves consistently superior performance against 12 state-of-the-art approaches. Salient object detection is a fundamental topic in computer vision, which has promising application prospects. The previous methods based on RGB-D may potentially suffer from the incompatibility of multi-modal feature fusion and the insufficiency of multi-scale feature aggregation. To tackle these two dilemmas, we propose a novel multi-modal and multi-scale refined network (M2RNet). Specifically, three essential components are presented in this network. The nested dual attention module (NDAM) explicitly exploits the combined features of RGB and depth flows. The adjacent interactive aggregation module (AIAM) gradually integrates the neighbor features of high, middle and low levels. The joint hybrid optimization loss (JHOL) makes the predictions have a prominent outline. Extensive experiments quantitatively and qualitatively demonstrate that our method outperforms other state-of-the-art approaches.