Robot Unknown Objects Instance Segmentation Based on Collaborative Weight Assignment RGB-Depth Fusion Strategy

Unknown objects instance-aware segmentation (UOIS) is crucial for the operation of autonomous robots, especially in unstructured scenes with unknown objects. As the primary data source types for robots, RGB and Depth are not fully exploited by existing studies due to the inherent information differe...

Full description

Saved in:
Bibliographic Details
Published in:IEEE/ASME transactions on mechatronics 2024-06, Vol.29 (3), p.2032-2043
Main Authors: Tian, Hongkun, Song, Kechen, Tong, Ling, Man, Yi, Yan, Yunhui
Format: Article
Language:English
Subjects:
Collaboration
Convolutional neural networks
Feature extraction
Grasping
Instance segmentation
Instance-aware segmentation
Mechatronics
Modules
Qualitative analysis
RGB-D fusion
robotic operation
Robots
Semantics
Strategy
unknown objects
Weight measurement
weight shared strategy
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Unknown objects instance-aware segmentation (UOIS) is crucial for the operation of autonomous robots, especially in unstructured scenes with unknown objects. As the primary data source types for robots, RGB and Depth are not fully exploited by existing studies due to the inherent information differences between RGB (2-D appearance) and Depth (3-D geometry). Therefore, it is challenging to fully exploit the features of both modalities to achieve segmentation for instances of unknown objects. This article proposes a collaborative weight assignment (CWA) fusion strategy for fusing RGB and Depth (RGB-D). It contains three carefully designed modules, motivational pixel weight assignment(MPWA) module, dual-direction spatial weight assignment (DSWA) module, and stepwise global feature aggregation (SGFA) module. Our method aims to adaptively assign fusion weights between two modalities to exploit RGB-D features from multiple dimensions better. On the popular Graspnet-1Bilion and WISDOM RGB-D robot operation datasets, the proposed method achieves competitive performance with state-of-the-art techniques, proving our approach can make good use of the information between the two modalities. Furthermore, we have deployed the fusion model on the AUBO i5 robotic manipulation platform to test its segmentation and grasping optimization effects oriented to unknown objects. The proposed method achieves robust performance through qualitative and quantitative analysis experiments.
ISSN:1083-4435
1941-014X
DOI:10.1109/TMECH.2023.3327865