Supervised Autonomy for Remote Teleoperation of Hybrid Wheel-Legged Mobile Manipulator Robots

• Main Authors: Cheong, Samuel, Pang Chen, Tai, Acar, Cihan, You, Yangwei, Chen, Yuda, Leong Sim, Wan, Peng Tee, Keng
• Format: Conference Proceeding
• Language: English
• Subjects: Manipulators; Mobile robots; Robot kinematics; System integration; Three-dimensional displays; User interfaces; Visualization
• ISSN: 2153-0866
• DOI: 10.1109/IROS51168.2021.9635997

This paper proposes an improved supervised autonomy framework for remote teleoperation of a quadrupedal bimanual mobile manipulator in an unknown environment, with the usage of advanced perception technology and allowing the operator to easily assist the robot with decision making for executing tasks on the fly. First, the perception system uses lightweight deep neural network-based Single Shot Detector (SSD) MobileNet on RGB images to detect objects and highlight them to the human operator via an intuitive interactive visualization interface. After object and action selections are made by the operator, segmentation of object point cloud and 3D surfaces based on random sample consensus is performed, followed by object pose localization by using keypoint extraction. Based on the localized object, mobile manipulation motion to perform the operator-selected action is planned and executed with the help of a state estimator for the hybrid wheel-legged robot. Thanks to the autonomy of the robot in perception and manipulation, the complexity of teleoperating the robot is reduced to specifying the essential task objectives. Experimental results on the real robot, with full system integration, for 2 task scenarios, namely passage clearing and object retrieval, demonstrate a high average success rate of 92.2% over a total of 90 trials.