Development of an immersive SLAM-based VR system for teleoperation of a mobile manipulator in an unknown environment

• Published in: Computers in industry 2021-11, Vol.132, p.103502, Article 103502
• Main Authors: Kuo, Chen-Yu, Huang, Chun-Chi, Tsai, Chih-Hsuan, Shi, Yun-Shuo, Smith, Shana
• Format: Article
• Language: English
• Subjects: Immersive; Remote mobile manipulator; SLAM; Virtual reality; Visual aids
• ISSN: 0166-3615; 1872-6194
• DOI: 10.1016/j.compind.2021.103502

•A SLAM-based VR system is developed for teleoperation of a mobile manipulator.•VR visual aids are used to show the direction and depth information.•The grasping error of the SLAM-based VR teleoperation is 0.5 cm.•The placing threshold of the SLAM-based VR teleoperation is 1.5 cm. Operating a remote robot in an unknown environment is a challenge. Usually, a camera is attached to the robot to capture the images of the remote scenes. However, it is difficult to perceive a 3D environment using a 2D display. In this study, a real-time simultaneous localization and mapping (SLAM)-based virtual reality (VR) 3D human-machine interaction system is developed to provide users with immersive telepresence to better operate a remote mobile manipulator in an unknown environment. An RGB-depth camera is mounted on the mobile platform to scan and reconstruct the remote environment in real time. A head mounted display is used to display the 3D reconstructed environment, with a virtual copy of the actual robot in it, to provide users the feeling of immersion at the remote site. Visual aids are added in the 3D reconstructed environment to guide users with the direction and distance information. Users can control the remote robot by manipulating the virtual copy of the actual robot in the VR environment, without being constrained in the robot first person view. The system is compared with a video-based teleoperation and a direct on-site operation. Experimental results show that the grasping error and placing threshold of the SLAM-based VR teleoperation are 0.5 cm and 1.5 cm, respectively. The average error of the reconstructed environment is less than 4%. Within the accuracy of the reconstructed environment, the SLAM-based VR teleoperation has a higher grasping success rate and lower operation time, compared to the video-based teleoperation. This study shows a promising application of immersive SLAM-based VR in interactive telerobotic operations in a remote unknown environment.