Learning a simulation-based visual policy for real-world peg in unseen holes

• Published in: Review of scientific instruments 2023-10, Vol.94 (10)
• Main Authors: Xie, Liang, Yu, Hongxiang, Xu, Kechun, Yang, Tong, Wang, Minhang, Lu, Haojian, Xiong, Rong, Wang, Yue
• Format: Article
• Language: English
• Subjects: Electric vehicle charging; Image segmentation; Learning; Modules; Scientific apparatus & instruments; Simulation; Virtual sensors
• ISSN: 0034-6748; 1089-7623
• DOI: 10.1063/5.0168544

This paper proposes a learning-based visual peg-in-hole that enables training with several shapes in simulation and adapting to arbitrary unseen shapes in the real world with minimal sim-to-real cost. The core idea is to decouple the generalization of the sensory-motor policy from the design of a fast-adaptable perception module and a simulated generic policy module. The framework consists of a segmentation network (SN), a virtual sensor network (VSN), and a controller network (CN). Concretely, the VSN is trained to measure the pose of the unseen shape from a segmented image. After that, given the shape-agnostic pose measurement, the CN is trained to achieve a generic peg-in-hole. Finally, when applying to real unseen holes, we only have to fine-tune the SN required by the simulated VSN + CN. To further minimize the transfer cost, we propose to automatically collect and annotate the data for the SN after one-minute human teaching. Simulated and real-world results are presented under the configuration of eye-to/in-hand. An electric vehicle charging system with the proposed policy inside achieves a 10/10 success rate in 2–3 s, using only hundreds of auto-labeled samples for the SN transfer.