End-to-end nonprehensile rearrangement with deep reinforcement learning and simulation-to-reality transfer

• Published in: Robotics and autonomous systems 2019-09, Vol.119, p.119-134
• Main Authors: Yuan, Weihao, Hang, Kaiyu, Kragic, Danica, Wang, Michael Y., Stork, Johannes A.
• Format: Article
• Language: English
• Subjects: Deep reinforcement learning; Nonprehensile rearrangement; Transfer learning
• ISSN: 0921-8890; 1872-793X; 1872-793X
• DOI: 10.1016/j.robot.2019.06.007

Nonprehensile rearrangement is the problem of controlling a robot to interact with objects through pushing actions in order to reconfigure the objects into a predefined goal pose. In this work, we rearrange one object at a time in an environment with obstacles using an end-to-end policy that maps raw pixels as visual input to control actions without any form of engineered feature extraction. To reduce the amount of training data that needs to be collected using a real robot, we propose a simulation-to-reality transfer approach. In the first step, we model the nonprehensile rearrangement task in simulation and use deep reinforcement learning to learn a suitable rearrangement policy, which requires in the order of hundreds of thousands of example actions for training. Thereafter, we collect a small dataset of only 70 episodes of real-world actions as supervised examples for adapting the learned rearrangement policy to real-world input data. In this process, we make use of newly proposed strategies for improving the reinforcement learning process, such as heuristic exploration and the curation of a balanced set of experiences. We evaluate our method in both simulation and real setting using a Baxter robot to show that the proposed approach can effectively improve the training process in simulation, as well as efficiently adapt the learned policy to the real world application, even when the camera pose is different from simulation. Additionally, we show that the learned system not only can provide adaptive behavior to handle unforeseen events during executions, such as distraction objects, sudden changes in positions of the objects, and obstacles, but also can deal with obstacle shapes that were not present in the training process. •Deep reinforcement learning allows a robot to solve non-prehensile rearrangement.•The policy can be trained in simulation and transferred to the real world.•Task-specific informed exploration assists off-policy learning.•A curated replay memory provides more informative gradient for Q-learning.•Success rated are 85% and 81% in simulation and reality for random scenes.