This&That: Language-Gesture Controlled Video Generation for Robot Planning

We propose a robot learning method for communicating, planning, and executing a wide range of tasks, dubbed This&That. We achieve robot planning for general tasks by leveraging the power of video generative models trained on internet-scale data containing rich physical and semantic context. In t...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-07
Main Authors: Wang, Boyang, Sridhar, Nikhil, Chao, Feng, Van der Merwe, Mark, Fishman, Adam, Fazeli, Nima, Park, Jeong Joon
Format: Article
Language:English
Subjects:
Cloning
Communication
Controllability
Robots
Task planning (robotics)
Translating
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We propose a robot learning method for communicating, planning, and executing a wide range of tasks, dubbed This&That. We achieve robot planning for general tasks by leveraging the power of video generative models trained on internet-scale data containing rich physical and semantic context. In this work, we tackle three fundamental challenges in video-based planning: 1) unambiguous task communication with simple human instructions, 2) controllable video generation that respects user intents, and 3) translating visual planning into robot actions. We propose language-gesture conditioning to generate videos, which is both simpler and clearer than existing language-only methods, especially in complex and uncertain environments. We then suggest a behavioral cloning design that seamlessly incorporates the video plans. This&That demonstrates state-of-the-art effectiveness in addressing the above three challenges, and justifies the use of video generation as an intermediate representation for generalizable task planning and execution. Project website: https://cfeng16.github.io/this-and-that/.
ISSN:2331-8422