KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data

Building generalist robotic systems involves effectively endowing robots with the capabilities to handle novel objects in an open-world setting. Inspired by the advances of large pre-trained models, we propose Keypoint Affordance Learning from Imagined Environments (KALIE), which adapts pre-trained...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-09
Main Authors: Tang, Grace, Rajkumar, Swetha, Zhou, Yifei, Homer Rich Walke, Levine, Sergey, Kuan Fang
Format: Article
Language:English
Subjects:
Automation
Data points
Image manipulation
Image quality
Natural language processing
Predictive control
Robot control
Robotics
Vision
Visual observation
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Building generalist robotic systems involves effectively endowing robots with the capabilities to handle novel objects in an open-world setting. Inspired by the advances of large pre-trained models, we propose Keypoint Affordance Learning from Imagined Environments (KALIE), which adapts pre-trained Vision Language Models (VLMs) for robotic control in a scalable manner. Instead of directly producing motor commands, KALIE controls the robot by predicting point-based affordance representations based on natural language instructions and visual observations of the scene. The VLM is trained on 2D images with affordances labeled by humans, bypassing the need for training data collected on robotic systems. Through an affordance-aware data synthesis pipeline, KALIE automatically creates massive high-quality training data based on limited example data manually collected by humans. We demonstrate that KALIE can learn to robustly solve new manipulation tasks with unseen objects given only 50 example data points. Compared to baselines using pre-trained VLMs, our approach consistently achieves superior performance.
ISSN:2331-8422