ReplanVLM: Replanning Robotic Tasks With Visual Language Models

Large language models (LLMs) have gained increasing popularity in robotic task planning due to their exceptional abilities in text analytics and generation, as well as their broad knowledge of the world. However, they fall short in decoding visual cues. LLMs have limited direct perception of the wor...

Full description

Saved in:
Bibliographic Details
Published in:IEEE robotics and automation letters 2024-11, Vol.9 (11), p.10201-10208
Main Authors: Mei, Aoran, Zhu, Guo-Niu, Zhang, Huaxiang, Gan, Zhongxue
Format: Article
Language:English
Subjects:
Adaptation models
AI-enabled robotics
Artificial intelligence
Chatbots
Computational modeling
Decoding
Error analysis
Error correction
Error correction & detection
Error correction codes
Grasping (robotics)
human-robot collaboration
Human-robot interaction
Large language models
Planning
Predictive models
Robotics
task planning
Task planning (robotics)
visual language models
Visual perception
Visual tasks
Visualization
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Large language models (LLMs) have gained increasing popularity in robotic task planning due to their exceptional abilities in text analytics and generation, as well as their broad knowledge of the world. However, they fall short in decoding visual cues. LLMs have limited direct perception of the world, which leads to a deficient grasp of the current state of the world. By contrast, the emergence of visual language models (VLMs) fills this gap by integrating visual perception modules, which can enhance the autonomy of robotic task planning. Despite these advancements, VLMs still face challenges, such as the potential for task execution errors, even when provided with accurate instructions. To address such issues, this letter proposes a ReplanVLM framework for robotic task planning. In this study, we focus on error correction interventions. An internal error correction mechanism and an external error correction mechanism are presented to correct errors under corresponding phases. A replan strategy is developed to replan tasks or correct error codes when task execution fails. Experimental results on real robots and in simulation environments have demonstrated the superiority of the proposed framework, with higher success rates and robust error correction capabilities in open-world tasks.
ISSN:2377-3766
2377-3766
DOI:10.1109/LRA.2024.3471457