Unifying 3D Representation and Control of Diverse Robots with a Single Camera

Mirroring the complex structures and diverse functions of natural organisms is a long-standing challenge in robotics. Modern fabrication techniques have dramatically expanded feasible hardware, yet deploying these systems requires control software to translate desired motions into actuator commands....

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-07
Main Authors: Sizhe Lester Li, Zhang, Annan, Chen, Boyuan, Matusik, Hanna, Liu, Chao, Rus, Daniela, Sitzmann, Vincent
Format: Article
Language:English
Subjects:
Actuation
Actuators
Automation
Biomimetics
Cameras
Closed loops
Control systems
Feedback control
Industrial robots
Manufacturing engineering
Material properties
Robot arms
Robot control
Robotics
Robots
Three dimensional models
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Mirroring the complex structures and diverse functions of natural organisms is a long-standing challenge in robotics. Modern fabrication techniques have dramatically expanded feasible hardware, yet deploying these systems requires control software to translate desired motions into actuator commands. While conventional robots can easily be modeled as rigid links connected via joints, it remains an open challenge to model and control bio-inspired robots that are often multi-material or soft, lack sensing capabilities, and may change their material properties with use. Here, we introduce Neural Jacobian Fields, an architecture that autonomously learns to model and control robots from vision alone. Our approach makes no assumptions about the robot's materials, actuation, or sensing, requires only a single camera for control, and learns to control the robot without expert intervention by observing the execution of random commands. We demonstrate our method on a diverse set of robot manipulators, varying in actuation, materials, fabrication, and cost. Our approach achieves accurate closed-loop control and recovers the causal dynamic structure of each robot. By enabling robot control with a generic camera as the only sensor, we anticipate our work will dramatically broaden the design space of robotic systems and serve as a starting point for lowering the barrier to robotic automation.
ISSN:2331-8422