System of Robot Learning from Multi-Modal Demonstration and Natural Language Instruction

Collaborative robots are set to play an important role in the future of the manufacturing industry. They need to be able to work outside of the fencing and perform new tasks to individual customer specifications. The necessity of frequent robot re-programming is a great challenge for small and mediu...

Full description

Saved in:
Bibliographic Details
Published in:Procedia CIRP 2022, Vol.107, p.914-919
Main Authors: Lu, Shuang, Berger, Julia, Schilp, Johannes
Format: Article
Language:English
Subjects:
Cognitive robotics
Human robot collaboration
Sensor
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Collaborative robots are set to play an important role in the future of the manufacturing industry. They need to be able to work outside of the fencing and perform new tasks to individual customer specifications. The necessity of frequent robot re-programming is a great challenge for small and medium sized companies alike. Learning from demonstration is a promising approach that aims to enable robots to acquire from their end users new task knowledge consisting of a sequence of actions, the associated skills, and the context in which the task is executed. Current systems have limited support for integrating semantics and environmental changes. This paper introduces a system combining several modalities as demonstration interfaces, including natural language instruction, visual observation and hand-guiding, which enables the robot to learn a task comprising a goal concept, a plan and basic actions, with consideration for the current environment state. The task thus learned can then be generalized to similar tasks involving different initial and goal states.
ISSN:2212-8271
2212-8271
DOI:10.1016/j.procir.2022.05.084