Deep Episodic Memory for Verbalization of Robot Experience

The ability to verbalize robot experience in natural language is key for a symbiotic human-robot interaction. While first works approached this problem using template-based verbalization on symbolic episode data only, we explore a novel way in which deep learning methods are used for the creation of...

Full description

Saved in:
Bibliographic Details
Published in:IEEE robotics and automation letters 2021-07, Vol.6 (3), p.5808-5815
Main Authors: Brmann, Leonard, Peller-Konrad, Fabian, Constantin, Stefan, Asfour, Tamim, Waibel, Alex
Format: Article
Language:English
Subjects:
AI-enabled robotics
Coders
Encoding
Human engineering
Knowledge discovery
Learning from experience
Memory management
natural dialog for HRI
Natural language
Natural languages
Robots
Semantics
sensorimotor learning
Speech encoders
Task analysis
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The ability to verbalize robot experience in natural language is key for a symbiotic human-robot interaction. While first works approached this problem using template-based verbalization on symbolic episode data only, we explore a novel way in which deep learning methods are used for the creation of an episodic memory from experiences as well as the verbalization of such experience in natural language. To this end, we first collected a complex dataset consisting of more than a thousand multimodal robot episode recordings both from simulation as well as real robot executions, together with representative natural language questions and answers about the robot's past experience. Second, we propose and evaluate an episodic memory verbalization model consisting of a speech encoder and decoder based on the Transformer architecture, combined with an LSTM-based episodic memory auto-encoder, and evaluate the model on simulated and real data from robot execution examples. Our experimental results provide a proof-of-concept for episodic-memory-based verbalization of robot experience.
ISSN:2377-3766
2377-3766
DOI:10.1109/LRA.2021.3085166