Model-Based and Model-Free Replay Mechanisms for Reinforcement Learning in Neurorobotics

Experience replay is widely used in AI to bootstrap reinforcement learning (RL) by enabling an agent to remember and reuse past experiences. Classical techniques include shuffled-, reversed-ordered- and prioritized-memory buffers, which have different properties and advantages depending on the natur...

Full description

Saved in:
Bibliographic Details
Published in:Frontiers in neurorobotics 2022-06, Vol.16, p.864380-864380
Main Authors: Massi, Elisa, Barthélemy, Jeanne, Mailly, Juliane, Dromnelle, Rémi, Canitrot, Julien, Poniatowski, Esther, Girard, Benoît, Khamassi, Mehdi
Format: Article
Language:English
Subjects:
Algorithms
Artificial Intelligence
Brain research
Cognitive science
Computational neuroscience
Computer Science
Decomposition
hippocampal replay
Hippocampus
Learning
Life Sciences
Machine Learning
Memory
model-based
model-free
Nervous system
Neurons and Cognition
neurorobotics
Neuroscience
Neurosciences
Psychology and behavior
Reinforcement
reinforcement learning
Robotics
Robots
Simulation
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Experience replay is widely used in AI to bootstrap reinforcement learning (RL) by enabling an agent to remember and reuse past experiences. Classical techniques include shuffled-, reversed-ordered- and prioritized-memory buffers, which have different properties and advantages depending on the nature of the data and problem. Interestingly, recent computational neuroscience work has shown that these techniques are relevant to model hippocampal reactivations recorded during rodent navigation. Nevertheless, the brain mechanisms for orchestrating hippocampal replay are still unclear. In this paper, we present recent neurorobotics research aiming to endow a navigating robot with a neuro-inspired RL architecture (including different learning strategies, such as model-based (MB) and model-free (MF), and different replay techniques). We illustrate through a series of numerical simulations how the specificities of robotic experimentation (e.g., autonomous state decomposition by the robot, noisy perception, state transition uncertainty, non-stationarity) can shed new lights on which replay techniques turn out to be more efficient in different situations. Finally, we close the loop by raising new hypotheses for neuroscience from such robotic models of hippocampal replay.
ISSN:1662-5218
1662-5218
DOI:10.3389/fnbot.2022.864380