Temporal Sequence Learning, Prediction, and Control: A Review of Different Models and Their Relation to Biological Mechanisms

• Published in: Neural computation 2005-02, Vol.17 (2), p.245-319
• Main Authors: Wörgötter, Florentin, Porr, Bernd
• Format: Article
• Language: English
• Subjects: Algorithms; Applied sciences; Artificial intelligence; Biological and medical sciences; Brain; Computer science; control theory; systems; Exact sciences and technology; Forecasting; Fundamental and applied biological sciences. Psychology; General aspects; Learning; Learning and adaptive systems; Mathematics; Mathematics in biology. Statistical analysis. Models. Metrology. Data processing in biology (general aspects); Neural networks; Neural Networks (Computer); Neurology; Probability and statistics; Probability theory and stochastic processes; Review; Sciences and techniques of general use; Special processes (renewal theory, markov renewal processes, semi-markov processes, statistical mechanics type models, applications); Time Factors
• ISSN: 0899-7667; 1530-888X
• DOI: 10.1162/0899766053011555

In this review, we compare methods for temporal sequence learning (TSL) across the disciplines machine-control, classical conditioning, neuronal models for TSL as well as spike-timing-dependent plasticity (STDP). This review introduces the most influential models and focuses on two questions: To what degree are reward-based (e.g., TD learning) and correlation-based (Hebbian) learning related? and How do the different models correspond to possibly underlying biological mechanisms of synaptic plasticity? We first compare the different models in an open-loop condition, where behavioral feedback does not alter the learning. Here we observe that reward-based and correlation-based learning are indeed very similar. Machine control is then used to introduce the problem of closed-loop control (e.g., actor-critic architectures). Here the problem of evaluative (rewards) versus nonevaluative (correlations) feedback from the environment will be discussed, showing that both learning approaches are fundamentally different in the closed-loop condition. In trying to answer the second question, we compare neuronal versions of the different learning architectures to the anatomy of the involved brain structures (basal-ganglia, thalamus, and cortex) and the molecular biophysics of glutamatergic and dopaminergic synapses. Finally, we discuss the different algorithms used to model STDP and compare them to reward-based learning rules. Certain similarities are found in spite of the strongly different timescales. Here we focus on the biophysics of the different calcium-release mechanisms known to be involved in STDP.