Learning Continuous-Time Dynamics With Attention

Learning the hidden dynamics from sequence data is crucial. Attention mechanism can be introduced to spotlight on the region of interest for sequential learning. Traditional attention was measured between a query and a sequence based on a discrete-time state trajectory. Such a mechanism could not ch...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on pattern analysis and machine intelligence 2023-02, Vol.45 (2), p.1906-1918
Main Authors: Chien, Jen-Tzung, Chen, Yi-Hsiang
Format: Article
Language:English
Subjects:
causal self attention
Coders
Computational modeling
continuous-time attention
Costs
Data models
Differential equations
Dynamic structural analysis
Dynamics
Emotion recognition
Encoders-Decoders
Machine learning
Mathematical models
neural differential equation
Ordinary differential equations
Queries
Recurrent network
Recurrent neural networks
stochastic machine
Trajectory
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Learning the hidden dynamics from sequence data is crucial. Attention mechanism can be introduced to spotlight on the region of interest for sequential learning. Traditional attention was measured between a query and a sequence based on a discrete-time state trajectory. Such a mechanism could not characterize the irregularly-sampled sequence data. This paper presents an attentive differential network (ADN) where the attention over continuous-time dynamics is developed. The continuous-time attention is performed over the dynamics at all time. The missing information in irregular or sparse samples can be seamlessly compensated and attended. Self attention is computed to find the attended state trajectory. However, the memory cost for attention score between a query and a sequence is demanding since self attention treats all time instants as query points in an ordinary differential equation solver. This issue is tackled by imposing the causality constraint in causal ADN (CADN) where the query is merged up to current time. To enhance the model robustness, this study further explores a latent CADN where the attended dynamics are calculated in an encoder-decoder structure via Bayesian learning. Experiments on the irregularly-sampled actions, dialogues and bio-signals illustrate the merits of the proposed methods in action recognition, emotion recognition and mortality prediction, respectively.
ISSN:0162-8828
1939-3539
2160-9292
DOI:10.1109/TPAMI.2022.3162711