Long-term Action Forecasting Using Multi-headed Attention-based Variational Recurrent Neural Networks

Systems developed for predicting both the action and the amount of time someone might take to perform that action need to be aware of the inherent uncertainty in what humans do. Here, we present a novel hybrid generative model for action anticipation that attempts to capture the uncertainty in human...

Full description

Saved in:
Bibliographic Details
Main Authors: Loh, Siyuan Brandon, Roy, Debaditya, Fernando, Basura
Format: Conference Proceeding
Language:English
Subjects:
Computer vision
Conferences
Hybrid power systems
Predictive models
Recurrent neural networks
Training
Uncertainty
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Systems developed for predicting both the action and the amount of time someone might take to perform that action need to be aware of the inherent uncertainty in what humans do. Here, we present a novel hybrid generative model for action anticipation that attempts to capture the uncertainty in human actions. Our model uses a multi-headed attention-based variational generative model for action prediction (MAVAP), and Gaussian log-likelihood maximization to predict the corresponding action's duration. During training, we optimise three losses: a variational loss, a negative log-likelihood loss, and a discriminative cross-entropy loss. We evaluate our model on benchmark datasets (i.e., Breakfast and 50Salads) for action forecasting tasks and demonstrate improvements over prior methods using both ground truth observations and predicted features from an action segmentation network (i.e., MS-TCN++). We also show that factorizing the latent space across multiple Gaussian heads predicts better plausible future action sequences compared to a single Gaussian.
ISSN:2160-7516
DOI:10.1109/CVPRW56347.2022.00270