Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns

• Published in: IEEE transactions on pattern analysis and machine intelligence 2021-02, Vol.43 (2), p.516-531
• Main Authors: Xie, Jianwen, Zhu, Song-Chun, Wu, Ying Nian
• Format: Article
• Language: English
• Subjects: Algorithms; Data models; Deep generative models; Domains; dynamic textures; Dynamics; energy-based models; generative ConvNets; Generators; Heuristic algorithms; Machine learning; Solid modeling; spatial-temporal ConvNets; Synthesis; Three-dimensional displays; Training; Video sequences
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2019.2934852

Video sequences contain rich dynamic patterns, such as dynamic texture patterns that exhibit stationarity in the temporal domain, and action patterns that are non-stationary in either spatial or temporal domain. We show that an energy-based spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns. The model defines a probability distribution on the video sequence, and the log probability is defined by a spatial-temporal ConvNet that consists of multiple layers of spatial-temporal filters to capture spatial-temporal patterns of different scales. The model can be learned from the training video sequences by an "analysis by synthesis" learning algorithm that iterates the following two steps. Step 1 synthesizes video sequences from the currently learned model. Step 2 then updates the model parameters based on the difference between the synthesized video sequences and the observed training sequences. We show that the learning algorithm can synthesize realistic dynamic patterns. We also show that it is possible to learn the model from incomplete training sequences with either occluded pixels or missing frames, so that model learning and pattern completion can be accomplished simultaneously.