Abnormal Event Detection From Videos Using a Two-Stream Recurrent Variational Autoencoder

• Published in: IEEE transactions on cognitive and developmental systems 2020-03, Vol.12 (1), p.30-42
• Main Authors: Yan, Shiyang, Smith, Jeremy S., Lu, Wenjin, Zhang, Bailing
• Format: Article
• Language: English
• Subjects: Abnormal event detection; convolutional long-short term memory (LSTM); Event detection; Feature extraction; Hidden Markov models; Image reconstruction; Neural networks; reconstruction error probability; Semi-supervised learning; Surveillance; Surveillance systems; Task analysis; two-stream fusion; variational autoencoder (VAE); Video data; Videos
• ISSN: 2379-8920; 2379-8939
• DOI: 10.1109/TCDS.2018.2883368

With the massive deployment of distributed video surveillance systems, the automatic detection of abnormal events in video streams has become an urgent need. An abnormal event can be considered as a deviation from the regular scene; however, the distribution of normal and abnormal events is severely imbalanced, since the abnormal events do not frequently occur. To make use of a large number of video surveillance videos of regular scenes, we propose a semi-supervised learning scheme, which only uses the data that contains the ordinary scenes. The proposed model has a two-stream structure that is composed of the appearance and motion streams. For each stream, a recurrent variational autoencoder can model the probabilistic distribution of the normal data in a semi-supervised learning scheme. The appearance and motion features from the two streams can provide complementary information to describe this probabilistic distribution. Comprehensive experiments validate the effectiveness of our proposed scheme on several public benchmark data sets which include the Avenue, the Ped1, the Ped2, the Subway-entry, and the Subway-exit.