Video anomaly detection and localization via Gaussian Mixture Fully Convolutional Variational Autoencoder

• Published in: Computer vision and image understanding 2020-06, Vol.195, p.102920, Article 102920
• Main Authors: Fan, Yaxiang, Wen, Gongjian, Li, Deren, Qiu, Shaohua, Levine, Martin D., Xiao, Fei
• Format: Article
• Language: English
• Subjects: Anomaly detection; Dynamic flow; Gaussian mixture model; Two-stream network; Variational autoencoder; Video surveillance
• ISSN: 1077-3142; 1090-235X
• DOI: 10.1016/j.cviu.2020.102920

We present a novel end-to-end partially supervised deep learning approach for video anomaly detection and localization using only normal samples. The insight that motivates this study is that the normal samples can be associated with at least one Gaussian component of a Gaussian Mixture Model (GMM), while anomalies either do not belong to any Gaussian component. The method is based on Gaussian Mixture Variational Autoencoder, which can learn feature representations of the normal samples as a Gaussian Mixture Model trained using deep learning. A Fully Convolutional Network (FCN) that does not contain a fully-connected layer is employed for the encoder–decoder structure to preserve relative spatial coordinates between the input image and the output feature map. Based on the joint probabilities of each of the Gaussian mixture components, we introduce a sample energy based method to score the anomaly of image test patches. A two-stream network framework is employed to combine the appearance and motion anomalies, using RGB frames for the former and dynamic flow images, for the latter. We test our approach on two popular benchmarks (UCSD Dataset and Avenue Dataset). The experimental results verify the superiority of our method compared to the state of the art. •An end-to-end VAE framework has been considered for video anomaly detection.•A two-stream network employs dynamic flows for detecting the motion anomalies.•Sample energy method is applied to detect anomalies with probabilities of the GMM.•Experiment result on two public datasets demonstrate the superiority of our method.