TSDNN: tube sorting with deep neural networks for surveillance video synopsis

• Published in: Multimedia tools and applications 2024-01, Vol.83 (24), p.65059-65076
• Main Authors: Wang, Chenwu, Wu, Junsheng, Wang, Pei, Chen, Hao, Zhu, Zhixiang
• Format: Article
• Language: English
• Subjects: Artificial neural networks; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Deep learning; Guards; Machine learning; Multimedia Information Systems; Neural networks; Special Purpose and Application-Based Systems; Surveillance; Tubes; Unsupervised learning
• ISSN: 1573-7721; 1380-7501; 1573-7721
• DOI: 10.1007/s11042-023-18091-x

High-quality cameras collect a large amount of surveillance video that can be labor-consuming for security guards to browse and analyze. One way to ease the browsing burden is to condense a long surveillance video to a much shorter clip with the technology of video synopsis. This paper introduces deep neural networks to sort object tubes for video synopsis. To the best of our knowledge, this first deep learning-based approach for video synopsis. Our approach first estimates the static background and separates foreground objects from the video. The object tubes are then obtained after stacking the foreground areas of the same object. The tubes are then represented by deep features with a 3D CNN. Finally, a Transformer sorts the object tubes and gives the final locations of all tubes. Since our approach combines tube representation extraction with neural networks, we call our approach Tube Sorting with Deep Neural Networks (TSDNN). In addition, we optimize the network with unsupervised learning that utilizes activity, collision, and chronological losses. Experiments demonstrate that the proposed TSDNN produces condensed video with few collision and chronological disorder artifacts.