Selecting Salient Frames for Spatiotemporal Video Modeling and Segmentation

• Published in: IEEE transactions on image processing 2007-12, Vol.16 (12), p.3035-3046
• Main Authors: Song, Xiaomu, Fan, Guoliang
• Format: Article
• Language: English
• Subjects: Algorithms; Applied sciences; Artificial Intelligence; Computer Simulation; Data Interpretation, Statistical; Data mining; Engineering profession; Exact sciences and technology; Expectation maximization (EM); feature selection; frame saliency; Frames; Gaussian; Gaussian mixture models (GMMs); Image Enhancement - methods; Image Interpretation, Computer-Assisted - methods; Image processing; Image segmentation; Information, signal and communications theory; Layout; Mathematical models; Maximization; Medical services; Military computing; Miscellaneous; Models, Statistical; Object segmentation; Pattern Recognition, Automated - methods; Reproducibility of Results; Segmentation; Segments; Sensitivity and Specificity; Signal and communications theory; Signal processing; Signal representation. Spectral analysis; Signal, noise; Spatiotemporal phenomena; statistical video modeling; Studies; Subtraction Technique; Tasks; Telecommunications and information theory; Training; Video Recording - methods; video segmentation; Video sequences; Video sharing
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2007.908283

We propose a new statistical generative model for spatiotemporal video segmentation. The objective is to partition a video sequence into homogeneous segments that can be used as "building blocks" for semantic video segmentation. The baseline framework is a Gaussian mixture model (GMM)-based video modeling approach that involves a six-dimensional spatiotemporal feature space. Specifically, we introduce the concept of frame saliency to quantify the relevancy of a video frame to the GMM-based spatiotemporal video modeling. This helps us use a small set of salient frames to facilitate the model training by reducing data redundancy and irrelevance. A modified expectation maximization algorithm is developed for simultaneous GMM training and frame saliency estimation, and the frames with the highest saliency values are extracted to refine the GMM estimation for video segmentation. Moreover, it is interesting to find that frame saliency can imply some object behaviors. This makes the proposed method also applicable to other frame-related video analysis tasks, such as key-frame extraction, video skimming, etc. Experiments on real videos demonstrate the effectiveness and efficiency of the proposed method.