Space-time image sequence analysis: object tunnels and occlusion volumes

• Published in: IEEE transactions on image processing 2006-02, Vol.15 (2), p.364-376
• Main Authors: Ristivojevic, M., Konrad, J.
• Format: Article
• Language: English
• Subjects: Active contours; Active surfaces; Algorithms; Applied sciences; Artificial Intelligence; Competition; Exact sciences and technology; Frames; Image analysis; Image Enhancement - methods; Image Interpretation, Computer-Assisted - methods; Image processing; Image segmentation; Image sequence analysis; Image sequences; Imaging, Three-Dimensional - methods; Information Storage and Retrieval - methods; Information, signal and communications theory; level set methods; Mathematical models; Motion; Motion detection; Motion estimation; Object detection; Occlusion; Pattern Recognition, Automated - methods; Performance analysis; Segmentation; Signal processing; Spatiotemporal phenomena; Studies; Subtraction Technique; Surface chemistry; Telecommunications and information theory; Time Factors; Tunnels (transportation); Video Recording - methods; video segmentation; volume competition
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2005.860616

We address the issue of image sequence analysis jointly in space and time. While typical approaches to such an analysis consider two image frames at a time, we propose to perform this analysis jointly over multiple frames. We concentrate on spatiotemporal segmentation of image sequences and on analysis of occlusion effects therein. The segmentation process is three-dimensional (3-D); we search for a volume carved out by each moving object in the image sequence domain, or "object tunnel", a new space-time concept. We pose the problem in variational framework by using only motion information (no intensity edges). The resulting formulation can be viewed as volume competition, a 3-D generalization of region competition. We parameterize the unknown surface to be estimated, but rather than using an active-surface approach, we embed it into a higher dimensional function and apply the level-set methodology. We first develop simple models for the detection of moving objects over static background; no motion models are needed. Then, in order to improve segmentation accuracy, we incorporate motion models for objects and background. We further extend the method by including explicit models for occluded and newly exposed areas that lead to "occlusion volumes," another new space-time concept. Since, in this case, multiple volumes are sought, we apply a multiphase variant of the level-set method. We present various experimental results for synthetic and natural image sequences.