An Object-Based Approach to Image/Video-Based Synthesis and Processing for 3-D and Multiview Televisions

• Published in: IEEE transactions on circuits and systems for video technology 2009-06, Vol.19 (6), p.821-831
• Main Authors: CHAN, Shing-Chow, GAN, Zhi-Feng, NG, King-To, HO, Ka-Leung, SHUM, Heung-Yeung
• Format: Article
• Language: English
• Subjects: Algorithms; Applied sciences; Arrays; Broadcasting. Videocommunications. Audiovisual; Cameras; Detection, estimation, filtering, equalization, prediction; Dynamic image-based representations; Dynamical systems; Dynamics; Electronics; Exact sciences and technology; Image processing; Image segmentation; Image sequences; image-based rendering (IBR); Imaging devices; Information, signal and communications theory; Layout; object-based; plenoptic videos; Rendering; Rendering (computer graphics); Resilience; Scalability; Segmentation; Semiconductor electronics. Microelectronics. Optoelectronics. Solid state devices; Shape; Signal and communications theory; Signal processing; Signal, noise; Studies; Telecommunications; Telecommunications and information theory; Television; Tracking; Video sequences
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2009.2017302

This paper proposes an object-based approach to a class of dynamic image-based representations called ldquoplenoptic videos,rdquo where the plenoptic video sequences are segmented into image-based rendering (IBR) objects each with its image sequence, depth map, and other relevant information such as shape and alpha information. This allows desirable functionalities such as scalability of contents, error resilience, and interactivity with individual IBR objects to be supported. Moreover, the rendering quality in scenes with large depth variations can also be improved considerably. A portable capturing system consisting of two linear camera arrays was developed to verify the proposed approach. An important step in the object-based approach is to segment the objects in video streams into layers or IBR objects. To reduce the time for segmenting plenoptic videos under the semiautomatic technique, a new object tracking method based on the level-set method is proposed. Due to possible segmentation errors around object boundaries, natural matting with Bayesian approach is also incorporated into our system. Furthermore, extensions of conventional image processing algorithms to these IBR objects are studied and illustrated with examples. Experimental results are given to illustrate the efficiency of the tracking, matting, rendering, and processing algorithms under the proposed object-based framework.