Probability-Based Rendering for View Synthesis

• Published in: IEEE transactions on image processing 2014-02, Vol.23 (2), p.870-884
• Main Authors: Bumsub Ham, Dongbo Min, Changjae Oh, Do, Minh N., Kwanghoon Sohn
• Format: Article
• Language: English
• Subjects: Accuracy; Applied sciences; Cameras; correspondence matching; Detection, estimation, filtering, equalization, prediction; Exact sciences and technology; Image fusion; Image processing; Image-based rendering (IBR); Information, signal and communications theory; Probabilistic logic; random walk with restart (RWR); Rendering (computer graphics); Signal and communications theory; Signal processing; Signal, noise; Steady-state; Telecommunications and information theory; Three-dimensional displays; view interpolation
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2013.2295716

In this paper, a probability-based rendering (PBR) method is described for reconstructing an intermediate view with a steady-state matching probability (SSMP) density function. Conventionally, given multiple reference images, the intermediate view is synthesized via the depth image-based rendering technique in which geometric information (e.g., depth) is explicitly leveraged, thus leading to serious rendering artifacts on the synthesized view even with small depth errors. We address this problem by formulating the rendering process as an image fusion in which the textures of all probable matching points are adaptively blended with the SSMP representing the likelihood that points among the input reference images are matched. The PBR hence becomes more robust against depth estimation errors than existing view synthesis approaches. The MP in the steady-state, SSMP, is inferred for each pixel via the random walk with restart (RWR). The RWR always guarantees visually consistent MP, as opposed to conventional optimization schemes (e.g., diffusion or filtering-based approaches), the accuracy of which heavily depends on parameters used. Experimental results demonstrate the superiority of the PBR over the existing view synthesis approaches both qualitatively and quantitatively. Especially, the PBR is effective in suppressing flicker artifacts of virtual video rendering although no temporal aspect is considered. Moreover, it is shown that the depth map itself calculated from our RWR-based method (by simply choosing the most probable matching point) is also comparable with that of the state-of-the-art local stereo matching methods.