Multi-scale foreground-background separation for light field depth estimation with deep convolutional networks

• Published in: Pattern recognition letters 2023-07, Vol.171, p.138-147
• Main Authors: Lee, Jae Young, Hur, Jiwan, Choi, Jaehyun, Park, Rae-Hong, Kim, Junmo
• Format: Article
• Language: English
• Subjects: Depth estimation; Foreground-Background separation; Light field
• ISSN: 0167-8655
• DOI: 10.1016/j.patrec.2023.05.014

•For depth from light field, a framework based on the foreground-background separation with deep learning is presented.•A method of the multi-scale foreground-background separation is proposed.•To further improve around occlusion boundary, a loss function matching gradient magnitude is proposed. There are three viewpoints to divide the depth from light field (DFLF) methods: light field (LF) representation, signal processing, and learning viewpoints. In the signal processing viewpoint in details, the DFLF methods can be divided into three groups: cost-based, foreground-background separation (FBS)-based, and depth model-based methods. Although the deep learning techniques are widely implemented in the cost-based and depth model-based methods for the DFLF, they are not implemented in the FBS-based method, yet. In this paper, we propose a DFLF method based on the FBS with the deep learning techniques. Based on an observation that the angular aliasing artifacts are reduced in the resulting disparity maps obtained from spatially low-scale LF images, multi-scale (MS) FBS score maps are used in the proposed methods. To combine the MS FBS score maps into a single FBS score maps, four directional LF gradients, features from the 4-D LF, and features from the center view images are utilized. Based on the sensitivity and specificity loss, the gradient matching loss between the predicted and ground truth disparity maps is used to improve the performance around the occlusion boundary in the resulting disparity maps. The experimental results show that the proposed method performs reasonably in both the synthetic and real LF datasets. Especially, in the texture-less region in the real dataset, the resulting disparity maps obtained by the proposed method show that the artifacts are reduced compared to those by the existing DFLF methods.