DENAO: Monocular Depth Estimation Network With Auxiliary Optical Flow

• Published in: IEEE transactions on pattern analysis and machine intelligence 2021-08, Vol.43 (8), p.2598-2610
• Main Authors: Chen, Jingyu, Yang, Xin, Jia, Qizeng, Liao, Chunyuan
• Format: Article
• Language: English
• Subjects: Artificial neural networks; Cameras; Coders; Computer architecture; Computer vision; Convolutional neural network; depth estimation; Encoders-Decoders; Estimation; Flow nets; Optical communication; Optical computing; Optical fiber networks; optical flow; Optical flow (image analysis); Optical imaging; Pedestrians; Robot vision systems; Robotics; Training
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2020.2977021

Estimating depth from multi-view images captured by a localized monocular camera is an essential task in computer vision and robotics. In this study, we demonstrate that learning a convolutional neural network (CNN) for depth estimation with an auxiliary optical flow network and the epipolar geometry constraint can greatly benefit the depth estimation task and in turn yield large improvements in both accuracy and speed. Our architecture is composed of two tightly-coupled encoder-decoder networks, i.e., an optical flow net and a depth net, the core part being a list of exchange blocks between the two nets and an epipolar feature layer in the optical flow net to improve predictions of both depth and optical flow. Our architecture allows to input arbitrary number of multiview images with a linearly growing time cost for optical flow and depth estimation. Experimental result on five public datasets demonstrates that our method, named DENAO, runs at 38.46fps on a single Nvidia TITAN Xp GPU which is 5.15X\sim ∼ 142X faster than the state-of-the-art depth estimation methods Meanwhile, our DENAO can concurrently output predictions of both depth and optical flow, and performs on par with or outperforms the state-of-the-art depth estimation methods and optical flow methods.