ViTA: Video Transformer Adaptor for Robust Video Depth Estimation

• Published in: IEEE transactions on multimedia 2024, Vol.26, p.3302-3316
• Main Authors: Xian, Ke, Peng, Juewen, Cao, Zhiguo, Zhang, Jianming, Lin, Guosheng
• Format: Article
• Language: English
• Subjects: Ablation; Adapters; Bidirectional control; Computer vision; Estimation; Frame design; Frames (data processing); Image reconstruction; Optical flow; Optical flow (image analysis); Optical losses; Robust video depth estimation; Robustness (mathematics); spatio-temporal consistency loss; Streaming media; Training; Transformers; video transformer adaptor
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2023.3309559

Depth information plays a pivotal role in numerous computer vision applications, including autonomous driving, 3D reconstruction, and 3D content generation. When deploying depth estimation models in practical applications, it is essential to ensure that the models have strong generalization capabilities. However, existing depth estimation methods primarily concentrate on robust single-image depth estimation, leading to the occurrence of flickering artifacts when applied to video inputs. On the other hand, video depth estimation methods either consume excessive computational resources or lack robustness. To address the above issues, we propose ViTA, a video transformer adaptor, to estimate temporally consistent video depth in the wild. In particular, we leverage a pre-trained image transformer (i.e., DPT) and introduce additional temporal embeddings in the transformer blocks. Such designs enable our ViTA to output reliable results given an unconstrained video. Besides, we present a spatio-temporal consistency loss for supervision. The spatial loss computes the per-pixel discrepancy between the prediction and the ground truth in space, while the temporal loss regularizes the inconsistent outputs of the same point in consecutive frames. To find the correspondences between consecutive frames, we design a bi-directional warping strategy based on the forward and backward optical flow. During inference, our ViTA no longer requires optical flow estimation, which enables it to estimate spatially accurate and temporally consistent video depth maps with fine-grained details in real time. We conduct a detailed ablation study to verify the effectiveness of the proposed components. Extensive experiments on the zero-shot cross-dataset evaluation demonstrate that the proposed method is superior to previous methods.