MCD-Net: Toward RGB-D Video Inpainting in Real-World Scenes

Video inpainting gains an increasing amount of attention ascribed to its wide applications in intelligent video editing. However, despite tremendous progress made in RGB video inpainting, the existing RGB-D video inpainting models are still incompetent to inpaint real-world RGB-D videos, as they sim...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on image processing 2024, Vol.33, p.1095-1108
Main Authors: Hou, Jiacheng, Ji, Zhong, Yang, Jinyu, Wang, Chengjie, Zheng, Feng
Format: Article
Language:English
Subjects:
Annotations
Color
Convolution
Correlation
Datasets
Electronic mail
Image color analysis
Learning systems
multi-modal learning
RGB-D inpainting
RGB-D object removal
RGB-D video dataset
Three-dimensional displays
Transformers
Video data
Video inpainting
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Video inpainting gains an increasing amount of attention ascribed to its wide applications in intelligent video editing. However, despite tremendous progress made in RGB video inpainting, the existing RGB-D video inpainting models are still incompetent to inpaint real-world RGB-D videos, as they simply fuse color and depth via explicit feature concatenation, neglecting the natural modality gap. Moreover, current RGB-D video inpainting datasets are synthesized with homogeneous and delusive RGB-D data, which is far from real-world application and cannot provide comprehensive evaluation. To alleviate these problems and achieve real-world RGB-D video inpainting, on one hand, we propose a Mutually-guided Color and Depth Inpainting Network (MCD-Net), where color and depth are reciprocally leveraged to inpaint each other implicitly, mitigating the modality gap and fully exploiting cross-modal association for inpainting. On the other hand, we build a Video Inpainting with Depth (VID) dataset to supply diverse and authentic RGB-D video data with various object annotation masks to enable comprehensive evaluation for RGB-D video inpainting under real-world scenes. Experimental results on the DynaFill benchmark and our collected VID dataset demonstrate our MCD-Net not only yields the state-of-the-art quantitative performance but successfully achieves high-quality RGB-D video inpainting under real-world scenes. All resources are available at https://github.com/JCATCV/MCD-Net .
ISSN:1057-7149
1941-0042
DOI:10.1109/TIP.2024.3358675