Image Inpainting by End-to-End Cascaded Refinement with Mask Awareness

Inpainting arbitrary missing regions is challenging because learning valid features for various masked regions is nontrivial. Though U-shaped encoder-decoder frameworks have been witnessed to be successful, most of them share a common drawback of mask unawareness in feature extraction because all co...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on image processing 2021-01, Vol.PP, p.1-1
Main Authors: Zhu, Manyu, He, Dongliang, Li, Xin, Lia, Chao, Lib, Fu, Liu, Xiao, Ding, Errui, Zhang, Zhaoxiang
Format: Article
Language:English
Subjects:
Cascaded Refinement
Coders
Convolution
Decoding
Dynamic Filtering
Encoders-Decoders
Feature extraction
Image Inpainting
Image reconstruction
Kernel
Mask Awareness
Scaling factors
Shape
Task analysis
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Inpainting arbitrary missing regions is challenging because learning valid features for various masked regions is nontrivial. Though U-shaped encoder-decoder frameworks have been witnessed to be successful, most of them share a common drawback of mask unawareness in feature extraction because all convolution windows (or regions), including those with various shapes of missing pixels, are treated equally and filtered with fixed learned kernels. To this end, we propose our novel mask-aware inpainting solution. Firstly, a Mask-Aware Dynamic Filtering (MADF) module is designed to effectively learn multi-scale features for missing regions in the encoding phase. Specifically, filters for each convolution window are generated from features of the corresponding region of the mask. The second fold of mask awareness is achieved by adopting Point-wise Normalization (PN) in our decoding phase, considering that statistical natures of features at masked points differentiate from those of unmasked points. The proposed PN can tackle this issue by dynamically assigning point-wise scaling factor and bias. Lastly, our model is designed to be an end-to-end cascaded refinement one. Supervision information such as reconstruction loss, perceptual loss and total variation loss is incrementally leveraged to boost the inpainting results from coarse to fine. Effectiveness of the proposed framework is validated both quantitatively and qualitatively via extensive experiments on three public datasets including Places2, CelebA and Paris StreetView.
ISSN:1057-7149
1941-0042
DOI:10.1109/TIP.2021.3076310