Video Super-Resolution via a Spatio-Temporal Alignment Network

Deep convolutional neural network based video super-resolution (SR) models have achieved significant progress in recent years. Existing deep video SR methods usually impose optical flow to wrap the neighboring frames for temporal alignment. However, accurate estimation of optical flow is quite diffi...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on image processing 2022, Vol.31, p.1761-1773
Main Authors: Wen, Weilei, Ren, Wenqi, Shi, Yinghuan, Nie, Yunfeng, Zhang, Jingang, Cao, Xiaochun
Format: Article
Language:English
Subjects:
Adaptive filters
Alignment
Artificial neural networks
Estimation
Feature extraction
Frames (data processing)
Image reconstruction
Integrated optics
Motion compensation
Optical flow (image analysis)
Optical imaging
spatio-temporal adaptive filters
Superresolution
temporal consistency
Video super-resolution
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Deep convolutional neural network based video super-resolution (SR) models have achieved significant progress in recent years. Existing deep video SR methods usually impose optical flow to wrap the neighboring frames for temporal alignment. However, accurate estimation of optical flow is quite difficult, which tends to produce artifacts in the super-resolved results. To address this problem, we propose a novel end-to-end deep convolutional network that dynamically generates the spatially adaptive filters for the alignment, which are constituted by the local spatio-temporal channels of each pixel. Our method avoids generating explicit motion compensation and utilizes spatio-temporal adaptive filters to achieve the operation of alignment, which effectively fuses the multi-frame information and improves the temporal consistency of the video. Capitalizing on the proposed adaptive filter, we develop a reconstruction network and take the aligned frames as input to restore the high-resolution frames. In addition, we employ residual modules embedded with channel attention as the basic unit to extract more informative features for video SR. Both quantitative and qualitative evaluation results on three public video datasets demonstrate that the proposed method performs favorably against state-of-the-art super-resolution methods in terms of clearness and texture details.
ISSN:1057-7149
1941-0042
DOI:10.1109/TIP.2022.3146625