ConVRT: Consistent Video Restoration Through Turbulence with Test-time Optimization of Neural Video Representations

tmospheric turbulence presents a significant challenge in long-range imaging. Current restoration algorithms often struggle with temporal inconsistency, as well as limited generalization ability across varying turbulence levels and scene content different than the training data. To tackle these issu...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-12
Main Authors: Cai, Haoming, Chen, Jingxi, Feng, Brandon Y, Jiang, Weiyun, Xie, Mingyang, Zhang, Kevin, Veeraraghavan, Ashok, Metzler, Christopher
Format: Article
Language:English
Subjects:
Algorithms
Optimization
Representations
Restoration
Statistical correlation
Testing time
Turbulence
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:tmospheric turbulence presents a significant challenge in long-range imaging. Current restoration algorithms often struggle with temporal inconsistency, as well as limited generalization ability across varying turbulence levels and scene content different than the training data. To tackle these issues, we introduce a self-supervised method, Consistent Video Restoration through Turbulence (ConVRT) a test-time optimization method featuring a neural video representation designed to enhance temporal consistency in restoration. A key innovation of ConVRT is the integration of a pretrained vision-language model (CLIP) for semantic-oriented supervision, which steers the restoration towards sharp, photorealistic images in the CLIP latent space. We further develop a principled selection strategy of text prompts, based on their statistical correlation with a perceptual metric. ConVRT's test-time optimization allows it to adapt to a wide range of real-world turbulence conditions, effectively leveraging the insights gained from pre-trained models on simulated data. ConVRT offers a comprehensive and effective solution for mitigating real-world turbulence in dynamic videos.
ISSN:2331-8422