End-to-End Video Snapshot Compressive Imaging using Video Transformers

This paper presents a novel reconstruction algorithm for video Snapshot Compressive Imaging (SCI). Inspired by recent research works on Transformers and Self-Attention mechanism in computer vision, we propose the first video SCI reconstruction algorithm built upon Transformers to capture long-range...

Full description

Saved in:
Bibliographic Details
Main Authors: Saideni, Wael, Courreges, Fabien, Helbert, David, Cances, Jean Pierre
Format: Conference Proceeding
Language:English
Subjects:
computer vision
Deep learning
Image coding
Imaging
Memory management
optimization
Reconstruction algorithms
Streaming media
Transformers
video compressive sensing
Vision Transformers
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper presents a novel reconstruction algorithm for video Snapshot Compressive Imaging (SCI). Inspired by recent research works on Transformers and Self-Attention mechanism in computer vision, we propose the first video SCI reconstruction algorithm built upon Transformers to capture long-range spatio-temporal dependencies enabling the deep learning of feature maps. Our approach is based on a Spatiotempo-ral Convolutional Multi-head Attention (ST-ConvMHA) which enable to exploit the spatial and temporal information of the video scenes instead of using fully-connected attention layers. To evaluate the performances of our approach, we train our algorithm on DAVIS2017 dataset and we test the trained models on six benchmark datasets. The obtained results in terms of PSNR, SSIM and especially reconstruction time prove the ability of using our reconstruction approach for real-time applications. We truly believe that our research will motivate future works for more video reconstruction approaches.
ISSN:2154-512X
DOI:10.1109/IPTA54936.2022.9784128