Fast Spatial-Temporal Transformer Network

In computer vision, the restoration of missing regions in an image can be tackled with image inpainting techniques. Neural networks that perform inpainting in videos require the extraction of information from neighboring frames to obtain a temporally coherent result. The state-of-the-art methods for...

Full description

Saved in:
Bibliographic Details
Main Authors: Escher, Rafael Molossi, de Bem, Rodrigo Andrade, Drews, Paulo Lilles Jorge
Format: Conference Proceeding
Language:English
Subjects:
Computer vision
Data mining
Deep Learning
Graphics
Image restoration
Neural networks
Reformer Networks
Training
Transformer Networks
Transformers
Video Inpainting
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In computer vision, the restoration of missing regions in an image can be tackled with image inpainting techniques. Neural networks that perform inpainting in videos require the extraction of information from neighboring frames to obtain a temporally coherent result. The state-of-the-art methods for video inpainting are mainly based on Transformer Networks, which rely on attention mechanisms to handle temporal input data. However, such networks are highly costly, requiring considerable computational power for training and testing, which hinders its use on modest computing platforms. In this context, our goal is to reduce the computational complexity of state-of-the-art video inpainting methods, improving performance and facilitating its use in low-end GPUs. Therefore, we introduce the Fast Spatio-Temporal Transformer Network (FastSTTN), an extension of the Spatio-Temporal Transformer Network (STTN) in which the adoption of Reversible Layers reduces memory usage up to 7 times and execution time by approximately 2.2 times, while maintaining state-of-the-art video inpainting accuracy.
ISSN:2377-5416
DOI:10.1109/SIBGRAPI54419.2021.00018