Lightweight Modules for Efficient Deep Learning Based Image Restoration

• Published in: IEEE transactions on circuits and systems for video technology 2021-04, Vol.31 (4), p.1395-1410
• Main Authors: Lahiri, Avisek, Bairagya, Sourav, Bera, Sutanu, Haldar, Siddhant, Biswas, Prabir Kumar
• Format: Article
• Language: English
• Subjects: adversarial learning; Artificial intelligence; Artificial neural networks; Cell phones; Cellular telephones; Classification; CNN; Computational modeling; Convolution; Deep learning; efficient neural networks; Electronic devices; generative adversarial network (GAN); Image classification; Image denoising; image inpainting; Image processing; Image restoration; image super-resolution; Kernel; Lightweight; Low level; Machine learning; Modules; Neural networks; Noise reduction; Task analysis
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2020.3007723

Low level image restoration is an integral component of modern artificial intelligence (AI) driven camera pipelines. Most of these frameworks are based on deep neural networks which present a massive computational overhead on resource constrained platform like a mobile phone. In this paper, we propose several lightweight low-level modules which can be used to create a computationally low cost variant of a given baseline model. Recent works for efficient neural networks design have mainly focused on classification. However, low-level image processing falls under the 'image-to-image' translation genre which requires some additional computational modules not present in classification. This paper seeks to bridge this gap by designing generic efficient modules which can replace essential components used in contemporary deep learning based image restoration networks. We also present and analyse our results highlighting the drawbacks of applying depthwise separable convolutional kernel (a popular method for efficient classification network) for sub-pixel convolution based upsampling (a popular upsampling strategy for low-level vision applications). This shows that concepts from domain of classification cannot always be seamlessly integrated into 'image-to-image' translation tasks. We extensively validate our findings on three popular tasks of image inpainting, denoising and super-resolution. Our results show that proposed networks consistently output visually similar reconstructions compared to full capacity baselines with significant reduction of parameters, memory footprint and execution speeds on contemporary mobile devices.