Fast and efficient implementation of image filtering using a side window convolutional neural network

• Published in: Signal processing 2020-11, Vol.176, p.107717, Article 107717
• Main Authors: Yin, Hui, Gong, Yuanhao, Qiu, Guoping
• Format: Article
• Language: English
• Subjects: Convolutional neural network; Edge-preserving filter; Residual learning; Side kernel convolution; Side window
• ISSN: 0165-1684; 1872-7557
• DOI: 10.1016/j.sigpro.2020.107717

•We propose a generic side window filtering (GSWF) framework. It generically abstracts the essence of edge-preserving filtering by feature calculation and fusion. The features are calculated in side windows which can prevent the edges from blurring. By setting proper weighting coefficients, the features can be fused to get the filtering result. In theory, given the right filtering kernel and weighting coefficients, GSWF can be used to approximate arbitrary edge-preserving filters.•Based on GSWF and by taking advantage of the powerful representative capability of convolutional neural network, we design a side window convolutional neural network (SW-CNN). It implements the feature calculation by a new convolutional strategy called side kernel convolution and the feature fusion by two convolutions. It is a fast and efficient implementation of image filtering by convolutional neural network. By supervised learning, the kernel and the weighting coefficients can be learned to simulate and accelerate arbitrary filtering operator.•SKC is a new convolutional strategy proposed in this paper. Different from the convolutional operation in traditional CNN, in SKC, the pixel to be processed is placed on a side or a corner rather than the center of the convolutional operation window. This new convolutional operation simulates the operation in SWF. In this way, the edges will not be blurred during feature calculation of SW-CNN, which enables it to solve the low-level processing problem by low-level operations. Moreover, SKC can be used as a module to replace the traditional convolution module in other neural networks.•Compared to state-of-art networks, such as DEAF and CAN, SW-CNN can achieve better or comparable results on implementing many local or global based filters with its trainable parameters only 0.65% and 4% of DEAF and CAN, respectively. On graphics processing unit (GPU), our implementation achieves more than 900 times acceleration for several filters and leads to state-of-the-art speed for neural network implementations of filters. Convolutional neural networks (CNNs) designed for object recognition have been successfully applied to low-level tasks such as image filtering. However, these networks are usually very large which occupy large memory space and demand very high computational capacity. This makes them unsuitable for real time low-level applications on smart and portable devices with limited memory and computational capacities. In this paper, we specif