Spectral-based convolutional neural network without multiple spatial-frequency domain switchings

• Published in: Neurocomputing (Amsterdam) 2019-10, Vol.364, p.152-167
• Main Authors: Ayat, Sayed Omid, Khalil-Hani, Mohamed, Ab Rahman, Ab Al-Hadi, Abdellatef, Hamdan
• Format: Article
• Language: English
• Subjects: Activation function; Batch normalization; Convolutional neural network; Embedded systems; Frequency domain; Spectral representation
• ISSN: 0925-2312; 1872-8286
• DOI: 10.1016/j.neucom.2019.06.094

Recent researches have shown that spectral representation provides a significant speed-up in the massive computation workload of convolution operations in the inference (feed-forward) algorithm of Convolutional Neural Networks (CNNs). This approach results in reducing the computational complexity of the classification task, which makes spectral-based CNN suitable for implementation on embedded platform that typically has constrained resources. However, a major challenge in this approach is that the mathematical formulation of a nonlinear activation function in spectral (frequency) domain is currently not available; hence, computation of the activation functions in each layer has to be performed in the spatial domain. This results in several spatial-frequency domain switchings that are computationally very costly, and as such, it would be advantageous to strictly stay in the frequency domain. Hence, in this work, a novel Spectral Rectified Linear Unit (SReLU) for the activation function is proposed, that makes it possible for the computations to remain in the frequency domain, and therefore avoids the multiple compute-intensive domain transformations. To further optimize the classification speed of the network, an efficient spectral-based CNN model is presented that uses only the lower frequency components by way of fusing the convolutional and sub-sampling layers. Additionally, we provide and utilize a frequency domain equivalent of the conventional batch normalization layer that results in improving the accuracy of the network. Experimental results indicate that the proposed spectral-based CNN model achieves up to 17.02  ×  and 3.45  ×  faster classification speed (without considerable accuracy loss) on AT&T face recognition and MNIST digit/fashion classification datasets, respectively, as compared to the equivalent models in the spatial domain, hence outperforming conventional approaches significantly.