Compressing convolutional neural networks with hierarchical Tucker-2 decomposition

• Published in: Applied soft computing 2023-01, Vol.132, p.109856, Article 109856
• Main Authors: Gabor, Mateusz, Zdunek, Rafał
• Format: Article
• Language: English
• Subjects: Convolutional neural networks; Hierarchical Tucker decomposition; Tensor decomposition
• ISSN: 1568-4946; 1872-9681
• DOI: 10.1016/j.asoc.2022.109856

Convolutional neural networks (CNNs) play a crucial role and achieve top results in computer vision tasks but at the cost of high computational cost and storage complexity. One way to solve this problem is the approximation of the convolution kernel using tensor decomposition methods. In this way, the original kernel is replaced with a sequence of kernels in a lower-dimensional space. This study proposes a novel CNN compression technique based on the hierarchical Tucker-2 (HT-2) tensor decomposition and makes an important contribution to the field of neural network compression based on low-rank approximations. We demonstrate the effectiveness of our approach on many CNN architectures on CIFAR-10 and ImageNet datasets. The obtained results show a significant reduction in parameters and FLOPS with a minor drop in classification accuracy. Compared to different state-of-the-art compression methods, including pruning and matrix/tensor decomposition, the HT-2, as a new alternative, outperforms most of the cited methods. The implementation of the proposed approach is very straightforward and can be easily coded in every deep learning library. •Hierarchical Tucker-2 decomposition is applied to compress CNNs.•The proposed method was compared with the state-of-the-art CNN compression methods.•Substantial parameter and flops compression is obtained at marginal accuracy drop.•New notation for the Kruskal convolution is introduced.