RePaIR: Repaired pruning at initialization resilience

• Published in: Neural networks 2024-12, Vol.184, p.107086, Article 107086
• Main Authors: Zhao, Haocheng, Guan, Runwei, Man, Ka Lok, Yu, Limin, Yue, Yutao
• Format: Article
• Language: English
• Subjects: Lipschitz; Neural network; Pruning at initialization; Unstructured pruning
• ISSN: 0893-6080; 1879-2782; 1879-2782
• DOI: 10.1016/j.neunet.2024.107086

Over the past decade, the size of neural network models has gradually increased in both breadth and depth, leading to a growing interest in the application of neural network pruning. Unstructured pruning provides fine-grained sparsity and achieves better inference acceleration under specific hardware support. Unstructured Pruning at Initialization (PaI) optimizes the iterative pruning pipeline, but sparse weights increase the risk of underfitting during training. More importantly, almost all PaI algorithms focus only on obtaining the best pruning mask without considering whether the retained weights are suitable for training. Introducing Lipschitz constants during model initialization can reduce the risk of model underfitting and overfitting. As a result, we firstly analyze the impact of Lipschitz initialization on model training and propose the Repaired Initialization (ReI) algorithm for common modules with BatchNorm. Then, we utilize the same idea to repair the weight of unstructured pruned model, and name it Repaired Pruning at Initialization Resilience (RePaIR) algorithm. Extensive experiments and demonstrate that our proposed ReI and RePaIR can improve the training robustness of unpruned and pruned models, respectively, and achieve up to 1.7% accuracy gain with the same sparse pruning mask on TinyImageNet. Futhermore, we provide an improved SynFlow algorithm called Repair SynFlow (ReSynFlow), which employs Lipschitz scaling to overcome the problem of score computation in deeper models. ReSynFlow can effectively improve the maximum compression rate and is suitable for deeper models, with an accuracy improvement of up to 1.3% compared to the SynFlow algorithm on TinyImageNet. •A novel Lipschitz-based initialization algorithm for BatchNorm.•A novel Lipschitz-based weight scaling algorithm for Pruning at Initialization.•An improved SynFlow pruning at initialization algorithm based on Lipschitz.