SoftAct: A High-Precision Softmax Architecture for Transformers Supporting Nonlinear Functions

• Published in: IEEE transactions on circuits and systems for video technology 2024-09, Vol.34 (9), p.8912-8923
• Main Authors: Fu, Yuzhe, Zhou, Changchun, Huang, Tianling, Han, Eryi, He, Yifan, Jiao, Hailong
• Format: Article
• Language: English
• Subjects: Computer architecture; Convolutional neural networks; Costs; Deep learning; Detection algorithms; Inference algorithms; nonlinear functions; Nonlinear systems; overall efficiency; Quantization (signal); softmax; Software architecture; sparsity detection; Transformer-based networks; Transformers
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2024.3386779

Transformer-based deep learning networks are revolutionizing our society. The convolution and attention co-designed (CAC) Transformers have demonstrated superior performance compared to the conventional Transformer-based networks. However, CAC Transformer networks contain various nonlinear functions, such as softmax and complex activation functions, which require high precision hardware design yet typically with significant cost in area and power consumption. To address these challenges, SoftAct, a compact and high-precision algorithm-hardware co-designed architecture, is proposed to implement both softmax and nonlinear activation functions in CAC Transformer accelerators. An improved softmax algorithm with penalties is proposed to maintain precision in hardware. A stage-wise full zero detection method is developed to skip redundant computation in softmax. A compact and reconfigurable architecture with a symmetrically designed linear fitting module is proposed to achieve nonlinear functions. The SoftAct architecture is designed in an industrial 28-nm CMOS technology with the MobileViT-xxs network classifying the ImageNet-1k dataset as the benchmark. Compared with the state of the art, SoftAct improves up to 5.87% network accuracy under 8-bit quantization, 153.2\times area efficiency, and 1435\times overall efficiency.