AIMMI: Audio and Image Multi-Modal Intelligence via a Low-Power SoC With 2-MByte On-Chip MRAM for IoT Devices

• Published in: IEEE journal of solid-state circuits 2024-10, Vol.59 (10), p.3488-3501
• Main Authors: Fan, Zichen, Zhang, Qirui, An, Hyochan, Xu, Boxun, Xu, Li, Tseng, Chien-Wei, Peng, Yimai, Bejarano-Carbo, Andrea, Abillama, Pierre, Cao, Ang, Liu, Bowen, Lee, Changwoo, Wang, Zhehong, Kim, Hun-Seok, Blaauw, David, Sylvester, Dennis
• Format: Article
• Language: English
• Subjects: Artificial neural networks; Back propagation networks; Energy efficiency; Face recognition; Generative adversarial networks; Image coding; Intelligence; Internet of Things; Leakage; Machine learning; Magnetoresistivity; Neural networks; non-volatile memory; Power management; Program processors; Signal processing; System on chip; system-on-chip (SoC); ultra-low power
• ISSN: 0018-9200; 1558-173X
• DOI: 10.1109/JSSC.2024.3410306

In this article, we present an ultra-low-power multi-modal signal processing system on chip (SoC) [audio and image multi-modal intelligence (AIMMI)] that integrates a versatile deep neural network (DNN) engine with audio and image signal processing accelerators for multi-modal Internet-of-Things (IoT) intelligence. In order to get high energy efficiency under resource-constrained IoT scenarios, AIMMI features three efficiency-boosting techniques: 1) 2-MB on-chip non-volatile magnetoresistive RAM (MRAM) to store all DNN weights with MRAM-cache microarchitecture that incorporates dynamic power gating to reduce both leakage and dynamic power consumption; 2) a deliberate power management scheme that enables optimized power modes under different operating situations; and 3) a novel reconfigurable neural engine (NE) with energy-efficient dataflow for comprehensive DNN instructions. Fabricated in TSMC 22-nm ultra-low leakage (ULL) technology with MRAM, AIMMI achieves up to 3-10-TOPS/W peak energy efficiency and consumes only 0.25-3.84 mW. It demonstrates convolutional neural network (CNN), generative adversarial network (GAN), and back-propagation (BP) operations on a single accelerator SoC for multi-modal fusion, outperforming state-of-the-art DNN processors by 1.4 \times -4.5 \times in energy efficiency.