A 0.82 μW CIS-Based Action Recognition SoC With Self-Adjustable Frame Resolution for Always-on IoT Devices

• Published in: IEEE transactions on circuits and systems. II, Express briefs Express briefs, 2021-05, Vol.68 (5), p.1700-1704
• Main Authors: Ryu, Junha, Park, Gwangtae, Im, Dongseok, Kim, Ji-Hoon, Yoo, Hoi-Jun
• Format: Article
• Language: English
• Subjects: Algorithms; Always-on; CMOS; CMOS image sensor; deep neural network; Event detection; Generators; human action recognition; Human activity recognition; Human motion; Image resolution; Integrated circuits; Internet of Things; motion detection; Motion perception; Object recognition; Power consumption; Power demand; Power management; Prediction algorithms; resolution-adaptive; Signal resolution; Skips; System on chip
• ISSN: 1549-7747; 1558-3791
• DOI: 10.1109/TCSII.2021.3067151

An always-on video-based human action recognition (HAR) system on chip (SoC) integrated with a CMOS image sensor (CIS) is proposed for the Internet of Things (IoT) devices. The proposed SoC is the first always-on integrated circuit (IC) performing the full process of HAR in a single chip. To resolve large power consumption from vision sensor and compute-intensive DNN operation, the proposed SoC operates in two different modes; 1) in adaptive frame resolution based human action recognition (AFR-HAR) mode, CIS resolution prediction algorithm and self-adjustable CIS reduce 42.9-91.8% of readout power by adaptively adjusting frame resolution. 2) In motion event detection (MED) mode, the motion event detection unit (MEDU) skips unnecessary imaging and DNN computation by monitoring motion events and leads to over 99% power saving. The proposed HAR SoC is simulated in 65-nm CMOS technology and occupies 8.56 mm 2 . It consumes only 0.82 {\mu }\text{W} when no motion is detected and 0.31-8.52 mW for evaluating human actions on the ActivityNet dataset.