A Multimode Vision Sensor With Temporal Contrast Pixel and Column-Parallel Local Binary Pattern Extraction for Dynamic Depth Sensing Using Stereo Vision

• Published in: IEEE journal of solid-state circuits 2023-10, Vol.58 (10), p.1-11
• Main Authors: Chiu, Min-Yang, Chen, Guan-Cheng, Hsu, Tzu-Hsiang, Liu, Ren-Shuo, Lo, Chung-Chuan, Tang, Kea-Tiong, Chang, Meng-Fan, Hsieh, Chih-Cheng
• Format: Article
• Language: English
• Subjects: Analog to digital conversion; Data transfer (computers); Feature extraction; frame difference (FD); global shutter; Information retrieval; local binary pattern (LBP); Mathematical analysis; Motion detection; Pixels; Power consumption; Power management; Pulse duration modulation; Pulse width modulation; region of interest (ROI); saliency detection; Sensors; Spatial data; Stereo vision; vision sensor; Vision sensors; Vision systems; Voltage
• ISSN: 0018-9200; 1558-173X
• DOI: 10.1109/JSSC.2023.3292051

This article presents a 0.56/0.8 V multimode vision sensor for temporal and spatial information extraction with processing in sensor (PIS) technique. The proposed six transistors and one capacitor (6T1C) temporal contrast pixel (TCP) structure using the exposure compensation scheme (ECS) can realize in-pixel temporal contrast calculation and motion event reporting (ER) with global shutter exposure and seamless frame difference (FD) based on the pulsewidth modulation (PWM) operation. In addition, the column-parallel local binary pattern (LBP) extraction provides further spatial feature information without the power-consuming analog-to-digital conversion to meet the power-efficient requirement for always-on-edge devices. According to the temporal and spatial information extracted from FD and LBP modes, this work realizes the PIS operations for dynamic depth calculation of a stereo vision system, which filters out the static scene and provides the depth information of the dynamic objects only. To further alleviate the data transfer bandwidth, the region of interest (ROI) capability is also embedded in the sensor for data windowing and object locating. The prototyped vision sensor was fabricated in TSMC's standard 0.18 \mu m CMOS process and verified. The vision sensor provides five operating modes that can adapt to different tasks, which include the raw image mode (IM), FD mode, ER mode, LBP mode, and ROI mode. The measurement results show a maximum frame rate of 540/819/540 frames/s at a power consumption of 390/162.6/151.9 \mu W with an achieved iFoM of 45.5/12.5/17.7 pJ/pixel \cdot frame in IM/ER/LBP mode, respectively.