Deep Sensing for Compressive Video Acquisition

• Published in: Sensors (Basel, Switzerland) Switzerland), 2023-08, Vol.23 (17), p.7535
• Main Authors: Yoshida, Michitaka, Torii, Akihiko, Okutomi, Masatoshi, Taniguchi, Rin-ichiro, Nagahara, Hajime, Yagi, Yasushi
• Format: Article
• Language: English
• Subjects: CMOS; compressive sensing; Deep learning; deep neural network; deep optics; deep sensing; Methods; Neural networks; Optimization; Sensors; video reconstruction
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s23177535

A camera captures multidimensional information of the real world by convolving it into two dimensions using a sensing matrix. The original multidimensional information is then reconstructed from captured images. Traditionally, multidimensional information has been captured by uniform sampling, but by optimizing the sensing matrix, we can capture images more efficiently and reconstruct multidimensional information with high quality. Although compressive video sensing requires random sampling as a theoretical optimum, when designing the sensing matrix in practice, there are many hardware limitations (such as exposure and color filter patterns). Existing studies have found random sampling is not always the best solution for compressive sensing because the optimal sampling pattern is related to the scene context, and it is hard to manually design a sampling pattern and reconstruction algorithm. In this paper, we propose an end-to-end learning approach that jointly optimizes the sampling pattern as well as the reconstruction decoder. We applied this deep sensing approach to the video compressive sensing problem. We modeled the spatio–temporal sampling and color filter pattern using a convolutional neural network constrained by hardware limitations during network training. We demonstrated that the proposed method performs better than the manually designed method in gray-scale video and color video acquisitions.