Simultaneous Learning Intensity and Optical Flow from High-speed Spike Stream

• Published in: IEEE transactions on circuits and systems for video technology 2024-12, p.1-1
• Main Authors: Zhu, Lin, Yan, Weiquan, Chang, Yi, Tian, Yonghong, Huang, Hua
• Format: Article
• Language: English
• Subjects: Bio-inspired camera; Cameras; Decoding; Estimation; Feature extraction; Image motion analysis; Image reconstruction; Neurons; Optical flow; Optical flow estimation; Spiking neural network; Streaming media; Vision sensors
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2024.3516478

Bio-inspired vision sensors, which emulate the human retina by recording light intensity as binary spikes, have gained increasing interest in recent years. Among them, the spike camera is capable of perceiving fine textures by simulating a small retinal region called the fovea and producing high temporal resolution (20,000 Hz) spatiotemporal spike streams. To bridge the gap between binary spike streams and human vision in high-speed scenes, reconstructing intensity and optical flow from high temporal resolution spikes is particularly important. In this paper, we present a hybrid SNN-ANN network designed for simultaneous intensity and optical flow learning from spike streams. To adaptively extract spatial and temporal features from continuous spike streams, we propose a spiking neuron module with dense connections that efficiently processes both short-term and long-term spike data, while maintaining low power consumption characteristics. Subsequently, we introduce two decoders for optical flow and intensity estimation that complement each other. A temporal-aware warping module, based on flow features, is specifically designed to align the temporal features of the intensity decoder, thereby reducing motion artifacts. Concurrently, improved intensity features contribute to more accurate flow feature predictions, resulting in a mutually beneficial relationship within our network. To evaluate the effectiveness of our proposed network, we conduct experiments on both simulated and real spike datasets. Our network outperforms existing state-of-the-art spike-based reconstruction and optical flow estimation methods, demonstrating its potential for advancing the field of bio-inspired vision sensors. Our code is available at https://github.com/LinZhu111/SLIO.