Real-World Spatial Synchronization of Event-CMOS Cameras through Deep Learning: A Novel CNN-DGCNN Approach

• Published in: Sensors (Basel, Switzerland) Switzerland), 2024-06, Vol.24 (13), p.4050
• Main Authors: Mizrahi, Dor, Laufer, Ilan, Zuckerman, Inon
• Format: Article
• Language: English
• Subjects: Accuracy; Calibration; Cameras; CMOS; Datasets; Deep learning; dynamic graph convolutional neural networks (DGCNN); event-based sensing; Fault diagnosis; image alignment; Neural networks; sensor fusion; Sensors
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s24134050

This paper presents a new deep-learning architecture designed to enhance the spatial synchronization between CMOS and event cameras by harnessing their complementary characteristics. While CMOS cameras produce high-quality imagery, they struggle in rapidly changing environments-a limitation that event cameras overcome due to their superior temporal resolution and motion clarity. However, effective integration of these two technologies relies on achieving precise spatial alignment, a challenge unaddressed by current algorithms. Our architecture leverages a dynamic graph convolutional neural network (DGCNN) to process event data directly, improving synchronization accuracy. We found that synchronization precision strongly correlates with the spatial concentration and density of events, with denser distributions yielding better alignment results. Our empirical results demonstrate that areas with denser event clusters enhance calibration accuracy, with calibration errors increasing in more uniformly distributed event scenarios. This research pioneers scene-based synchronization between CMOS and event cameras, paving the way for advancements in mixed-modality visual systems. The implications are significant for applications requiring detailed visual and temporal information, setting new directions for the future of visual perception technologies.