Driving Autonomy with Event-Based Cameras: Algorithm and Hardware Perspectives

• Main Authors: Rahman, Nael Mizanur, Kamal, Uday, Nagaraj, Manish, Roy, Shaunak, Mukhopadhyay, Saibal
• Format: Conference Proceeding
• Language: English
• Subjects: Associative memory; Cameras; event-based camera; Hardware; Heuristic algorithms; multi-object tracking; Robot vision systems; spiking neural network; Tracking; Transforms
• ISSN: 1558-1101
• DOI: 10.23919/DATE58400.2024.10546715

In high-speed robotics and autonomous vehicles, rapid environmental adaptation is necessary. Traditional cameras often face issues with motion blur and limited dynamic range. Event-based cameras address these by tracking pixel changes continuously and asynchronously, offering higher temporal resolution with minimal blur. In this work, we highlight our recent efforts in solving the challenge of processing event-camera data efficiently from both algorithm and hardware perspective. Specifically, we present how brain-inspired algorithms such as spiking neural networks (SNNs) can efficiently detect and track object motion from event-camera data. Next, we discuss how we can leverage associative memory structures for efficient event-based represen-tation learning. And finally, we show how our developed Application Specific Integrated Circuit (ASIC) architecture for low-latency, energy-efficient processing outperforms typical GPU/CPU solutions, thus enabling real-time event-based processing. With a 100x reduction in latency and a 1000x lower energy per event compared to state-of-the-art GPU/CPU setups, this enhances the front-end camera systems capability in autonomous vehicles to handle higher rates of event generation, improving control.