Brain-inspired multimodal hybrid neural network for robot place recognition

• Published in: Science robotics 2023-05, Vol.8 (78), p.eabm6996-eabm6996
• Main Authors: Yu, Fangwen, Wu, Yujie, Ma, Songchen, Xu, Mingkun, Li, Hongyi, Qu, Huanyu, Song, Chenhang, Wang, Taoyi, Zhao, Rong, Shi, Luping
• Format: Article
• Language: English
• Subjects: Algorithms; Animals; Brain - physiology; Humans; Neural Networks, Computer; Neurons - physiology; Robotics - methods
• ISSN: 2470-9476; 2470-9476
• DOI: 10.1126/scirobotics.abm6996

Place recognition is an essential spatial intelligence capability for robots to understand and navigate the world. However, recognizing places in natural environments remains a challenging task for robots because of resource limitations and changing environments. In contrast, humans and animals can robustly and efficiently recognize hundreds of thousands of places in different conditions. Here, we report a brain-inspired general place recognition system, dubbed NeuroGPR, that enables robots to recognize places by mimicking the neural mechanism of multimodal sensing, encoding, and computing through a continuum of space and time. Our system consists of a multimodal hybrid neural network (MHNN) that encodes and integrates multimodal cues from both conventional and neuromorphic sensors. Specifically, to encode different sensory cues, we built various neural networks of spatial view cells, place cells, head direction cells, and time cells. To integrate these cues, we designed a multiscale liquid state machine that can process and fuse multimodal information effectively and asynchronously using diverse neuronal dynamics and bioinspired inhibitory circuits. We deployed the MHNN on Tianjic, a hybrid neuromorphic chip, and integrated it into a quadruped robot. Our results show that NeuroGPR achieves better performance compared with conventional and existing biologically inspired approaches, exhibiting robustness to diverse environmental uncertainty, including perceptual aliasing, motion blur, light, or weather changes. Running NeuroGPR as an overall multi-neural network workload on Tianjic showcases its advantages with 10.5 times lower latency and 43.6% lower power consumption than the commonly used mobile robot processor Jetson Xavier NX.