Mammalian-brain-inspired neuromorphic motion-cognition nerve achieves cross-modal perceptual enhancement

• Published in: Nature communications 2023-03, Vol.14 (1), p.1344-1344, Article 1344
• Main Authors: Jiang, Chengpeng, Liu, Jiaqi, Ni, Yao, Qu, Shangda, Liu, Lu, Li, Yue, Yang, Lu, Xu, Wentao
• Format: Article
• Language: English
• Subjects: 639/301/1005/1007; 639/301/357/995; 639/925/927/1007; Animal cognition; Animal species; Animals; Automation; Brain; Brain - physiology; Carrier mobility; Cognition; Cross-modal; Cues; Current carriers; Fabrication; Human motion; Humanities and Social Sciences; Humans; Mammals; Manufacturing engineering; Motion detection; multidisciplinary; Nanoparticles; Nerves; Neuromorphic computing; Perception; Robotics; Science; Science (multidisciplinary); Sensory integration; Space Perception; Spatial discrimination; Thin films; Vestibular system; Visual Perception - physiology
• ISSN: 2041-1723; 2041-1723
• DOI: 10.1038/s41467-023-36935-w

Perceptual enhancement of neural and behavioral response due to combinations of multisensory stimuli are found in many animal species across different sensory modalities. By mimicking the multisensory integration of ocular-vestibular cues for enhanced spatial perception in macaques, a bioinspired motion-cognition nerve based on a flexible multisensory neuromorphic device is demonstrated. A fast, scalable and solution-processed fabrication strategy is developed to prepare a nanoparticle-doped two-dimensional (2D)-nanoflake thin film, exhibiting superior electrostatic gating capability and charge-carrier mobility. The multi-input neuromorphic device fabricated using this thin film shows history-dependent plasticity, stable linear modulation, and spatiotemporal integration capability. These characteristics ensure parallel, efficient processing of bimodal motion signals encoded as spikes and assigned with different perceptual weights. Motion-cognition function is realized by classifying the motion types using mean firing rates of encoded spikes and postsynaptic current of the device. Demonstrations of recognition of human activity types and drone flight modes reveal that the motion-cognition performance match the bio-plausible principles of perceptual enhancement by multisensory integration. Our system can be potentially applied in sensory robotics and smart wearables. Inspired by the multisensory cue integration in macaque’s brain for spatial perception, the authors develop a neuromorphic motion-cognition nerve that achieves cross-modal perceptual enhancement for robotics and wearable applications.