BrainQN: Enhancing the Robustness of Deep Reinforcement Learning with Spiking Neural Networks

• Published in: Advanced intelligent systems 2024-09, Vol.6 (9), p.n/a
• Main Authors: Feng, Shuo, Cao, Jian, Ou, Zehong, Chen, Guang, Zhong, Yi, Wang, Zilin, Yan, Juntong, Chen, Jue, Wang, Bingsen, Zou, Chenglong, Feng, Zebang, Wang, Yuan
• Format: Article
• Language: English
• Subjects: Algorithms; Artificial neural networks; Brain; Decision making; Deep learning; deep reinforcement learning; efficiency; Energy consumption; Energy efficiency; Low noise; Machine learning; Network latency; Neural networks; neuromorphic chips; Neurons; Power; Power consumption; Robustness; Sparsity; spiking neural networks
• ISSN: 2640-4567; 2640-4567
• DOI: 10.1002/aisy.202400075

As the third‐generation network succeeding artificial neural networks (ANNs), spiking neural networks (SNNs) offer high robustness and low energy consumption. Inspired by biological systems, the limitations of low robustness and high‐power consumption in deep reinforcement learning (DRL) are addressed by introducing SNNs. The Brain Q‐network (BrainQN) is proposed, which replaces the neurons in the classic Deep Q‐learning (DQN) algorithm with SNN neurons. BrainQN is trained using surrogate gradient learning (SGL) and ANN‐to‐SNN conversion methods. Robustness tests with input noise reveal BrainQN's superior performance, achieving an 82.14% increase in rewards under low noise and 71.74% under high noise compared to DQN. These findings highlight BrainQN's robustness and superior performance in noisy environments, supporting its application in complex scenarios. SGL‐trained BrainQN is more robust than ANN‐to‐SNN conversion under high noise. The differences in network output correlations between noisy and original inputs, along with training algorithm distinctions, explain this phenomenon. BrainQN successfully transitioned from a simulated Pong environment to a ball‐catching robot with dynamic vision sensors (DVS). On the neuromorphic chip PAICORE, it shows significant advantages in latency and power consumption compared to Jetson Xavier NX. This article addresses the limitations of deep reinforcement learning (DRL) by introducing spiking neural networks (SNNs). This article proposes the Brain Q‐network (BrainQN), which replaces the neurons in the classic Deep Q‐learning (DQN) with SNN neurons. BrainQN demonstrates excellent performance in terms of robustness against noise attacks and power consumption.