A Brain-Robot Interaction System by Fusing Human and Machine Intelligence

• Published in: IEEE transactions on neural systems and rehabilitation engineering 2019-03, Vol.27 (3), p.533-542
• Main Authors: Mao, Xiaoqian, Li, Wei, Lei, Chengwei, Jin, Jing, Duan, Feng, Chen, Sherry
• Format: Article
• Language: English
• Subjects: Algorithms; Artificial Intelligence; Automation; Brain; Brain robot interaction; Brain-Computer Interfaces; Cameras; Electroencephalography; Event-related potentials; Event-Related Potentials, P300 - physiology; Evoked Potentials, Somatosensory - physiology; Fuzzy Logic; human intelligence; Humanoid; Humanoid robots; Humans; Hybrid systems; Image processing; Image Processing, Computer-Assisted; improved fuzzy color extractor (IFCE); Intelligence; Machine intelligence; multi-sensor fusion; Multisensor fusion; Robot control; Robot sensing systems; Robotics - methods; Robots; Task analysis; Vision, Ocular - physiology; Visual evoked potentials
• ISSN: 1534-4320; 1558-0210
• DOI: 10.1109/TNSRE.2019.2897323

This paper presents a new brain-robot interaction system by fusing human and machine intelligence to improve the real-time control performance. This system consists of a hybrid P300 and steady-state visual evoked potential (SSVEP) mode conveying a human being's intention, and the machine intelligence combining a fuzzy-logic-based image processing algorithm with multi-sensor fusion technology. A subject selects an object of interest via P300, and the classification algorithm transfers the corresponding parameters to an improved fuzzy color extractor for object extraction. A central vision tracking strategy automatically guides the NAO humanoid robot to the destination selected by the subject intentions represented by brainwaves. During this process, human supervises the system at high level, while machine intelligence assists the robot in accomplishing tasks by analyzing image feeding back from the camera, distance monitoring using out-of-gauge alarms from sonars, and collision detecting from bumper sensors. In this scenario, the SSVEP takes over the situations in which the machine intelligence cannot make decisions. The experimental results show that the subjects can control the robot to a destination of interest, with fewer commands than only using a brain-robot interface. Therefore, the fusion of human and machine intelligence greatly alleviates the brain load and enhances the robot executive efficiency of a brain-robot interaction system.