Multiscale Temporal Self-Attention and Dynamical Graph Convolution Hybrid Network for EEG-Based Stereogram Recognition

• Published in: IEEE transactions on neural systems and rehabilitation engineering 2022, Vol.30, p.1191-1202
• Main Authors: Shen, Lili, Sun, Mingyang, Li, Qunxia, Li, Beichen, Pan, Zhaoqing, Lei, Jianjun
• Format: Article
• Language: English
• Subjects: Attention; Brain modeling; Classification; Convolution; Datasets; Depth perception; DRDS; dynamical graph convolution; EEG; Electrodes; Electroencephalography; Feature extraction; Human performance; Modules; multi-scale convolution; Recognition; self-attention; Space perception; Stereograms; Strabismus; Sun; Task analysis; Temporal variations
• ISSN: 1534-4320; 1558-0210
• DOI: 10.1109/TNSRE.2022.3173724

Stereopsis is the ability of human beings to get the 3D perception on real scenarios. The conventional stereopsis measurement is based on subjective judgment for stereograms, leading to be easily affected by personal consciousness. To alleviate the issue, in this paper, the EEG signals evoked by dynamic random dot stereograms (DRDS) are collected for stereogram recognition, which can help the ophthalmologists diagnose strabismus patients even without real-time communication. To classify the collected Electroencephalography (EEG) signals, a novel multi-scale temporal self-attention and dynamical graph convolution hybrid network (MTS-DGCHN) is proposed, including multi-scale temporal self-attention module, dynamical graph convolution module and classification module. Firstly, the multi-scale temporal self-attention module is employed to learn time continuity information, where the temporal self-attention block is designed to highlight the global importance of each time segments in one EEG trial, and the multi-scale convolution block is developed to further extract advanced temporal features in multiple receptive fields. Meanwhile, the dynamical graph convolution module is utilized to capture spatial functional relationships between different EEG electrodes, in which the adjacency matrix of each GCN layer is adaptively tuned to explore the optimal intrinsic relationship. Finally, the temporal and spatial features are fed into the classification module to obtain prediction results. Extensive experiments are conducted on collected datasets i.e., SRDA and SRDB, and the results demonstrate the proposed MTS-DGCHN achieves outstanding classification performance compared with the other methods. The datasets are available at https://github.com/YANGeeg/TJU-SRD-datasets and the code is at https://github.com/YANGeeg/MTS-DGCHN .