Estimating Systemic Cognitive States from a Mixture of Physiological and Brain Signals

• Published in: Topics in cognitive science 2024-07, Vol.16 (3), p.485-526
• Main Authors: Scheutz, Matthias, Aeron, Shuchin, Aygun, Ayca, de Ruiter, J.P., Fantini, Sergio, Fernandez, Cristianne, Haga, Zachary, Nguyen, Thuan, Lyu, Boyang
• Format: Article
• Language: English
• Subjects: Adult; Attention - physiology; Brain - physiology; Classifying cognitive states; Cognition - physiology; Cognitive state inferences; Domain generalization; Electroencephalography - methods; Female; Galvanic Skin Response - physiology; Heart Rate - physiology; Humans; Machine Learning; Male; Physiological and brain sensing; Physiology; Spectroscopy, Near-Infrared - methods; Young Adult
• ISSN: 1756-8757; 1756-8765; 1756-8765
• DOI: 10.1111/tops.12669

As human–machine teams are being considered for a variety of mixed‐initiative tasks, detecting and being responsive to human cognitive states, in particular systematic cognitive states, is among the most critical capabilities for artificial systems to ensure smooth interactions with humans and high overall team performance. Various human physiological parameters, such as heart rate, respiration rate, blood pressure, and skin conductance, as well as brain activity inferred from functional near‐infrared spectroscopy or electroencephalogram, have been linked to different systemic cognitive states, such as workload, distraction, or mind–wandering among others. Whether these multimodal signals are indeed sufficient to isolate such cognitive states across individuals performing tasks or whether additional contextual information (e.g., about the task state or the task environment) is required for making appropriate inferences remains an important open problem. In this paper, we introduce an experimental and machine learning framework for investigating these questions and focus specifically on using physiological and neurophysiological measurements to learn classifiers associated with systemic cognitive states like cognitive load, distraction, sense of urgency, mind wandering, and interference. Specifically, we describe a multitasking interactive experimental setting used to obtain a comprehensive multimodal data set which provided the foundation for a first evaluation of various standard state‐of‐the‐art machine learning techniques with respect to their effectiveness in inferring systemic cognitive states. While the classification success of these standard methods based on just the physiological and neurophysiological signals across subjects was modest, which is to be expected given the complexity of the classification problem and the possibility that higher accuracy rates might not in general be achievable, the results nevertheless can serve as a baseline for evaluating future efforts to improve classification, especially methods that take contextual aspects such as task and environmental states into account. As human‐machine teams are being considered for a variety of mixed‐initiative tasks, detecting and being responsive to human systematic cognitive states is among the most critical capabilities for artificial systems to ensure smooth interactions with humans and high overall team performance. We introduce an experimental and machine learning framework for in