fNIRS-based classification of mind-wandering with personalized window selection for multimodal learning interfaces

• Published in: Journal on multimodal user interfaces 2021-09, Vol.15 (3), p.257-272
• Main Authors: Liu, Ruixue, Walker, Erin, Friedman, Leah, Arrington, Catherine M., Solovey, Erin T.
• Format: Article
• Language: English
• Subjects: Algorithms; Artificial neural networks; Brain; Classification; Classifiers; Computer Science; Feasibility studies; Image Processing and Computer Vision; Infrared spectra; Machine learning; Medical imaging; Near infrared radiation; Neural networks; Original Paper; Signal,Image and Speech Processing; User Interfaces and Human Computer Interaction; Windows (intervals)
• ISSN: 1783-7677; 1783-8738
• DOI: 10.1007/s12193-020-00325-z

Automatic detection of an individual’s mind-wandering state has implications for designing and evaluating engaging and effective learning interfaces. While it is difficult to differentiate whether an individual is mind-wandering or focusing on the task only based on externally observable behavior, brain-based sensing offers unique insights to internal states. To explore the feasibility, we conducted a study using functional near-infrared spectroscopy (fNIRS) and investigated machine learning classifiers to detect mind-wandering episodes based on fNIRS data, both on an individual level and a group level, specifically focusing on automated window selection to improve classification results. For individual-level classification, by using a moving window method combined with a linear discriminant classifier, we found the best windows for classification and achieved a mean F1-score of 74.8%. For group-level classification, we proposed an individual-based time window selection (ITWS) algorithm to incorporate individual differences in window selection. The algorithm first finds the best window for each individual by using embedded individual-level classifiers and then uses these windows from all participants to build the final classifier. The performance of the ITWS algorithm is evaluated when used with eXtreme gradient boosting, convolutional neural networks, and deep neural networks. Our results show that the proposed algorithm achieved significant improvement compared to the previous state of the art in terms of brain-based classification of mind-wandering, with an average F1-score of 73.2%. This builds a foundation for mind-wandering detection for both the evaluation of multimodal learning interfaces and for future attention-aware systems.