Multimodal integration for data-driven classification of mental fatigue during construction equipment operations: Incorporating electroencephalography, electrodermal activity, and video signals

• Published in: Developments in the built environment 2023-10, Vol.15, p.100198, Article 100198
• Main Authors: Mehmood, Imran, Li, Heng, Umer, Waleed, Arsalan, Aamir, Anwer, Shahnawaz, Mirza, Mohammed Aquil, Ma, Jie, Antwi-Afari, Maxwell Fordjour
• Format: Article
• Language: English
• Subjects: Construction equipment operators; Construction safety; Machine learning; Mental fatigue; Multimodal data
• ISSN: 2666-1659; 2666-1659
• DOI: 10.1016/j.dibe.2023.100198

Construction equipment operations that require high levels of attention can cause mental fatigue, which can lead to inefficiencies and accidents. Previous studies classified mental fatigue using single-modal data with acceptable accuracy. However, mental fatigue is a multimodal problem, and no single modality is superior. Moreover, none of the previous studies in construction industry have investigated multimodal data fusion for classifying mental fatigue and whether such an approach would improve mental fatigue detection. This study proposes a novel approach using three machine learning models and multimodal data fusion to classify mental fatigue states. Electroencephalography, electrodermal activity, and video signals were acquired during an excavation operation, and the decision tree model using multimodal sensor data fusion outperformed other models with 96.2% accuracy and 96.175%–98.231% F1 scores. Multimodal sensor data fusion can aid in the development of a real-time system to classify mental fatigue and improve safety management at construction sites. •Mental fatigue states were classified among equipment operators.•The EEG, EDA, and facial feature data of the operators were acquired during construction operations.•The performance of the three machine learning models was assessed.•High accuracy was achieved by integrating the data from all the sensors.•The decision tree classifier achieved the best performance with an accuracy of 96.2%.