NLP-Based Approach for Predicting HMI State Sequences Towards Monitoring Operator Situational Awareness

• Published in: Sensors (Basel, Switzerland) Switzerland), 2020-06, Vol.20 (11), p.3228
• Main Authors: V. P. Singh, Harsh, Mahmoud, Qusay H.
• Format: Article
• Language: English
• Subjects: Human Machine Interface (HMI); human-in-the-loop (HITL); natural language processing (NLP); sequence-to-sequence (seq2seq); situational awareness (SA)
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s20113228

A novel approach presented herein transforms the Human Machine Interface (HMI) states, as a pattern of visual feedback states that encompass both operator actions and process states, from a multi-variate time-series to a natural language processing (NLP) modeling domain. The goal of this approach is to predict operator response patterns for n − a h e a d time-step window given k − l a g g e d past HMI state patterns. The NLP approach offers the possibility of encoding (semantic) contextual relations within HMI state patterns. Towards which, a technique for framing raw HMI data for supervised training using sequence-to-sequence (seq2seq) deep-learning machine translation algorithms is presented. In addition, a custom Seq2Seq convolutional neural network (CNN) NLP model based on current state-of-the-art design elements such as attention, is compared against a standard recurrent neural network (RNN) based NLP model. Results demonstrate comparable effectiveness of both the designs of NLP models evaluated for modeling HMI states. RNN NLP models showed higher ( ≈ 26 % ) forecast accuracy, in general for both in-sample and out-of-sample test datasets. However, custom CNN NLP model showed higher ( ≈ 53 % ) validation accuracy indicative of less over-fitting with the same amount of available training data. The real-world application of the proposed NLP modeling of industrial HMIs, such as in power generating stations control rooms, aviation (cockpits), and so forth, is towards the realization of a non-intrusive operator situational awareness monitoring framework through prediction of HMI states.