DeepStealth: Game-Based Learning Stealth Assessment With Deep Neural Networks

• Published in: IEEE transactions on learning technologies 2020-04, Vol.13 (2), p.312-325
• Main Authors: Min, Wookhee, Frankosky, Megan H., Mott, Bradford W., Rowe, Jonathan P., Smith, Andy, Wiebe, Eric, Boyer, Kristy Elizabeth, Lester, James C.
• Format: Article
• Language: English
• Subjects: Adaptation models; Artificial Intelligence; Artificial neural networks; Bayesian analysis; Bayesian Statistics; Computation; Computational modeling; Computational thinking; Computer science; Conditional random fields; Deep learning; Educational Environment; Educational Games; Game Based Learning; Games; Hidden Markov models; Machine learning; Middle School Students; Neural networks; Performance prediction; Predictive models; Predictor Variables; stealth assessment; Student Evaluation; Students; Task analysis; Thinking Skills
• ISSN: 1939-1382; 1939-1382; 2372-0050
• DOI: 10.1109/TLT.2019.2922356

A distinctive feature of game-based learning environments is their capacity for enabling stealth assessment. Stealth assessment analyzes a stream of fine-grained student interaction data from a game-based learning environment to dynamically draw inferences about students' competencies through evidence-centered design. In evidence-centered design, evidence models have been traditionally designed using statistical rules authored by domain experts that are encoded using Bayesian networks. This article presents DeepStealth, a deep learning-based stealth assessment framework, that yields significant reductions in the feature engineering labor that has previously been required to create stealth assessments. DeepStealth utilizes end-to-end trainable deep neural network-based evidence models. Using this framework, evidence models are devised using a set of predictive features captured from raw, low-level interaction data to infer evidence for competencies. We investigate two deep learning-based evidence models, long short-term memory networks (LSTMs) and n-gram encoded feedforward neural networks (FFNNs). We compare these models' predictive performance for inferring students' knowledge to linear-chain conditional random fields (CRFs) and naïve Bayes models. We perform feature set-level analyses of game trace logs and external pre-learning measures, and we examine the models' early prediction capacity. The framework is evaluated using data collected from 182 middle school students interacting with a game-based learning environment for middle grade computational thinking. Results indicate that LSTM-based stealth assessors outperform competitive baseline approaches with respect to predictive accuracy and early prediction capacity. We find that LSTMs, FFNNs, and CRFs all benefit from combined feature sets derived from both game trace logs and external pre-learning measures.