Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill

• Published in: User modeling and user-adapted interaction 2013-03, Vol.23 (1), p.1-39
• Main Authors: Sao Pedro, Michael A., de Baker, Ryan S. J., Gobert, Janice D., Montalvo, Orlando, Nakama, Adam
• Format: Article
• Language: English
• Subjects: Analysis; Artificial intelligence; Behavior; Computer Science; Construction; Data collection; Design; Detectors; Estimates; Estimating; Experiments; Hypotheses; Interactive learning; Knowledge; Learning; Management of Computing and Information Systems; Mathematical models; Middle schools; Multimedia Information Systems; Original Paper; Physical sciences; Run time (computers); School environment; Science education; Sensors; Skills; Students; Studies; User Interfaces and Human Computer Interaction
• ISSN: 0924-1868; 1573-1391
• DOI: 10.1007/s11257-011-9101-0

We present work toward automatically assessing and estimating science inquiry skills as middle school students engage in inquiry within a physical science microworld. Towards accomplishing this goal, we generated machine-learned models that can detect when students test their articulated hypotheses, design controlled experiments, and engage in planning behaviors using two inquiry support tools. Models were trained using labels generated through a new method of manually hand-coding log files, “text replay tagging”. This approach led to detectors that can automatically and accurately identify these inquiry skills under student-level cross-validation. The resulting detectors can be applied at run-time to drive scaffolding intervention. They can also be leveraged to automatically score all practice attempts, rather than hand-classifying them, and build models of latent skill proficiency. As part of this work, we also compared two approaches for doing so, Bayesian Knowledge-Tracing and an averaging approach that assumes static inquiry skill level. These approaches were compared on their efficacy at predicting skill before a student engages in an inquiry activity, predicting performance on a paper-style multiple choice test of inquiry, and predicting performance on a transfer task requiring data collection skills. Overall, we found that both approaches were effective at estimating student skills within the environment. Additionally, the models’ skill estimates were significant predictors of the two types of inquiry transfer tests.