Enhancing Automated Scoring of Math Self-Explanation Quality Using LLM-Generated Datasets: A Semi-Supervised Approach

• Published in: Computers (Basel) 2023-11, Vol.12 (11), p.217
• Main Authors: Nakamoto, Ryosuke, Flanagan, Brendan, Yamauchi, Taisei, Dai, Yiling, Takami, Kyosuke, Ogata, Hiroaki
• Format: Article
• Language: English
• Subjects: Accuracy; Analysis; automated scoring; Automation; Computational linguistics; data augmentation; Datasets; Education; Language; language learning model (LLM); Language processing; Large language models; Machine learning; Mechanization; Model accuracy; Natural language interfaces; Neural networks; Problem solving; self-explanation; semi-supervised learning; Students; Synthetic data
• ISSN: 2073-431X; 2073-431X
• DOI: 10.3390/computers12110217

In the realm of mathematics education, self-explanation stands as a crucial learning mechanism, allowing learners to articulate their comprehension of intricate mathematical concepts and strategies. As digital learning platforms grow in prominence, there are mounting opportunities to collect and utilize mathematical self-explanations. However, these opportunities are met with challenges in automated evaluation. Automatic scoring of mathematical self-explanations is crucial for preprocessing tasks, including the categorization of learner responses, identification of common misconceptions, and the creation of tailored feedback and model solutions. Nevertheless, this task is hindered by the dearth of ample sample sets. Our research introduces a semi-supervised technique using the large language model (LLM), specifically its Japanese variant, to enrich datasets for the automated scoring of mathematical self-explanations. We rigorously evaluated the quality of self-explanations across five datasets, ranging from human-evaluated originals to ones devoid of original content. Our results show that combining LLM-based explanations with mathematical material significantly improves the model’s accuracy. Interestingly, there is an optimal limit to how many synthetic self-explanation data can benefit the system. Exceeding this limit does not further improve outcomes. This study thus highlights the need for careful consideration when integrating synthetic data into solutions, especially within the mathematics discipline.