Auto-Scoring of Math Self-Explanations by Combining Visual and Language Analysis

• Main Authors: Nakamoto, Ryosuke, Flanagan, Brendan, Dai, Yiling, Takami, Kyosuke, Ogata, Hiroaki
• Format: Conference Proceeding
• Language: English
• Subjects: Accuracy; Adaptation models; Auto-Scoring; CLIP; Education; Mathematical models; Problem-solving; Self-Explanation; Visual Analysis; Visualization
• ISSN: 2161-377X
• DOI: 10.1109/ICALT61570.2024.00054

In the field of mathematics education, self-explanation is recognized as a critical facilitator for learners to articulate their understanding of complex mathematical concepts and problem-solving techniques. With the emergence of digital learning platforms, the potential to utilize such self-explanations for automated evaluation has expanded, yet significant challenges remain. This study introduces a method that integrates vision and language models to enhance the accuracy of automated evaluations of self-explanations in mathematics quizzes. By leveraging the CLIP encoder, we utilize features from both handwritten images and textual self-explanations, aiming to incorporate the characteristics of handwritten solutions that have been overlooked by text-only evaluations. Models were developed to include self-explanations alone (baseline) and those that integrate image features, using both the original and a fine-tuned CLIP encoder adapted to our dataset of self-explanations and handwritten images. Experimental results demonstrated that the model utilizing the fine-tuned CLIP significantly outperformed the baseline, showing a notable reduction in MAE. Conversely, the model employing the original CLIP encoder exhibited decreased performance compared to the baseline, revealing the complex interplay between integrating self-explanations and image features. These findings suggest that the benefits of embedding image features depend on the quality and appropriateness of the visual data incorporated.