Harnessing large language models to auto-evaluate the student project reports

• Published in: Computers and education. Artificial intelligence 2024-12, Vol.7, p.100268, Article 100268
• Main Authors: Du, Haoze, Jia, Qinjin, Gehringer, Edward, Wang, Xianfang
• Format: Article
• Language: English
• Subjects: Auto-evaluation; CGP-BLCS; CPTB; Large language models; Student project reports
• ISSN: 2666-920X; 2666-920X
• DOI: 10.1016/j.caeai.2024.100268

Addressing the problem of the difficulty in providing timely and reasonable feedback evaluation for student project reports, this paper proposes a method based on LLMs (Large Language Models) that can automatically generate instant feedback evaluations for student project reports. Three LLMs, namely BART (Bidirectional and Auto-Regressive Transformer), CPTB (chatgpt_paraphraser_on_T5_base), and CGP-BLCS (chatgpt-gpt4-prompts-bart-large-cnn-samsum), were designed to generate instant text feedback pre-training models for student project reports. The effectiveness of the feedback was evaluated using ROUGE Metrics, BERT Scores, and human expert evaluations. Experiments showed that the lightweight, fine-tuned BART model, when trained on a larger dataset of 80%, generated effective feedback evaluations for student project reports. When trained on a smaller dataset of 20%, both the BART and CPTB models had unsatisfactory overall performance, while the fine-tuned CGP-BLCS model was able to generate feedback evaluations that approached human-level evaluations. The detailed descriptions of the methods used with the LLMs for generating effective text feedback evaluations for student project reports will be useful to AI computer programmers, researchers, and computer science instructional designers for improving their courses and future research.