Understanding the effectiveness of automated feedback: Using process data to uncover the role of behavioral engagement

• Published in: Computers and education 2024-12, Vol.223, p.105163, Article 105163
• Main Authors: Schiller, Ronja, Fleckenstein, Johanna, Mertens, Ute, Horbach, Andrea, Meyer, Jennifer
• Format: Article
• Language: English
• Subjects: Automated feedback; Behavioral engagement; Process-oriented measurement; Secondary education
• ISSN: 0360-1315
• DOI: 10.1016/j.compedu.2024.105163

In the last couple of years, feedback research has shifted towards a feedback-as-process approach, taking a learner-centered perspective and focusing on the proactive role of the learner in feedback effectiveness. Process measures can provide new insights into the role of the learner by making learners’ actual behavioral engagement visible. We conducted an experimental study, comparing two groups (feedback vs. no feedback) of English-as-a-foreign-language learners in lower secondary schools (N = 189). The learners completed a writing task and revised it with or without feedback. A second writing task served as a transfer task. Performance was automatically assessed using a scoring algorithm. To determine the level of learners’ behavioral engagement during the text revision, we used the revision time and the edit distance (i.e., a similarity measure) as behavioral measures. Our analyses showed a positive effect of feedback on text revision. We found a full mediation of the effect of feedback on text revision through revision time with an estimated portion of mediation (POM) of .63∗∗∗ and a partial mediation of the feedback effect on text revision through the edit distance with a POM of .30∗∗. We did not find significant mediation effects of either engagement variable regarding performance in a transfer task. Our findings contribute to the understanding of feedback effectiveness, highlighting the central role of learner engagement in the feedback process. •We investigate feedback effectiveness from a process-oriented perspective.•Log-data and computer-linguistic features as objective indicators of engagement.•Feedback is associated with higher levels of engagement during text revision.•Behavioral engagement mediates positive effect of feedback on revision performance.•We did not find an effect of the feedback on a transfer task.