Data-driven Exploration of Engagement with Workplace-based Assessment in the Clinical Skills Domain

• Published in: International journal of artificial intelligence in education 2021-12, Vol.31 (4), p.1022-1052
• Main Authors: Piotrkowicz, Alicja, Wang, Kaiwen, Hallam, Jennifer, Dimitrova, Vania
• Format: Article
• Language: English
• Subjects: A festschrift in honour of Jim Greer; Artificial Intelligence; Clinical Experience; Colleges & universities; Computation; Computer Science; Computers and Education; Curricula; Data analysis; Data Collection; Digital data; Education; Educational Environment; Educational Improvement; Educational Practices; Educational Technology; Evaluation Methods; Feedback; Higher education; Informal Education; Learner Engagement; Learning; Learning Analytics; Learning Processes; Learning Strategies; Lifelong Learning; Mathematical analysis; Medical Education; Medical personnel; Medical Students; Performance Based Assessment; Professionals; Skills; Student participation; Students; Teaching; Teaching Methods; Undergraduate Students; User Interfaces and Human Computer Interaction; Workplace Learning
• ISSN: 1560-4292; 1560-4306
• DOI: 10.1007/s40593-021-00264-0

The paper presents a multi-faceted data-driven computational approach to analyse workplace-based assessment (WBA) of clinical skills in medical education. Unlike formal university-based part of the degree, the setting of WBA can be informal and only loosely regulated, as students are encouraged to take every opportunity to learn from the clinical setting. For clinical educators and placement coordinators it is vital to follow and analyse students’ engagement with WBA while on placements, in order to understand how students are participating in the assessment, and what improvements can be made. We analyse digital data capturing the students’ WBA attempts and comments on how the assessments went, using process mining and text analytics. We compare Year 1 cohorts across three years, focusing on differences between primary vs. secondary care placements. The main contribution of the work presented in this paper is the exploration of computational approaches for multi-faceted, data-driven assessment analytics for workplace learning which includes:(i) a set of features for analysing clinical skills WBA data, (ii) analysis of the temporal aspects ofthat data using process mining, and (iii) utilising text analytics to compare student reflections on WBA. We show how assessment data captured during clinical placements can provide insights about the student engagement and inform the medical education practice. Our work is inspired by Jim Greer’s vision that intelligent methods and techniques should be adopted to address key challenges faced by educational practitioners in order to foster improvement of learning and teaching. In the broader AI in Education context, the paper shows the application of AI methods to address educational challenges in a new informal learning domain - practical healthcare placements in higher education medical training.