Randomised controlled trial of students access to resources in an examination

• Published in: Medical education 2021-08, Vol.55 (8), p.951-960
• Main Authors: Tweed, Mike, Desrosiers, Jen, Wilkinson, Tim J.
• Format: Article
• Language: English
• Subjects: Medical education; Medical students; Multiple choice; Students
• ISSN: 0308-0110; 1365-2923
• DOI: 10.1111/medu.14534

Introduction Assessment of healthcare professionals should be authentic to clinical practice. As clinicians regularly use resources in practice, similar resources should be available to those sitting assessment. There is limited information on the impacts of open‐book (resource) assessments on standard setting for use in high‐stakes assessments. This research aims to explore the effects on standard setting and student perceptions when open vs closed resources are available in high‐stakes assessment of medical students. Methods Students sat multiple‐choice question (MCQ) examinations under both closed‐ and open‐resource conditions in a randomised crossover design. A standard setting panel set pass‐marks for both closed‐ and open‐resource conditions of delivery, and we compared these pass‐marks with each other and with actual performance. Students responded to a survey on perceptions of open‐resource assessments. Results The pass‐mark was set higher when panellists considered open‐resource conditions compared to closed conditions (59% vs 47%), but actual student performance showed no difference in scores between the two conditions. The net effect was that the pass rate was higher for closed than open‐resource conditions (71% vs 34%). Open‐resource conditions increased the time to complete the questions. The students perceived that open resource was more authentic but was more time‐consuming and would require different preparation. Regarding the acceptability of including open resources in high‐stakes assessment, the responses of students were mixed. Discussion Pass standards based solely on judgements by panellists experienced in closed‐resource conditions might not be applicable under open conditions. Questions vary in how much time accessing resources may take and the degree of assistance in selecting the correct answer. A programme of assessment could be constructed to include both closed‐ and open‐resource condition assessments depending on the questions’ content and format. Open‐resource conditions may promote assessment preparation that focuses more on seeking and evaluating resources rather than learning facts. When student use of open‐resources is novel a disconnect emerges as standard setting panelists perceive a need to increase the pass‐mark while the availability of resources actually slows student responding, resulting in no effect on score.