Effect of moderation on rubric criteria for inter-rater reliability in an objective structured clinical examination with real patients

• Published in: Fujita Medical Journal 2022, Vol.8(3), pp.83-87
• Main Authors: Watari, Tetsuro, Koyama, Soichiro, Kato, Yusaku, Paku, Yonho, Kanada, Yoshikiyo, Sakurai, Hiroaki
• Format: Article
• Language: English
• Subjects: Inter-rater agreement; Moderation; Original; OSCE; Real patients; Video assessment
• ISSN: 2189-7247; 2189-7255; 2189-7255
• DOI: 10.20407/fmj.2021-010

Objectives: Objective structured clinical examinations (OSCEs) are used to assess clinical competence in medical education. Evaluations using video-recorded OSCEs are effective in reducing costs in terms of time and human resources. To improve inter-rater reliability, these evaluations undergo moderation in the form of a discussion between the raters to obtain consistency in grading according to the rubric criteria. We examined the effect of moderation related to the rubric criteria on the inter-rater reliability of a video-recorded OSCE with real patients.Methods: Forty OSCE videos in which students performed range-of-motion tests at shoulder abduction on real patients were assessed by two raters. The two raters scored videos 1 to 10 without moderation and videos 11 to 40 with moderation each time. The inter-rater reliability of the OSCE was calculated using the weighted kappa coefficient.Results: The mean scores of the weighted kappa coefficients were 0.49 for videos 1 to 10, 0.57 for videos 11 to 20, 0.66 for videos 21 to 30, and 0.82 for videos 31 to 40.Conclusions: An assessment of video-recorded OSCEs was conducted with real patients in a real clinical setting. Repeated moderation improved the inter-rater reliability. This study suggests the effectiveness of moderation in OSCEs with real patients.