Reliability of assessment of medical students’ non-technical skills using a behavioural marker system: does clinical experience matter?

• Published in: BMJ simulation & technology enhanced learning 2021-07, Vol.7 (5), p.285-292
• Main Authors: Clarke, Benjamin, Smith, Samantha E, Phillips, Emma Claire, Hamilton, Ailsa, Kerins, Joanne, Tallentire, Victoria R
• Format: Article
• Language: English
• Subjects: Agreements; Clinical medicine; Cognition & reasoning; Communication; Decision making; Intensive care; Medical students; Original Research; Ratings & rankings; Self awareness; Skills; Statistical analysis; Statistical methods; Teamwork
• ISSN: 2056-6697; 2056-6697
• DOI: 10.1136/bmjstel-2020-000705

IntroductionNon-technical skills are recognised to play an integral part in safe and effective patient care. Medi-StuNTS (Medical Students’ Non-Technical Skills) is a behavioural marker system developed to enable assessment of medical students’ non-technical skills. This study aimed to assess whether newly trained raters with high levels of clinical experience could achieve reliability coefficients of >0.7 and to compare differences in inter-rater reliability of raters with varying clinical experience.MethodsForty-four raters attended a workshop on Medi-StuNTS before independently rating three videos of medical students participating in immersive simulation scenarios. Data were grouped by raters’ levels of clinical experience. Inter-rater reliability was assessed by calculating intraclass correlation coefficients (ICC).ResultsEleven raters with more than 10 years of clinical experience achieved single-measure ICC of 0.37 and average-measures ICC of 0.87. Fourteen raters with more than or equal to 5 years and less than 10 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC of 0.59. Nineteen raters with less than 5 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC 0.65.ConclusionsUsing 11 newly trained raters with high levels of clinical experience produced highly reliable ratings that surpassed the prespecified inter-rater reliability standard; however, a single rater from this group would not achieve sufficiently reliable ratings. This is consistent with previous studies using other medical behavioural marker systems. This study demonstrated a decrease in inter-rater reliability of raters with lower levels of clinical experience, suggesting caution when using this population as raters for assessment of non-technical skills.