Measuring deliberate reflection in residents: validation and psychometric properties of a measurement tool

• Published in: BMC medical education 2023-08, Vol.23 (1), p.1-606, Article 606
• Main Authors: Blum, Richard H, Mai, Christine L, Mitchell, John D, Saddawi-Konefka, Daniel, Cooper, Jeffrey B, Shorten, George, DunnGalvin, Audrey
• Format: Article
• Language: English
• Subjects: Agreements; Anesthesia; Anesthesia Trainees; Anesthesiology; Assessment; Cognition & reasoning; Construct Validity; Educational Experience; Experiential learning; Feedback (Response); Graduate Medical Education; Graduate Students; Hospitals; Independent Study; Interrater Reliability; Knowledge Level; Learning Experience; Learning Processes; Management; Medical education; Metacognition; Methods; Participant Characteristics; Pilot projects; Psychological aspects; Psychometrics; Quantitative psychology; Rating Scales; REFLECT Rubric & Validation; Reflection; Reflective Practice; Residents (Medicine); Scoring Rubrics; Teaching Methods; Thinking Skills; Trainees; Training; Validation studies; Validity; Vignettes
• ISSN: 1472-6920; 1472-6920
• DOI: 10.1186/s12909-023-04536-2

Purpose Reflective capacity is "the ability to understand critical analysis of knowledge and experience to achieve deeper meaning." In medicine, there is little provision for post-graduate medical education to teach deliberate reflection. The feasibility, scoring characteristics, reliability, validation, and adaptability of a modified previously validated instrument was examined for its usefulness assessing reflective capacity in residents as a step toward developing interventions for improvement. Methods Third-year residents and fellows from four anesthesia training programs were administered a slightly modified version of the Reflection Evaluation for Learners' Enhanced Competencies Tool (REFLECT) in a prospective, observational study at the end of the 2019 academic year. Six written vignettes of imperfect anesthesia situations were created. Subjects recorded their perspectives on two randomly assigned vignettes. Responses were scored using a 5-element rubric; average scores were analyzed for psychometric properties. An independent self-report assessment method, the Cognitive Behavior Survey: Residency Level (rCBS) was used to examine construct validity. Internal consistency (ICR, Cronbach's alpha) and interrater reliability (weighted kappa) were examined. Pearson correlations were used between the two measures of reflective capacity. Results 46/136 invited subjects completed 2/6 randomly assigned vignettes. Interrater agreement was high (k = 0.85). The overall average REFLECT score was 1.8 (1-4 scale) with good distribution across the range of scores. ICR for both the REFLECT score (mean 1.8, sd 0.5; [alpha] = 0.92) and the reflection scale of the rCBS (mean 4.5, sd 1.1; [alpha] = 0.94) were excellent. There was a significant correlation between REFLECT score and the rCBS reflection scale (r = .44, p < 0.01). Conclusions This study demonstrates feasibility, reliability, and sufficiently robust psychometric properties of a modified REFLECT rubric to assess graduate medical trainees' reflective capacity and established construct/convergent validity to an independent measure. The instrument has the potential to assess the effectiveness of interventions intended to improve reflective capacity. Keywords: Reflection, Reflective Practice, Anesthesia Trainees, Assessment, REFLECT Rubric & Validation