Proficiency Levels and Validity Evidence for Scoring Metrics for a Virtual Reality and Inanimate Robotic Surgery Simulation Curriculum

• Published in: Journal of surgical education 2024-04, Vol.81 (4), p.589-596
• Main Authors: Tellez, Juan C., Radi, Imad, Alterio, Rodrigo E., Nagaraj, Madhuri B., Baker, Haley B., Scott, Daniel J., Zeh, Herbert J., Polanco, Patricio M.
• Format: Article
• Language: English
• Subjects: Clinical Competence; Computer Simulation; construct validity; Curriculum; Humans; proficiency-based training; robotic surgery; Robotic Surgical Procedures - education; SimNow; simulation; Virtual Reality
• ISSN: 1931-7204; 1878-7452
• DOI: 10.1016/j.jsurg.2024.01.004

•A standardized and validated robotic surgery curriculum is needed.•There is limited validity data to support Intuitive's latest VR simulator, SimNow.•This analysis generated validity evidence for 15 SimNow tasks and 3 inanimate tasks.•Appropriate curriculum content and scoring metrics may improve learning efficiency. Our institution recently implemented a virtual reality (VR) skills curriculum for general surgery residents using the SimNow simulator. Based on a content alignment study, we revised the curriculum to include only 20 of 33 VR tasks and we added 3 previously validated inanimate tasks. The purpose of this study was to establish expert-derived proficiency levels for all tasks and to evaluate the validity of the scoring for the VR tasks. Two expert robotic surgeons performed 5 repetitions of each VR and inanimate task. The trimmed mean (lowest scoring attempt and outliers [>2 standard deviations] were eliminated) was defined as the expert level for each task. For the VR tasks, expert levels were compared to resident performance to evaluate validity. This study was conducted at the University of Texas Southwestern Medical Center (Dallas, TX), a tertiary care academic teaching hospital. Two expert robotic surgeons participated in this study. The data from 42 residents (PGY2-4) who completed the original curriculum was used to represent novice performance. Comparison of expert levels and resident performance was statistically significant for 15 VR tasks (supporting validity) and approached significance (p = 0.06, 0.09) for 2 VR tasks; expert levels were designated as proficiency levels for these 17 tasks. Group comparisons were clearly not significant (p = 0.2-0.8) for 3 VR tasks; 2 of these 3 tasks were retained as introductory exercises (with 3 repetitions required) and 1 was excluded. For the 3 inanimate tasks, expert levels minus 2 standard deviations were designated as proficiency levels. This analysis generated validity evidence for 15 VR tasks and established expert-derived proficiency levels for 17 VR tasks and 3 inanimate tasks. Our proposed curriculum now consists of 19 VR and 3 inanimate tasks using the selected proficiency levels. We anticipate that this design will maximize curriculum efficiency and effectiveness.