Administration of the American Board of Anesthesiology's virtual APPLIED Examination: successes, challenges, and lessons learned

• Published in: BMC medical education 2024-07, Vol.24 (1), p.749-14, Article 749
• Main Authors: Keegan, Mark T, Harman, Ann E, McLoughlin, Thomas M, Macario, Alex, Deiner, Stacie G, Gaiser, Robert R, Warner, David O, Suresh, Santhanam, Sun, Huaping
• Format: Article
• Language: English
• Subjects: Anesthesiology; Anesthesiology - education; Assessment Centers (Personnel); Candidates; Care and treatment; Certification - standards; Clinical Competence - standards; Communication Strategies; COVID-19; Diagnosis; Educational Measurement - methods; Examiners; Graduate Medical Education; Humans; Infrastructure; Likert Scales; Medical education; Online Surveys; Oral manifestations of general diseases; Pandemics; Practice; Professionalism; Psychometrics; SARS-CoV-2; Scoring; Specialty Boards; Time zones; United States; Virtual APPLIED Examination; Virtual candidate experiences; Virtual examiner experiences; Virtual Objective Structured Clinical Examination; Virtual Standardized Oral Examination
• ISSN: 1472-6920; 1472-6920
• DOI: 10.1186/s12909-024-05694-7

In response to the COVID-19 pandemic, the American Board of Anesthesiology transitioned from in-person to virtual administration of its APPLIED Examination, assessing more than 3000 candidates for certification purposes remotely in 2021. Four hundred examiners were involved in delivering and scoring Standardized Oral Examinations (SOEs) and Objective Structured Clinical Examinations (OSCEs). More than 80% of candidates started their exams on time and stayed connected throughout the exam without any problems. Only 74 (2.5%) SOE and 45 (1.5%) OSCE candidates required rescheduling due to technical difficulties. Of those who experienced "significant issues", concerns with OSCE technical stations (interpretation of monitors and interpretation of echocardiograms) were reported most frequently (6% of candidates). In contrast, 23% of examiners "sometimes" lost connectivity during their multiple exam sessions, on a continuum from minor inconvenience to inability to continue. 84% of SOE candidates and 89% of OSCE candidates described "smooth" interactions with examiners and standardized patients/standardized clinicians, respectively. However, only 71% of SOE candidates and 75% of OSCE candidates considered themselves to be able to demonstrate their knowledge and skills without obstacles. When compared with their in-person experiences, approximately 40% of SOE examiners considered virtual evaluation to be more difficult than in-person evaluation and believed the remote format negatively affected their development as an examiner. The virtual format was considered to be less secure by 56% and 40% of SOE and OSCE examiners, respectively. The retirement of exam materials used virtually due to concern for compromise had implications for subsequent exam development. The return to in-person exams in 2022 was prompted by multiple factors, especially concerns regarding standardization and security. The technology is not yet perfect, especially for testing in-person communication skills and displaying dynamic exam materials. Nevertheless, the American Board of Anesthesiology's experience demonstrated the feasibility of conducting large-scale, high-stakes oral and performance exams in a virtual format and highlighted the adaptability and dedication of candidates, examiners, and administering board staff.