Validity and Interrater Reliability of a Regional Mock Oral Board Examination

• Published in: Journal of surgical education 2013-05, Vol.70 (3), p.402-407
• Main Authors: Falcone, John L., MD, Gagne, Daniel J., MD, Lee, Kenneth K.W., MD, Hamad, Giselle G., MD
• Format: Article
• Language: English
• Subjects: certification; Certification - standards; educational measurement; Educational Measurement - standards; general surgery; General Surgery - education; Humans; Internship and Residency; Interpersonal Skills and Communication; Medical Knowledge; Patient Care; Predictive Value of Tests; regional medical programs; reliability and validity; Reproducibility of Results; Retrospective Studies; specialty boards; Specialty Boards - standards; Surgery; United States
• ISSN: 1931-7204; 1878-7452
• DOI: 10.1016/j.jsurg.2012.12.004

Background There have been decreasing pass rates recently on the American Board of Surgery Certifying Examination (ABSCE). General surgery residents from the University of Pittsburgh Medical Center, the West Penn Allegheny Health System, the Conemaugh Health System, and Mercy Hospital participate in a mock oral board examination, which is similar to the ABSCE. The aims of the study are to compare examinee performance on the mock oral boards with the ABSCE and to evaluate the interrater reliability of examiner pairs. Methods In this retrospective study from 2003 to 2010, outcomes on the mock oral boards and the first attempt of the ABSCE for chief residents were compared for the 4 regional residency programs. Interrater reliability for examiner pairs was evaluated with agreement and kappa statistics. Nonparametric statistics were performed, with α = 0.05. Results A total of 32 of 38 (84.2%) chief residents passed the mock oral boards. The median score for each of the 3 rooms was 6 (clear pass). A total of 37 of 38 (97.4%) residents passed the ABSCE. The sensitivity of the mock oral boards was 83.8%, with a positive predictive value of 96.9%, and an accuracy of 81.6%. A total of 25 of 47 (53.2%) examiner pairs were from the same residency institution, whereas 22 of 47 (46.8%) were from different institutions. The median agreement was 100% (interquartile range (IQR) [100% - 100%]). The median kappa statistic was 1.00 (IQR [0.38-1.00]). The Mann-Whitney U tests showed no difference in agreement or kappa for examiner pairs from the same or from different institutions (p> 0.05). Conclusions The mock oral boards have substantial sensitivity and positive predictive value in relation to the ABSCE. There are also very high levels of interrater agreement and interrater reliability. This regional mock oral board examination is valuable for ABSCE preparation.