The feasibility of a multi-format Web-based assessment of physicians’ communication skills

• Published in: Patient education and counseling 2011-09, Vol.84 (3), p.359-367
• Main Authors: Kim, Sara, Brock, Douglas M, Hess, Brian J, Holmboe, Eric S, Gallagher, Thomas H, Lipner, Rebecca S, Mazor, Kathleen M
• Format: Article
• Language: English
• Subjects: Assessment; Clinical Competence - standards; Clinical Competence - statistics & numerical data; Communication; Communication skills; Computer based; Doctor-Patient communication; Educational Measurement - methods; Educational Status; Feasibility Studies; Feedback; Female; Health Care Surveys; Humans; Internal Medicine; Internet; Male; Nursing; Online Systems; Physician-Patient Relations; Physicians - standards; Physicians - statistics & numerical data; Practice Patterns, Physicians' - standards; Practice Patterns, Physicians' - statistics & numerical data; Reliability; Reproducibility of Results; Statistics as Topic; Surveys and Questionnaires; United States; Validation; Videotape Recording
• ISSN: 0738-3991; 1873-5134
• DOI: 10.1016/j.pec.2011.04.003

Abstract Objective Little is known about the best approaches and format for measuring physicians’ communication skills in an online environment. This study examines the reliability and validity of scores from two Web-based communication skill assessment formats. Methods We created two online communication skill assessment formats: (a) MCQ (multiple-choice questions) consisting of video-based multiple-choice questions; (b) multi-format including video-based multiple-choice questions with rationales, Likert-type scales, and free text responses of what physicians would say to a patient. We randomized 100 general internists to each test format. Peer and patient ratings collected via the American Board of Internal Medicine (ABIM) served as validity sources. Results Seventy-seven internists completed the tests (MCQ: 38; multi-format: 39). The adjusted reliability was 0.74 for both formats. Excellent communicators, as based on their peer and patient ratings, performed slightly better on both tests than adequate communicators, though this difference was not statistically significant. Physicians in both groups rated test format innovative (4.2 out of 5.0). Conclusion The acceptable reliability and participants’ overall positive experiences point to the value of ongoing research into rigorous Web-based communication skills assessment. Practice implications With efficient and reliable scoring, the Web offers an important way to measure and potentially enhance physicians’ communication skills.