Comparison of the adequacies of the OSCE and vOSCE to assess the competencies required under Brazilian medical curriculum guidelines: a multicenter study

• Published in: BMC medical education 2025-01, Vol.25 (1), p.54-13, Article 54
• Main Authors: Sampaio Carvalho, Hervaldo, Barros Melo, Maria do Carmo, Kakehasi, Fabiana Maria, Menezes Ferri Liu, Priscila, Gontijo Minafra Silveira Santos, Fernanda, de Melo Martins, Ricardo Luiz, Vale de Castro Monteiro, Marilene, Chaves Campos, Marcelo Esteves, Esteves, Roberto Zonato, Belloni Torsani, Matheus, Lopes Calvo Tiberio, Iolanda de Fátima, Inca Atahualpa Urdiales, Akihito, De Marco Novellino, Andrea Mora, Viana Zuza Diniz, Rosiane, Ferreira Moura, Helena, de Vilhena Toledo, Maria Alice, Machado Saldanha, Célia Regina, Pedrosa Ribeiro Alves Oliveira, Andrea, Zen Tempski, Patricia, de Souza Santos, Itamar, de Arruda Martins, Milton
• Format: Article
• Language: English
• Subjects: Audiovisual Aids; Brazil; Clinical Competence - standards; Competency based education; COVID-19 - epidemiology; Curricula; Curriculum; Delphi Technique; Education; Education, Medical - standards; Educational evaluation; Educational Measurement; Educational Measurement - methods; Educational Measurement - standards; Guidelines as Topic; Humans; Medical Education; SARS-CoV-2; Surveys and Questionnaires
• ISSN: 1472-6920; 1472-6920
• DOI: 10.1186/s12909-024-06631-4

The COVID-19 pandemic has led to the popularization of information and communication technology in medical education. This study aimed to compare the objective structured clinical examination (OSCE) and the virtual objective structured clinical examination (vOSCE), based on expert opinion, as tools for assessing the competencies needed under medical curriculum guidelines in Brazil. In this multicenter study, the suitability levels of the OSCE and vOSCE for assessing the competencies needed under the Brazilian National Curriculum Guidelines (DCNs) were compared. The DCNs encompass five groups of domains, namely, health education; general; health management; public health needs; and individual health needs. Sixteen teaching experts served as judges. They responded to a questionnaire with the provided Likert scores indicating the ability of the OSCE and vOSCE to assess each type of competency. Virtual meetings were held via the Delphi method and a focus group-based approach to enable the experts to reach a consensus. A total of 200 items included in the guidelines (100 for the OSCE and 100 for the vOSCE) were evaluated across a total of 3,200 responses. Analysis via the Likert scale revealed a high proportion of positive evaluations for the use of the vOSCE, with kappa values ​​ranging from 0.4 to 0.72. The values ​​of the absolute agreement (positive view) between examiners and equivalence with the kappa statistic for the adoption of the vOSCE ranged from 0.38 to 0.72. Via the use of virtual meetings (Delphi and focus groups), consensus was reached regarding the capacity of the OSCE and vOSCE to evaluate 97% and 92% of these items, respectively. The vOSCE can be employed to assess relevant competencies. However, it faces limitations regarding 8% (8/100) of the items, and some items (3%) cannot be assessed via either method. The difference in results between the two assessment methodologies (OSCE and vOSCE) is 5%. It is necessary to invest in the creation of instruments that can be used to apply the vOSCE and to conduct cost‒benefit analysis of its broader application in health education institutions. We conclude that the vOSCE is an effective tool for assessing most competencies of medical professionals and students required under Brazilian guidelines.