Inter-rater reliability of healthcare professional skills' portfolio assessments: The Andalusian Agency for Healthcare Quality model

• Published in: Medwave 2014-07, Vol.14 (6), p.e5998-e5998
• Main Authors: Almuedo-Paz, Antonio, Herrera-Usagre, Manuel, Buiza-Camacho, Begoña, Julián-Carrión, José, Carrascosa-Salmoral, María del Pilar, Martín-García, Sheila María, Salguero-Cabalgante, Rocío
• Format: Article
• Language: eng ; spa
• Subjects: accreditation; certification; Clinical Competence; clinical competences; Health Personnel - standards; Humans; Models, Theoretical; Observer Variation; portfolio assessment; quality assurance; Quality of Health Care; Reproducibility of Results; Spain
• ISSN: 0717-6384; 0717-6384
• DOI: 10.5867/medwave.2014.06.5998

This study aims to determine the reliability of assessment criteria used for a portfolio at the Andalusian Agency for Healthcare Quality (ACSA). all competences certification processes, regardless of their discipline. 2010-2011. Three types of tests are used: 368 certificates, 17 895 reports and 22 642 clinical practice reports (N=3 010 candidates). The tests were evaluated in pairs by the ACSA team of raters using two categories: valid and invalid. The percentage agreement in assessments of certificates was 89.9%; for the reports of clinical practice, 85.1%; and for clinical practice reports, 81.7%. The inter-rater agreement coefficients (kappa) ranged from 0.468 to 0.711. The results of this study show that the inter-rater reliability of assessments varies from fair to good. Compared with other similar studies, the results put the reliability of the model in a comfortable position. Criteria were reviewed and progressive automation of evaluations was done.