Innovations in Competence Assessment: Design and Initial Validation of the Vignette Matching Assessment Tool (VMAT)

• Published in: Training and education in professional psychology 2021-05, Vol.15 (2), p.106-116
• Main Authors: Gonsalvez, Craig J., Deane, Frank P., Terry, Josephine, Nasstasia, Yasmina, Shires, Alice
• Format: Article
• Language: English
• Subjects: Clinical Psychology; Competence; Female; Human; Male; Professional Competence; Psychometrics; Test Bias; Test Construction; Vignette Measure
• ISSN: 1931-3918; 1931-3926
• DOI: 10.1037/tep0000302

This study responds to an urgent need for innovative approaches to competence assessment in the face of strong evidence that several current instruments may be vulnerable to systematic rating biases. Building on previous research, the current study aimed to (a) design a full catalogue of vignettes for the Vignette Matching Assessment Tool (VMAT); (b) establish a matrix of calibration scores for the catalogue of vignettes by having experts rate each of the vignettes; (c) evaluate its psychometric properties by comparing competence ratings derived from the VMAT with those from a conventional instrument, the Clinical Psychology Practicum Competencies Rating Scale (CΨPRS); and (d) assess whether the VMAT reduced halo and leniency biases. An initial catalogue of 52 vignettes was recalibrated by a large sample of experts (N = 45) to establish a final catalogue of 41 standardized vignettes across 10 domains and multiple stages of competence. The VMAT used the matrix of calibrated vignettes to help supervisors anchor competence trajectories attained by trainees for each of 10 competency domains. The results provide initial support for the validation of the VMAT. The VMAT attenuated halo effects, and most supervisors endorsed the VMAT as their preferred instrument. Compared to the CΨPRS, cutoff scores on the VMAT identified a larger number of trainees with serious competence problems (n = 3 vs. n = 1). The current study has important implications for competence assessments and benchmarking within psychology and valuable cross-disciplinary applications. Public Significance Statement Accurate measurement of competence during and at the end of training is important to ensure trainees are capable of providing quality care to the public. This study describes the creation of a new instrument to measure competence and demonstrates that the new instrument provides more credible assessments than other available instruments. These findings have potential benefits for trainees, training institutions, and the public.