Development and Evaluation of an Audio Coding System for Assessing Providers' Integrity to Collaborative Problem Solving in Youth-Service Settings

• Published in: Professional psychology, research and practice research and practice, 2022-12, Vol.53 (6), p.640-650
• Main Authors: Pollastri, Alisha R., Wang, Lu, Raftery-Helmer, Jacquelyn N., Hurley, Sarah, Eddy, Christopher J., Sisson, Jocelyn, Thompson, Nicholas, Ablon, J. Stuart
• Format: Article
• Language: English
• Subjects: Behavior problems; Child development; Coding Scheme; Collaboration; Discriminant validity; Evidence Based Practice; Female; Health Personnel; Home based treatment; Human; Integrity; Interrater reliability; Male; Measurement; Morality; Problem Solving; Psychometrics; Quantitative psychology; Rating; Test Construction; Test Reliability; Test Validity; Treatment methods
• ISSN: 0735-7028; 1939-1323
• DOI: 10.1037/pro0000476

Collaborative Problem Solving (CPS) is a widely disseminated, neurobiologically based, trauma-sensitive treatment for children's challenging behavior. Measuring treatment integrity is critical to support implementation and continued research on the approach. This article presents the development and psychometric evaluation of an observational system, the CPS Manualized Expert-Rated Integrity Coding System (CPS-MEtRICS), for measuring CPS treatment integrity. Audio recordings of in-home treatment sessions (159 CPS and 82 treatment-as-usual) were independently rated by pairs of seven trained coders for integrity and the affective aspect of client-provider relationship. Results showed satisfactory interrater reliability (intraclass correlations [ICCs] 0.64-0.86). Additionally, the integrity scores on the CPS-MEtRICS discriminated between providers delivering CPS and treatment-as-usual, while these two groups did not differ in general client-provider affective bond, supporting the construct and discriminant validity of ratings using the system. This study provided evidence for the reliability of the CPS-MEtRICS and the validity of its integrity ratings, which can be used for research on CPS and which will provide the foundation for other CPS integrity measurement tools useful in practice settings. This article also provides a model that can be used when developing integrity measurement systems for other therapeutic approaches. Public Significance Statement This study provides psychometric support for a system that can be used to measure whether practitioners are using the Collaborative Problem Solving (CPS) treatment approach with integrity, or as intended by the developers of the approach. Researchers and practitioners can use this system to advance their skill and understanding of CPS, and its development provides a model to support the creation of integrity measurement systems for other therapeutic approaches.