Construct validity of the school-implementation climate scale

• Published in: Grantee Submission 2022-01, Vol.3, p.1-26334895221116065
• Main Authors: Thayer, Andrew J., Cook, Clayton R., Davis, Chayna, Brown, Eric C., Locke, Jill, Ehrhart, Mark G., Aarons, Gregory A., Picozzi, Elissa, Lyon, Aaron R.
• Format: Article
• Language: English
• Subjects: Construct Validity; Educational Environment; Educational Improvement; Evidence Based Practice; Factor Analysis; Organizational Climate; Original Empirical Research; Positive Behavior Supports; Program Implementation; Psychometrics; Test Reliability
• ISSN: 2633-4895; 2633-4895
• DOI: 10.1177/26334895221116065

Background Implementation climate is an organizational construct theorized to facilitate the adoption and delivery of evidence-based practices. Within schools, teachers often are tasked with implementing universal prevention programs. Therefore, they are ideal informants when assessing school implementation climate for initial and continuous implementation improvement efforts. The purpose of this study was to examine the construct validity (i.e., factor structure and convergent/divergent validity) of a school-adapted measure of strategic implementation climate called the School Implementation Climate Scale (SICS). Methods Confirmatory factor analyses of SICS data, collected from 441 teachers in 52 schools, were used to compare uncorrelated and correlated first-order factor models and a second-order hierarchical model. Correlations with other school measures were examined to assess SICS convergent and divergent validities. Results Results demonstrated acceptable internal consistency for each SICS subscale (αs > 0.80 for all subscales) and construct validity of the hypothesized factor structure of the SICS with three new scales. The hierarchical second-order factor structure with eight first-order factors was found to best model the SICS data. Correlations with other school measures were in the expected direction and magnitude. Conclusions Results from this study provide psychometric evidence that supports the use of the SICS to inform the implementation research and practice in schools. Plain Language Summary Schools are busy trying to implement various universal programs and systems to help support kids in their growth. Beginning and sustaining these efforts is quite challenging, and there is need for tools and ideas to help those implementation efforts. One concept is implementation climate, which is broadly the school staff’s perception of the implementation support for a given practice. However, no measure currently exists to help schools assess their implementation climate. The goal of our study was to adapt a measure of implementation climate used in other settings to the school environment. We used feedback from educational experts to make changes and used various analyses to determine if the newly adapted measure was psychometrically sound. Findings suggest the new measure is usable to guide implementation efforts in schools.