Is It Safe? Reliability and Validity of Structured Versus Unstructured Child Safety Judgments

• Published in: Child & youth care forum 2017-10, Vol.46 (5), p.745-768
• Main Authors: Bartelink, Cora, de Kwaadsteniet, Leontien, ten Berge, Ingrid J., Witteman, Cilia L. M.
• Format: Article
• Language: English
• Subjects: Behavioral Science and Psychology; Child Abuse; Child and School Psychology; Child psychology; Child Safety; Children; Comparative Analysis; Evaluation; Interrater Reliability; Measures; Occupational safety; Original Paper; Predictions; Predictive Validity; Professionals; Protective factors; Psychology; Reliability; Resilience (Psychology); Risk; Risk Assessment; Risk factors; Safety; Test Reliability; Test Validity; Unsafe; Validity; Value Judgment; Vignettes
• ISSN: 1053-1890; 1573-3319
• DOI: 10.1007/s10566-017-9405-2

Background The LIRIK, an instrument for the assessment of child safety and risk, is designed to improve assessments by guiding professionals through a structured evaluation of relevant signs, risk factors, and protective factors. Objective We aimed to assess the interrater agreement and the predictive validity of professionals’ judgments made with the LIRIK in comparison to unstructured judgments. Method In study 1, professionals made safety and risk judgments for 12 vignettes with the LIRIK (group 1, n  = 36) or without an instrument (group 2, n  = 43). In study 2, we compared professionals’ safety and risk judgments for 370 children made with the LIRIK (group 1, n  = 278) or with no instrument (group 2, n  = 92), with outcomes indicating actual unsafety in files 6 months later. Results In study 1, agreement about safety and risks was poor to moderate in both groups. Differences between groups were small and inconsistent. In study 2, the predictive validity of judgments was weak to moderate in both groups. In neither group had unsafe outcomes increased consistently when unsafety or risks were assessed as higher. Conclusions Judgments made with the LIRIK were not more reliable or valid than unstructured professional judgments. These findings raise important questions about the value of risk assessment instruments and about how professional safety and risk judgments can be improved.