Inter-Rater Agreement of Intensivists Evaluating the Goal Concordance of Preference-Sensitive ICU Interventions

• Published in: Journal of pain and symptom management 2018-09, Vol.56 (3), p.406-413.e3
• Main Authors: Turnbull, Alison E., Sahetya, Sarina K., Colantuoni, Elizabeth, Kweku, Josephine, Nikooie, Roozbeh, Curtis, J. Randall
• Format: Article
• Language: English
• Subjects: Advance directives; Age Factors; Aged; Agreements; Attitude of Health Personnel; Care plans; Clinical research; Cohort Studies; Critical Care; critical care outcomes; critical illness; Female; Goal-concordant care; Group decision making; Humans; Intensive care; Intensive Care Units; Intensive treatment; interobserver variability; Intervention; Length of Stay; Life sustaining treatment; Male; Middle Aged; Objectives; Older people; outcome measures; Patient Care Planning; Patient-Centered Care; Physicians - psychology; Psychometrics; Quality Assurance, Health Care; Random Allocation; Reliability; Reproducibility of Results; Research methodology; Severity; Severity of Illness Index; Sex Factors; Specialization
• ISSN: 0885-3924; 1873-6513
• DOI: 10.1016/j.jpainsymman.2018.06.003

Goal-concordant care has been identified as an important outcome of advance care planning and shared decision-making initiatives. However, validated methods for measuring goal concordance are needed. To estimate the inter-rater reliability of senior critical care fellows rating the goal concordance of preference-sensitive interventions performed in intensive care units (ICUs) while considering patient-specific circumstances as described in a previously proposed methodology. We identified ICU patients receiving preference-sensitive interventions in three adult ICUs at Johns Hopkins Hospital. A simulated cohort was created by randomly assigning each patient one of 10 sets of goals and preferences about limiting life support. Critical care fellows then independently reviewed patient charts and answered two questions: 1) Is this patient's goal achievable? and 2) Will performing this intervention help achieve the patient's goal? When the answer to both questions was yes, the intervention was rated as goal concordant. Inter-rater agreement was summarized by estimating intraclass correlation coefficient using mixed-effects models. Six raters reviewed the charts of 201 patients. Interventions were rated as goal concordant 22%–92% of the time depending on the patient's goal-limitation combination. Percent agreement between pairs of raters ranged from 59% to 86%. The intraclass correlation coefficient for ratings of goal concordance was 0.50 (95% CI 0.31–0.69) and was robust to patient age, gender, ICU, severity of illness, and lengths of stay. Inter-rater agreement between intensivists using a standardized methodology to evaluate the goal concordance of preference-sensitive ICU interventions was moderate. Further testing is needed before this methodology can be recommended as a clinical research outcome.