The effects of survey version on patient experience scores and plan rankings

• Published in: Health services research 2019-10, Vol.54 (5), p.1016-1022
• Main Authors: Beckett, Megan K., Elliott, Marc N., Burkhart, Q, Cleary, Paul D., Orr, Nate, Brown, Julie A., Gaillot, Sarah, Liu, Karin, Hays, Ron D.
• Format: Article
• Language: English
• Subjects: Adult; Aged; Aged, 80 and over; Beneficiaries; Clinical assessment; Female; Government programs; Health Care Surveys - standards; Humans; Male; Market surveys; Measuring Patient Experience; Medical research; Medicare; Medicare Part C - statistics & numerical data; Middle Aged; patient; Patient Satisfaction - statistics & numerical data; Polls & surveys; Quality of Health Care - statistics & numerical data; Questions; Random effects; Regression analysis; Statistical analysis; survey research and questionnaire design; Surveys; United States
• ISSN: 0017-9124; 1475-6773
• DOI: 10.1111/1475-6773.13172

Objective To assess the effect of changing survey questions on plan‐level patient experience measures and ratings. Data Source 2015 Medicare Advantage CAHPS Survey respondents. Study Design Ninety three randomly selected beneficiaries in each of 40 MA plans received a revised (5.0) CAHPS survey; 38 832 beneficiaries received version 4.0. Linear mixed‐effect regression predicted CAHPS measures from fixed effects for survey version and beneficiary characteristics and random effects for plan and plan‐by‐version random slope. Principal Findings Response rates were 42 percent for both versions. Removal of “try to” from screeners increased the percentage of respondents eligible for follow‐up questions. Version 5.0 caused a small increase (1‐3 points on a 0‐100 scale, P 3 points) in one. There was a small statistically significant increase in two unaltered measures. These changes were uniform across plans, so there would be no expected change compared to results using the legacy survey in the score distributions other than uniform mean shifts, and no expected effect on summary measures. Conclusions These analyses illustrate how to assess the impact of seemingly minor survey modifications for other national surveys considering changes and highlight the importance of screeners in instrument design.