Hypothetical case replacement can be used to quantify the robustness of trial results

• Published in: Journal of clinical epidemiology 2021-06, Vol.134, p.150-159
• Main Authors: Frank, Kenneth A., Lin, Qinyun, Maroulis, Spiro, Mueller, Anna S., Xu, Ran, Rosenberg, Joshua M., Hayter, Christopher S., Mahmoud, Ramy A., Kolak, Marynia, Dietz, Thomas, Zhang, Lixin
• Format: Article
• Language: English
• Subjects: Antihypertensives; Case replacement; Clinical decision making; Clinical importance; Clinical trials; Confidence intervals; Coronaviruses; COVID-19; Epidemiology; Evidence-based medicine; Fragility; Hydroxychloroquine; Inference; Medical research; Pandemics; Patients; Randomized controlled trials; Robustness; Robustness of findings; Statistical analysis; Statistical significance; Uncertainty
• ISSN: 0895-4356; 1878-5921
• DOI: 10.1016/j.jclinepi.2021.01.025

•We introduce a case replacement framework for sensitivity analysis of clinical trials.•The framework supports statements such as “The inference would change if xx of the treatment patients who experienced positive outcomes were replaced by hypothetical patients who did not receive a treatment.”•The framework complements the Fragility Index by accounting for the rarity of negative outcomes. For example, large case replacement is required when the Fragility Index is small but negative outcomes are rare.•The framework can be used for any threshold, including minimally important differences and statistical significance.•The framework applies to a broad set of models and research designs. We apply a general case replacement framework for quantifying the robustness of causal inferences to characterize the uncertainty of findings from clinical trials. We express the robustness of inferences as the amount of data that must be replaced to change the conclusion and relate this to the fragility of trial results used for dichotomous outcomes. We illustrate our approach in the context of an RCT of hydroxychloroquine on pneumonia in COVID-19 patients and a cumulative meta-analysis of the effect of antihypertensive treatments on stroke. We developed the Robustness of an Inference to Replacement (RIR), which quantifies how many treatment cases with positive outcomes would have to be replaced with hypothetical patients who did not receive a treatment to change an inference. The RIR addresses known limitations of the Fragility Index by accounting for the observed rates of outcomes. It can be used for varying thresholds for inference, including clinical importance. Because the RIR expresses uncertainty in terms of patient experiences, it is more relatable to stakeholders than P-values alone. It helps identify when results are statistically significant, but conclusions are not robust, while considering the rareness of events in the underlying data.