Assessment of Claims of Improved Prediction Beyond the Framingham Risk Score

• Published in: JAMA : the journal of the American Medical Association 2009-12, Vol.302 (21), p.2345-2352
• Main Authors: Tzoulaki, Ioanna, Liberopoulos, George, Ioannidis, John P. A
• Format: Article
• Language: English
• Subjects: Biological and medical sciences; Clinical outcomes; Coronary Disease; General aspects; Health Status Indicators; Humans; Medical sciences; Medicine; Models, Statistical; Prognosis; Reproducibility of Results; Risk Assessment; Risk Factors; Studies
• ISSN: 0098-7484; 1538-3598
• DOI: 10.1001/jama.2009.1757

CONTEXT With heightened interest in predictive medicine, many studies try to document information that can improve prediction of major clinical outcomes. OBJECTIVE To evaluate the reported design and analysis of studies that examined whether additional predictors improve predictive performance when added to the Framingham risk score (FRS), one of the most widely validated and cited clinical prediction scores. STUDY SELECTION Two independent investigators searched 1908 articles citing the article that described the FRS in 1998 until September 2009 through the ISI Web of Knowledge database. Articles were eligible if they included any analyses comparing the predictive performance of the FRS vs the FRS plus some additional predictor for a prospectively assessed outcome. DATA ANALYSES We recorded information on FRS calculation, modeling of additional predictors, outcomes assessed, population evaluated, subgroup analysis documentation, and flaws in the methods that may have affected the reported improvements in predictive ability. We also evaluated the correlation of reported design and analysis features with the predictive model discrimination and improvements with the additional predictors. RESULTS We evaluated 79 eligible articles. Forty-nine studies (62%) did not calculate the FRS as it has been proposed, 15 (19%) modeled the additional predictor in more than 1 way and presented only the best-fit or area-under-the-curve (AUC) results for only 1 model, 41 (52%) did not examine the original outcome that the FRS was developed for, 33 (42%) studied a population different from what the FRS was intended for, and 25 (32%) claimed improved prediction in 1 subgroup but only 7 (9%) formally tested subgroup differences. Evaluation of independence in multivariable regressions, discrimination in AUC, calibration, and reclassification were reported in 77, 36, 7, and 7 studies, respectively, but these methods were adequately documented in only 60, 13, 4, and 2 studies, respectively. Overall, 63 studies (80%) claimed some improved prediction. Increase in AUC was larger when the predictive performance of the FRS was lower (ρ = −0.57, P