Illusory generalizability of clinical prediction models

It is widely hoped that statistical models can improve decision-making related to medical treatments. Because of the cost and scarcity of medical outcomes data, this hope is typically based on investigators observing a model's success in one or two datasets or clinical contexts. We scrutinized...

Full description

Saved in:
Bibliographic Details
Published in:Science (American Association for the Advancement of Science) 2024-01, Vol.383 (6679), p.164-167
Main Authors: Chekroud, Adam M, Hawrilenko, Matt, Loho, Hieronimus, Bondar, Julia, Gueorguieva, Ralitza, Hasan, Alkomiet, Kambeitz, Joseph, Corlett, Philip R, Koutsouleris, Nikolaos, Krumholz, Harlan M, Krystal, John H, Paulus, Martin
Format: Article
Language:English
Subjects:
Adolescent
Adult
Aged
Aged, 80 and over
Antipsychotic Agents - therapeutic use
Artificial intelligence
Child
Clinical trials
Datasets
Female
Humans
Machine Learning
Male
Middle Aged
Model testing
Models, Statistical
Patients
Performance prediction
Prediction models
Prognosis
Schizophrenia - drug therapy
Treatment Outcome
Young Adult
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:It is widely hoped that statistical models can improve decision-making related to medical treatments. Because of the cost and scarcity of medical outcomes data, this hope is typically based on investigators observing a model's success in one or two datasets or clinical contexts. We scrutinized this optimism by examining how well a machine learning model performed across several independent clinical trials of antipsychotic medication for schizophrenia. Models predicted patient outcomes with high accuracy within the trial in which the model was developed but performed no better than chance when applied out-of-sample. Pooling data across trials to predict outcomes in the trial left out did not improve predictions. These results suggest that models predicting treatment outcomes in schizophrenia are highly context-dependent and may have limited generalizability.
ISSN:0036-8075
1095-9203
DOI:10.1126/science.adg8538