In a pilot study, automated real-time systematic review updates were feasible, accurate, and work-saving

• Published in: Journal of clinical epidemiology 2023-01, Vol.153, p.26-33
• Main Authors: Marshall, Iain J., Trikalinos, Thomas A., Soboczenski, Frank, Yun, Hye Sun, Kell, Gregory, Marshall, Rachel, Wallace, Byron C.
• Format: Article
• Language: English
• Subjects: Artificial Intelligence; Automatic pilots; Automation; Clinical trials; Coronaviruses; COVID-19; COVID-19 - epidemiology; COVID-19 - prevention & control; COVID-19 Vaccines; Epidemiology; Evidence based medicine; Humans; Living systematic reviews; Machine learning; Medical research; Natural language processing; Pilot Projects; PubMed; Recall; Reviews; Screening; Searching; Systematic review; Systematic reviews; Vaccination; Vaccines
• ISSN: 0895-4356; 1878-5921
• DOI: 10.1016/j.jclinepi.2022.08.013

The aim of this study is to describe and pilot a novel method for continuously identifying newly published trials relevant to a systematic review, enabled by combining artificial intelligence (AI) with human expertise. We used RobotReviewer LIVE to keep a review of COVID-19 vaccination trials updated from February to August 2021. We compared the papers identified by the system with those found by the conventional manual process by the review team. The manual update searches (last search date July 2021) retrieved 135 abstracts, of which 31 were included after screening (23% precision, 100% recall). By the same date, the automated system retrieved 56 abstracts, of which 31 were included after manual screening (55% precision, 100% recall). Key limitations of the system include that it is limited to searches of PubMed/MEDLINE, and considers only randomized controlled trial reports. We aim to address these limitations in future. The system is available as open-source software for further piloting and evaluation. Our system identified all relevant studies, reduced manual screening work, and enabled rolling updates on publication of new primary research. •We developed a hybrid human expert/artificial intelligence system to keep systematic reviews up to date.•The system continuously surveils PubMed/MEDLINE for new relevant articles, and notifies review authors.•A living abstract is made available, which shows the status of the review in real-time.•In a pilot, the system was effective and reduced workload in a systematic review of COVID-19 vaccination studies.