Can incorrect artificial intelligence (AI) results impact radiologists, and if so, what can we do about it? A multi-reader pilot study of lung cancer detection with chest radiography

• Published in: European radiology 2023-11, Vol.33 (11), p.8263-8269
• Main Authors: Bernstein, Michael H., Atalay, Michael K., Dibble, Elizabeth H., Maxwell, Aaron W. P., Karam, Adib R., Agarwal, Saurabh, Ward, Robert C., Healey, Terrance T., Baird, Grayson L.
• Format: Article
• Language: English
• Subjects: Artificial Intelligence; Chest; Data science; Diagnostic Radiology; Error reduction; Feedback; Human factors; Humans; Hypotheses; Imaging; Internal Medicine; Interventional Radiology; Lung cancer; Lung Neoplasms - diagnostic imaging; Medicine; Medicine & Public Health; Neuroradiology; Pilot Projects; Radiography; Radiologists; Radiology; Retrospective Studies; Ultrasound; X-rays
• ISSN: 1432-1084; 0938-7994; 1432-1084
• DOI: 10.1007/s00330-023-09747-1

Objective To examine whether incorrect AI results impact radiologist performance, and if so, whether human factors can be optimized to reduce error. Methods Multi-reader design, 6 radiologists interpreted 90 identical chest radiographs (follow-up CT needed: yes/no) on four occasions (09/20–01/22). No AI result was provided for session 1. Sham AI results were provided for sessions 2–4, and AI for 12 cases were manipulated to be incorrect (8 false positives (FP), 4 false negatives (FN)) (0.87 ROC-AUC). In the Delete AI (No Box) condition, radiologists were told AI results would not be saved for the evaluation. In Keep AI (No Box) and Keep AI (Box), radiologists were told results would be saved. In Keep AI (Box), the ostensible AI program visually outlined the region of suspicion. AI results were constant between conditions. Results Relative to the No AI condition (FN = 2.7%, FP = 51.4%), FN and FPs were higher in the Keep AI (No Box) (FN = 33.0%, FP = 86.0%), Delete AI (No Box) (FN = 26.7%, FP = 80.5%), and Keep AI (Box) (FN = to 20.7%, FP = 80.5%) conditions (all p s