The three ghosts of medical AI: Can the black-box present deliver?

• Published in: Artificial intelligence in medicine 2022-02, Vol.124, p.102158-102158, Article 102158
• Main Authors: Quinn, Thomas P., Jacobs, Stephan, Senadeera, Manisha, Le, Vuong, Coghlan, Simon
• Format: Article
• Language: English
• Subjects: Artificial Intelligence; Autonomy; Black-box; Challenges; Delivery of Health Care; Ethics; Humans; Machine Learning; Reproducibility of Results; Transparency; Trust; Xai
• ISSN: 0933-3657; 1873-2860
• DOI: 10.1016/j.artmed.2021.102158

Our title alludes to the three Christmas ghosts encountered by Ebenezer Scrooge in A Christmas Carol, who guide Ebenezer through the past, present, and future of Christmas holiday events. Similarly, our article takes readers through a journey of the past, present, and future of medical AI. In doing so, we focus on the crux of modern machine learning: the reliance on powerful but intrinsically opaque models. When applied to the healthcare domain, these models fail to meet the needs for transparency that their clinician and patient end-users require. We review the implications of this failure, and argue that opaque models (1) lack quality assurance, (2) fail to elicit trust, and (3) restrict physician-patient dialogue. We then discuss how upholding transparency in all aspects of model design and model validation can help ensure the reliability and success of medical AI. •Modern machine learning relies on powerful but intrinsically opaque models.•When applied to the healthcare domain, these models fail to meet the needs for transparency.•We review how opaque models lack quality assurance, fail to elicit trust, and restrict physician-patient dialogue.•We then discuss how transparency in model design and validation can help ensure the reliability of medical AI.