Examining explainable clinical decision support systems with think aloud protocols

• Published in: PloS one 2023-09, Vol.18 (9), p.e0291443
• Main Authors: Anjara, Sabrina G, Janik, Adrianna, Dunford-Stenger, Amy, Mc Kenzie, Kenneth, Collazo-Lorduy, Ana, Torrente, Maria, Costabello, Luca, Provencio, Mariano
• Format: Article
• Language: English
• Subjects: Artificial intelligence; Automation; Computer and Information Sciences; Context; Decision making; Decision support systems; Diagnosis; Engineers; Evidence-based medicine; Learning algorithms; Lung cancer; Lung diseases; Machine learning; Mathematical models; Medical care; Medicine and Health Sciences; Methods; Oncologists; Psychological aspects; Quality management; Technology application; Usability testing; User feedback; Vital signs
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0291443

Machine learning tools are increasingly used to improve the quality of care and the soundness of a treatment plan. Explainable AI (XAI) helps users in understanding the inner mechanisms of opaque machine learning models and is a driver of trust and adoption. Explanation methods for black-box models exist, but there is a lack of user studies on the interpretability of the provided explanations. We used a Think Aloud Protocol (TAP) to explore oncologists’ assessment of a lung cancer relapse prediction system with the aim of refining the purpose-built explanation model for better credibility and utility. Novel to this context, TAP is used as a neutral methodology to elicit experts’ thought processes and judgements of the AI system, without explicit prompts. TAP aims to elicit the factors which influenced clinicians’ perception of credibility and usefulness of the system. Ten oncologists took part in the study. We conducted a thematic analysis of their verbalized responses, generating five themes that help us to understand the context within which oncologists’ may (or may not) integrate an explainable AI system into their working day.