ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports

• Published in: European radiology 2024-05, Vol.34 (5), p.2817-2825
• Main Authors: Jeblick, Katharina, Schachtner, Balthasar, Dexl, Jakob, Mittermeier, Andreas, Stüber, Anna Theresa, Topalis, Johanna, Weber, Tobias, Wesp, Philipp, Sabel, Bastian Oliver, Ricke, Jens, Ingrisch, Michael
• Format: Article
• Language: English
• Subjects: Case studies; Chatbots; Diagnostic Radiology; Humans; Imaging; Imaging Informatics and Artificial Intelligence; Internal Medicine; Interventional Radiology; Language; Large language models; Medicine; Medicine & Public Health; Neuroradiology; Patient-Centered Care; Patients; Physician-Patient Relations; Questionnaires; Radiologists; Radiology; Radiology - methods; Radiology Information Systems; Surveys and Questionnaires; Ultrasound
• ISSN: 1432-1084; 0938-7994; 1432-1084
• DOI: 10.1007/s00330-023-10213-1

Objectives To assess the quality of simplified radiology reports generated with the large language model (LLM) ChatGPT and to discuss challenges and chances of ChatGPT-like LLMs for medical text simplification. Methods In this exploratory case study, a radiologist created three fictitious radiology reports which we simplified by prompting ChatGPT with “Explain this medical report to a child using simple language.” In a questionnaire, we tasked 15 radiologists to rate the quality of the simplified radiology reports with respect to their factual correctness, completeness, and potential harm for patients. We used Likert scale analysis and inductive free-text categorization to assess the quality of the simplified reports. Results Most radiologists agreed that the simplified reports were factually correct, complete, and not potentially harmful to the patient. Nevertheless, instances of incorrect statements, missed relevant medical information, and potentially harmful passages were reported. Conclusion While we see a need for further adaption to the medical field, the initial insights of this study indicate a tremendous potential in using LLMs like ChatGPT to improve patient-centered care in radiology and other medical domains. Clinical relevance statement Patients have started to use ChatGPT to simplify and explain their medical reports, which is expected to affect patient-doctor interaction. This phenomenon raises several opportunities and challenges for clinical routine. Key Points • Patients have started to use ChatGPT to simplify their medical reports, but their quality was unknown. • In a questionnaire, most participating radiologists overall asserted good quality to radiology reports simplified with ChatGPT. However, they also highlighted a notable presence of errors, potentially leading patients to draw harmful conclusions. • Large language models such as ChatGPT have vast potential to enhance patient-centered care in radiology and other medical domains. To realize this potential while minimizing harm, they need supervision by medical experts and adaption to the medical field. Graphical Abstract