Loading…

A chat about actinic keratosis: Examining capabilities and user experience of ChatGPT as a digital health technology in dermato‐oncology

Background The potential applications of artificial intelligence (AI) in dermatology are evolving rapidly. Chatbots are an emerging trend in healthcare that rely on large language models (LLMs) to generate answers to prompts from users. However, the factuality and user experience (UX) of such chatbo...

Full description

Saved in:
Bibliographic Details
Published in:JEADV clinical practice 2024-03, Vol.3 (1), p.258-265
Main Authors: Lent, Heather C., Ortner, Vinzent K., Karmisholt, Katrine E., Wiegell, Stine R., Nissen, Christoffer V., Omland, Silje H., Kamstrup, Maria R., Togsverd‐Bo, Katrine, Haedersdal, Merete
Format: Article
Language:English
Subjects:
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Background The potential applications of artificial intelligence (AI) in dermatology are evolving rapidly. Chatbots are an emerging trend in healthcare that rely on large language models (LLMs) to generate answers to prompts from users. However, the factuality and user experience (UX) of such chatbots remain to be evaluated in the context of dermato‐oncology. Objectives To examine the potential of Chat Generative Pretrained Transformer (ChatGPT) as a reliable source of information in the context of actinic keratosis (AK) and to evaluate clinicians' attitudes and UX with regard to the chatbot. Methods A set of 38 clinical questions were compiled and entered as natural language queries in separate, individual conversation threads in ChatGPT (OpenAI, default GPT 3.5). Questions pertain to patient education, diagnosis, and treatment. ChatGPT's responses were presented to a panel of 7 dermatologists for rating of factual accuracy, currency of information, and completeness of the response. Attitudes towards ChatGTP were explored qualitatively and quantitatively using a validated user experience questionnaire (UEQ). Results ChatGPT answered 12 questions (31.6%) with an accurate, current, and complete response. ChatGPT performed best for questions on patient education, including pathogenesis of AK and potential risk factors, but struggled with diagnosis and treatment. Major deficits were seen in grading AK, providing up‐to‐date treatment guidance, and asserting incorrect information with unwarranted confidence. Further, responses were considered verbose with an average word count of 198 (SD 55) and overly alarming of the risk of malignant transformation. Based on UEQ responses, the expert panel considered ChatGPT an attractive and efficient tool, scoring highest for speed of information retrieval, but deemed the chatbot inaccurate and verbose, scoring lowest for clarity. Conclusions While dermatologists rated ChatGPT high in UX, the underlying LLMs that enable such chatbots require further development to guarantee accuracy and concision required in a clinical setting.
ISSN:2768-6566
2768-6566
DOI:10.1002/jvc2.263