Assessing the readability, reliability, and quality of artificial intelligence chatbot responses to the 100 most searched queries about cardiopulmonary resuscitation: An observational study

• Published in: Medicine (Baltimore) 2024-05, Vol.103 (22), p.e38352
• Main Authors: Ömür Arça, Dilek, Erdemir, İsmail, Kara, Fevzi, Shermatov, Nurgazy, Odacioğlu, Mürüvvet, İbişoğlu, Emel, Hanci, Ferid Baran, Sağiroğlu, Gönül, Hanci, Volkan
• Format: Article
• Language: English
• Subjects: Artificial Intelligence; Cardiopulmonary Resuscitation - methods; Cardiopulmonary Resuscitation - standards; Comprehension; Cross-Sectional Studies; Humans; Reproducibility of Results; Surveys and Questionnaires
• ISSN: 0025-7974; 1536-5964; 1536-5964
• DOI: 10.1097/MD.0000000000038352

This study aimed to evaluate the readability, reliability, and quality of responses by 4 selected artificial intelligence (AI)-based large language model (LLM) chatbots to questions related to cardiopulmonary resuscitation (CPR). This was a cross-sectional study. Responses to the 100 most frequently asked questions about CPR by 4 selected chatbots (ChatGPT-3.5 [Open AI], Google Bard [Google AI], Google Gemini [Google AI], and Perplexity [Perplexity AI]) were analyzed for readability, reliability, and quality. The chatbots were asked the following question: "What are the 100 most frequently asked questions about cardio pulmonary resuscitation?" in English. Each of the 100 queries derived from the responses was individually posed to the 4 chatbots. The 400 responses or patient education materials (PEM) from the chatbots were assessed for quality and reliability using the modified DISCERN Questionnaire, Journal of the American Medical Association and Global Quality Score. Readability assessment utilized 2 different calculators, which computed readability scores independently using metrics such as Flesch Reading Ease Score, Flesch-Kincaid Grade Level, Simple Measure of Gobbledygook, Gunning Fog Readability and Automated Readability Index. Analyzed 100 responses from each of the 4 chatbots. When the readability values of the median results obtained from Calculators 1 and 2 were compared with the 6th-grade reading level, there was a highly significant difference between the groups (P