Battle of the bots: a comparative analysis of ChatGPT and bing AI for kidney stone-related questions

Objectives To evaluate and compare the performance of ChatGPT™ (Open AI ® ) and Bing AI™ (Microsoft ® ) for responding to kidney stone treatment-related questions in accordance with the American Urological Association (AUA) guidelines and assess factors such as appropriateness, emphasis on consultin...

Full description

Saved in:
Bibliographic Details
Published in:World journal of urology 2024-10, Vol.42 (1), p.600, Article 600
Main Authors: McMahon, Amber K., Terry, Russell S., Ito, Willian E., Molina, Wilson R., Whiles, Bristol B.
Format: Article
Language:English
Subjects:
Calculi
Chatbots
Comparative analysis
Guideline Adherence
Health care
Humans
Kidney Calculi - surgery
Kidney Calculi - therapy
Kidney stones
Kidneys
Medicine
Medicine & Public Health
Nephrolithiasis
Nephrology
Oncology
Patients
Practice Guidelines as Topic
Quality control
Surveys and Questionnaires
Urology
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Objectives To evaluate and compare the performance of ChatGPT™ (Open AI ® ) and Bing AI™ (Microsoft ® ) for responding to kidney stone treatment-related questions in accordance with the American Urological Association (AUA) guidelines and assess factors such as appropriateness, emphasis on consulting healthcare providers, references, and adherence to guidelines by each chatbot. Methods We developed 20 kidney stone evaluation and treatment-related questions based on the AUA Surgical Management of Stones guideline. Questions were asked to ChatGPT and Bing AI chatbots. We compared their responses utilizing the brief DISCERN tool as well as response appropriateness. Results ChatGPT significantly outperformed Bing AI for questions 1–3, which evaluate clarity, achievement, and relevance of responses (12.77 ± 1.71 vs. 10.17 ± 3.27; p  
ISSN:1433-8726
0724-4983
1433-8726
DOI:10.1007/s00345-024-05326-1