Can AI chatbots accurately answer patient questions regarding vasectomies?

• Published in: International journal of impotence research 2024-08
• Main Authors: Mouhawasse, Edwin, Haff, Christopher W, Kumar, Preet, Lack, Benjamin, Chu, Kevin, Bansal, Utsav, Dubin, Justin M
• Format: Article
• Language: English
• ISSN: 0955-9930; 1476-5489; 1476-5489
• DOI: 10.1038/s41443-024-00970-y

Artificial Intelligence (AI) has revolutionized the healthcare industry. There have been limited studies assessing AI model efficacy and accuracy in urology. To our knowledge, there is a lack in research looking at one of the most common urological procedures: the vasectomy. Ten frequently asked questions regarding vasectomies were individually entered into three different AI sources (ChatGPT, Bard & Bing) using free interfaces available to consumers. The responses were critically analyzed by three urologists and graded on a scale of 1 to 4 for clarity, accuracy, and evidence-based information, with 1 being the best and 4 being the worst. ChatGPT had the best average rating per question at 1.367, followed by Bard at 2.167 and Bing at 1.800(p = 0.000083). ChatGPT was found to provide significantly more satisfactory answers than both Bard (p = 0.00005) and Bing (p = 0.03988). The difference between Bard and Bing however was found to be insignificant (p = 0.09651). Overall, our study shows that AI Chatbots may provide mostly accurate information on frequently asked questions regarding vasectomies and is a reasonable resource for patients interested in the procedure to use. ChatGPT is the most accurate and concise of the chatbots assessed.