Accuracy assessment of ChatGPT responses to frequently asked questions regarding anterior cruciate ligament surgery

• Published in: The knee 2024-12, Vol.51, p.84-92
• Main Authors: Villarreal-Espinosa, Juan Bernardo, Berreta, Rodrigo Saad, Allende, Felicitas, Garcia, José Rafael, Ayala, Salvador, Familiari, Filippo, Chahla, Jorge
• Format: Article
• Language: English
• Subjects: ACL surgery; Anterior Cruciate Ligament Injuries - surgery; Anterior Cruciate Ligament Reconstruction; ChatGPT; Fellowships and Scholarships; Frequently asked questions (FAQs); Humans; Patient education; Reproducibility of Results; Surveys and Questionnaires
• ISSN: 0968-0160; 1873-5800; 1873-5800
• DOI: 10.1016/j.knee.2024.08.014

The emergence of artificial intelligence (AI) has allowed users to have access to large sources of information in a chat-like manner. Thereby, we sought to evaluate ChatGPT-4 response’s accuracy to the 10 patient most frequently asked questions (FAQs) regarding anterior cruciate ligament (ACL) surgery. A list of the top 10 FAQs pertaining to ACL surgery was created after conducting a search through all Sports Medicine Fellowship Institutions listed on the Arthroscopy Association of North America (AANA) and American Orthopaedic Society of Sports Medicine (AOSSM) websites. A Likert scale was used to grade response accuracy by two sports medicine fellowship-trained surgeons. Cohen’s kappa was used to assess inter-rater agreement. Reproducibility of the responses over time was also assessed. Five of the 10 responses received a ‘completely accurate’ grade by two-fellowship trained surgeons with three additional replies receiving a ‘completely accurate’ status by at least one. Moreover, inter-rater reliability accuracy assessment revealed a moderate agreement between fellowship-trained attending physicians (weighted kappa = 0.57, 95% confidence interval 0.15–0.99). Additionally, 80% of the responses were reproducible over time. ChatGPT can be considered an accurate additional tool to answer general patient questions regarding ACL surgery. None the less, patient–surgeon interaction should not be deferred and must continue to be the driving force for information retrieval. Thus, the general recommendation is to address any questions in the presence of a qualified specialist.