Can ChatGPT outperform a neurosurgical trainee? A prospective comparative study

• Published in: British journal of neurosurgery 2024-02, p.1-10
• Main Authors: Williams, Simon C, Starup-Hansen, Joachim, Funnell, Jonathan P, Hanrahan, John Gerrard, Valetopoulou, Alexandra, Singh, Navneet, Sinha, Saurabh, Muirhead, William R, Marcus, Hani J
• Format: Article
• Language: English
• ISSN: 0268-8697; 1360-046X
• DOI: 10.1080/02688697.2024.2308222

This study aimed to compare the performance of ChatGPT, a large language model (LLM), with human neurosurgical applicants in a neurosurgical national selection interview, to assess the potential of artificial intelligence (AI) and LLMs in healthcare and provide insights into their integration into the field. In a prospective comparative study, a set of neurosurgical national selection-style interview questions were asked to eight human participants and ChatGPT in an online interview. All participants were doctors currently practicing in the UK who had applied for a neurosurgical National Training Number. Interviews were recorded, anonymised, and scored by three neurosurgical consultants with experience as interviewers for national selection. Answers provided by ChatGPT were used as a template for a virtual interview. Interview transcripts were subsequently scored by neurosurgical consultants using criteria utilised in real national selection interviews. Overall interview score and subdomain scores were compared between human participants and ChatGPT. For overall score, ChatGPT fell behind six human competitors and did not achieve a mean score higher than any individuals who achieved training positions. Several factors, including factual inaccuracies and deviations from expected structure and style may have contributed to ChatGPT's underperformance. LLMs such as ChatGPT have huge potential for integration in healthcare. However, this study emphasises the need for further development to address limitations and challenges. While LLMs have not surpassed human performance yet, collaboration between humans and AI systems holds promise for the future of healthcare.