Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study

• Published in: JMIR medical education 2024-04, Vol.10, p.e54067-e54067
• Main Authors: Hudon, Alexandre, Kiepura, Barnabé, Pelletier, Myriam, Phan, Véronique
• Format: Article
• Language: English
• Subjects: Artificial Intelligence; Canada; Chatbots; Chatbots and Conversational Agents; Design; e-Learning and Medical Education; Education, Medical, Undergraduate; Educational materials; Generative Language Models Including ChatGPT; Humans; Hypotheses; Internal validity; Learning; Likert scale; Machine Learning; Medical education; Medical research; Medical students; Multimedia; Multiple choice; New Methods and Approaches in Medical Education; Original Paper; Preliminary Experiences with New Educational Technology; Psychiatrists; Psychiatry; Taxonomy; Teachers; Teaching methods
• ISSN: 2369-3762; 2369-3762
• DOI: 10.2196/54067

Undergraduate medical studies represent a wide range of learning opportunities served in the form of various teaching-learning modalities for medical learners. A clinical scenario is frequently used as a modality, followed by multiple-choice and open-ended questions among other learning and teaching methods. As such, script concordance tests (SCTs) can be used to promote a higher level of clinical reasoning. Recent technological developments have made generative artificial intelligence (AI)-based systems such as ChatGPT (OpenAI) available to assist clinician-educators in creating instructional materials. The main objective of this project is to explore how SCTs generated by ChatGPT compared to SCTs produced by clinical experts on 3 major elements: the scenario (stem), clinical questions, and expert opinion. This mixed method study evaluated 3 ChatGPT-generated SCTs with 3 expert-created SCTs using a predefined framework. Clinician-educators as well as resident doctors in psychiatry involved in undergraduate medical education in Quebec, Canada, evaluated via a web-based survey the 6 SCTs on 3 criteria: the scenario, clinical questions, and expert opinion. They were also asked to describe the strengths and weaknesses of the SCTs. A total of 102 respondents assessed the SCTs. There were no significant distinctions between the 2 types of SCTs concerning the scenario (P=.84), clinical questions (P=.99), and expert opinion (P=.07), as interpretated by the respondents. Indeed, respondents struggled to differentiate between ChatGPT- and expert-generated SCTs. ChatGPT showcased promise in expediting SCT design, aligning well with Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria, albeit with a tendency toward caricatured scenarios and simplistic content. This study is the first to concentrate on the design of SCTs supported by AI in a period where medicine is changing swiftly and where technologies generated from AI are expanding much faster. This study suggests that ChatGPT can be a valuable tool in creating educational materials, and further validation is essential to ensure educational efficacy and accuracy.