Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination

The study aimed to evaluate the performance of two Large Language Models (LLMs): ChatGPT (based on GPT-3.5) and GPT-4 with two temperature parameter values, on the Polish Medical Final Examination (MFE). The models were tested on three editions of the MFE from: Spring 2022, Autumn 2022, and Spring 2...

Full description

Saved in:
Bibliographic Details
Published in:Scientific reports 2023-11, Vol.13 (1), p.20512-13, Article 20512
Main Authors: Rosoł, Maciej, Gąsior, Jakub S., Łaba, Jonasz, Korzeniewski, Kacper, Młyńczak, Marcel
Format: Article
Language:English
Subjects:
639/166
639/705
692/700
Benchmarking
Education, Medical
Humanities and Social Sciences
Humans
Language
multidisciplinary
Performance evaluation
Physical Examination
Poland
Science
Science (multidisciplinary)
Spring
Statistical analysis
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The study aimed to evaluate the performance of two Large Language Models (LLMs): ChatGPT (based on GPT-3.5) and GPT-4 with two temperature parameter values, on the Polish Medical Final Examination (MFE). The models were tested on three editions of the MFE from: Spring 2022, Autumn 2022, and Spring 2023 in two language versions—English and Polish. The accuracies of both models were compared and the relationships between the correctness of answers with the answer’s metrics were investigated. The study demonstrated that GPT-4 outperformed GPT-3.5 in all three examinations regardless of the language used. GPT-4 achieved mean accuracies of 79.7% for both Polish and English versions, passing all MFE versions. GPT-3.5 had mean accuracies of 54.8% for Polish and 60.3% for English, passing none and 2 of 3 Polish versions for temperature parameter equal to 0 and 1 respectively while passing all English versions regardless of the temperature parameter value. GPT-4 score was mostly lower than the average score of a medical student. There was a statistically significant correlation between the correctness of the answers and the index of difficulty for both models. The overall accuracy of both models was still suboptimal and worse than the average for medical students. This emphasizes the need for further improvements in LLMs before they can be reliably deployed in medical settings. These findings suggest an increasing potential for the usage of LLMs in terms of medical education.
ISSN:2045-2322
2045-2322
DOI:10.1038/s41598-023-46995-z