MedExpQA: Multilingual benchmarking of Large Language Models for Medical Question Answering

• Published in: Artificial intelligence in medicine 2024-09, Vol.155, p.102938, Article 102938
• Main Authors: Alonso, Iñigo, Oronoz, Maite, Agerri, Rodrigo
• Format: Article
• Language: English
• Subjects: Large Language Models; Medical Question Answering; Multilinguality; Natural Language Processing; Retrieval Augmented Generation
• ISSN: 0933-3657; 1873-2860; 1873-2860
• DOI: 10.1016/j.artmed.2024.102938

Large Language Models (LLMs) have the potential of facilitating the development of Artificial Intelligence technology to assist medical experts for interactive decision support. This potential has been illustrated by the state-of-the-art performance obtained by LLMs in Medical Question Answering, with striking results such as passing marks in licensing medical exams. However, while impressive, the required quality bar for medical applications remains far from being achieved. Currently, LLMs remain challenged by outdated knowledge and by their tendency to generate hallucinated content. Furthermore, most benchmarks to assess medical knowledge lack reference gold explanations which means that it is not possible to evaluate the reasoning of LLMs predictions. Finally, the situation is particularly grim if we consider benchmarking LLMs for languages other than English which remains, as far as we know, a totally neglected topic. In order to address these shortcomings, in this paper we present MedExpQA, the first multilingual benchmark based on medical exams to evaluate LLMs in Medical Question Answering. To the best of our knowledge, MedExpQA includes for the first time reference gold explanations, written by medical doctors, of the correct and incorrect options in the exams. Comprehensive multilingual experimentation using both the gold reference explanations and Retrieval Augmented Generation (RAG) approaches show that performance of LLMs, with best results around 75 accuracy for English, still has large room for improvement, especially for languages other than English, for which accuracy drops 10 points. Therefore, despite using state-of-the-art RAG methods, our results also demonstrate the difficulty of obtaining and integrating readily available medical knowledge that may positively impact results on downstream evaluations for Medical Question Answering. Data, code, and fine-tuned models will be made publicly available.11https://huggingface.co/datasets/HiTZ/MedExpQA. •MedExpQA: the first multilingual benchmark for MedicalQA including gold reference explanations.•Comparison of gold and automatically extracted medical knowledge via RAG techniques.•Fine-tuning makes redundant the external knowledge obtained via RAG.•Overall performance of LLMs with or without RAG still has large room for improvement.•Performance for French, Italian and Spanish lower for every LLM in every setting.