Uncovering Language Disparity of ChatGPT on Retinal Vascular Disease Classification: Cross-Sectional Study

Benefiting from rich knowledge and the exceptional ability to understand text, large language models like ChatGPT have shown great potential in English clinical environments. However, the performance of ChatGPT in non-English clinical settings, as well as its reasoning, have not been explored in dep...

Full description

Saved in:
Bibliographic Details
Published in:Journal of medical Internet research 2024-01, Vol.26 (2), p.e51926-e51926
Main Authors: Liu, Xiaocong, Wu, Jiageng, Shao, An, Shen, Wenyue, Ye, Panpan, Wang, Yao, Ye, Juan, Jin, Kai, Yang, Jie
Format: Article
Language:English
Subjects:
Ability
Allied Health Personnel
Angiography
Asian People
Blood circulation disorders
Chatbots
Chinese languages
Classification
Computational linguistics
Cross-Sectional Studies
Diabetes
Diabetic retinopathy
Diagnostic imaging
Disease
False information
Hallucinations
Humans
Inference
Language
Language processing
Macular degeneration
Medical diagnosis
Medical personnel
Medical records
Medical research
Medicine, Experimental
Misinformation
Multimedia
Natural language interfaces
Ophthalmology
Original Paper
Robustness
Vascular disease
Vascular Diseases
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Benefiting from rich knowledge and the exceptional ability to understand text, large language models like ChatGPT have shown great potential in English clinical environments. However, the performance of ChatGPT in non-English clinical settings, as well as its reasoning, have not been explored in depth. This study aimed to evaluate ChatGPT's diagnostic performance and inference abilities for retinal vascular diseases in a non-English clinical environment. In this cross-sectional study, we collected 1226 fundus fluorescein angiography reports and corresponding diagnoses written in Chinese and tested ChatGPT with 4 prompting strategies (direct diagnosis or diagnosis with a step-by-step reasoning process and in Chinese or English). Compared with ChatGPT using Chinese prompts for direct diagnosis that achieved an F -score of 70.47%, ChatGPT using English prompts for direct diagnosis achieved the best diagnostic performance (80.05%), which was inferior to ophthalmologists (89.35%) but close to ophthalmologist interns (82.69%). As for its inference abilities, although ChatGPT can derive a reasoning process with a low error rate (0.4 per report) for both Chinese and English prompts, ophthalmologists identified that the latter brought more reasoning steps with less incompleteness (44.31%), misinformation (1.96%), and hallucinations (0.59%) (all P
ISSN:1438-8871
1439-4456
1438-8871
DOI:10.2196/51926