BioMedGPT: An Open Multimodal Large Language Model for BioMedicine

• Published in: IEEE journal of biomedical and health informatics 2024-11, p.1-12
• Main Authors: Luo, Yizhen, Zhang, Jiahuan, Fan, Siqi, Yang, Kai, Hong, Massimo, Wu, Yushuai, Qiao, Mu, Nie, Zaiqing
• Format: Article
• Language: English
• Subjects: Amino acids; Atoms; Bioinformatics; Biological system modeling; Biomedical imaging; Biomedical NLP; Large language models; Molecular biophysics; Molecules; Multimodal Large Language Models; Natural languages; Proteins; Training
• ISSN: 2168-2194; 2168-2208
• DOI: 10.1109/JBHI.2024.3505955

Recent advances in large language models (LLMs) like ChatGPT have shed light on the development of knowledgeable and versatile AI research assistants in various scientific domains. However, they fall short in biomedical applications due to a lack of proprietary biomedical knowledge and deficiencies in handling biological sequences for molecules and proteins. To address these issues, we present BioMedGPT, a multimodal large language model for assisting biomedical research. We first incorporate domain expertise into LLMs by incremental pre-training on large-scale biomedical literature. Then, we harmonize 2D molecular graphs, protein sequences, and natural language within a unified, parameter-efficient fusion architecture by fine-tuning on multimodal question-answering datasets. Through comprehensive experiments, we show that BioMedGPT performs on par with human experts in comprehending biomedical documents and answering research questions. It also exhibits promising capability in analyzing intricate functions and properties of novel molecules and proteins, surpassing state-of-the-art LLMs by 17.1% and 49.8% absolute gains respectively in ROUGE-L on molecule and protein question-answering. Our models, datasets, and codes are open-sourced at https://github.com/PharMolix/OpenBioMed .