Clinical Prompt Learning With Frozen Language Models

• Published in: IEEE transaction on neural networks and learning systems 2024-11, Vol.35 (11), p.16453-16463
• Main Authors: Taylor, Niall, Zhang, Yi, Joyce, Dan W., Gao, Ziming, Kormilitzin, Andrey, Nevado-Holgado, Alejo
• Format: Article
• Language: English
• Subjects: Adaptation models; Algorithms; Bit error rate; Clinical decision support; Computer architecture; few-shot learning; Humans; Machine Learning; Natural Language Processing; Neural Networks, Computer; pretrained language models (PLMs); prompt learning; Task analysis; Training; Transformers; Tuning
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2023.3294633

When the first transformer-based language models were published in the late 2010s, pretraining with general text and then fine-tuning the model on a task-specific dataset often achieved the state-of-the-art performance. However, more recent work suggests that for some tasks, directly prompting the pretrained model matches or surpasses fine-tuning in performance with few or no model parameter updates required. The use of prompts with language models for natural language processing (NLP) tasks is known as prompt learning. We investigated the viability of prompt learning on clinically meaningful decision tasks and directly compared this with more traditional fine-tuning methods. Results show that prompt learning methods were able to match or surpass the performance of traditional fine-tuning with up to 1000 times fewer trainable parameters, less training time, less training data, and lower computation resource requirements. We argue that these characteristics make prompt learning a very desirable alternative to traditional fine-tuning for clinical tasks, where the computational resources of public health providers are limited, and where data can often not be made available or not be used for fine-tuning due to patient privacy concerns. The complementary code to reproduce the experiments presented in this work can be found at https://github.com/NtaylorOX/Public_Clinical_Prompt .