Using large language models for safety-related table summarization in clinical study reports

• Published in: JAMIA open 2024-07, Vol.7 (2), p.ooae043
• Main Authors: Landman, Rogier, Healey, Sean P, Loprinzo, Vittorio, Kochendoerfer, Ulrike, Winnier, Angela Russell, Henstock, Peter V, Lin, Wenyi, Chen, Aqiu, Rajendran, Arthi, Penshanwar, Sushant, Khan, Sheraz, Madhavan, Subha
• Format: Article
• Language: English
• Subjects: Case Report; Clinical trials; Computational linguistics; Language processing; Natural language interfaces; Pharmaceutical industry
• ISSN: 2574-2531; 2574-2531
• DOI: 10.1093/jamiaopen/ooae043

Objectives The generation of structured documents for clinical trials is a promising application of large language models (LLMs). We share opportunities, insights, and challenges from a competitive challenge that used LLMs for automating clinical trial documentation. Materials and Methods As part of a challenge initiated by Pfizer (organizer), several teams (participant) created a pilot for generating summaries of safety tables for clinical study reports (CSRs). Our evaluation framework used automated metrics and expert reviews to assess the quality of AI-generated documents. Results The comparative analysis revealed differences in performance across solutions, particularly in factual accuracy and lean writing. Most participants employed prompt engineering with generative pre-trained transformer (GPT) models. Discussion We discuss areas for improvement, including better ingestion of tables, addition of context and fine-tuning. Conclusion The challenge results demonstrate the potential of LLMs in automating table summarization in CSRs while also revealing the importance of human involvement and continued research to optimize this technology. Lay Summary Large language models (LLMs) have shown promise in automating the creation of structured documents for clinical trials. In a recent competition organized by Pfizer, teams developed a pilot program to generate summaries of safety tables for clinical study reports (CSRs) using LLMs. They evaluated the quality of the AI-generated documents using automated metrics and expert reviews. The analysis revealed differences in performance among the solutions, particularly in terms of factual accuracy and concise writing. Most participants used a model called generative pre-trained transformer (GPT) with prompt engineering. Areas for improvement were identified, such as better handling of tables, adding context, and fine-tuning the models. The challenge results demonstrated the potential of LLMs in automating the summarization of tables in CSRs. However, the study also emphasized the importance of human involvement and ongoing research to optimize this technology. Creating CSRs is a time-consuming process, and one of the challenges is extracting relevant information from tables. It is important to include additional information and consider connections across different tables. Continued research will be done to address these issues.