X-ray Made Simple: Radiology Report Generation and Evaluation with Layman's Terms

• Published in: arXiv.org 2024-10
• Main Authors: Zhao, Kun, Xiao, Chenghao, Tang, Chen, Yang, Bohao, Ye, Kai, Noura Al Moubayed, Zhan, Liang, Lin, Chenghua
• Format: Article
• Language: English
• Subjects: Datasets; Learning; Radiology; Scaling laws; Semantics
• ISSN: 2331-8422

Radiology Report Generation (RRG) has achieved significant progress with the advancements of multimodal generative models. However, the evaluation in the domain suffers from a lack of fair and robust metrics. We reveal that, high performance on RRG with existing lexical-based metrics (e.g. BLEU) might be more of a mirage - a model can get a high BLEU only by learning the template of reports. This has become an urgent problem for RRG due to the highly patternized nature of these reports. In this work, we un-intuitively approach this problem by proposing the Layman's RRG framework, a layman's terms-based dataset, evaluation and training framework that systematically improves RRG with day-to-day language. We first contribute the translated Layman's terms dataset. Building upon the dataset, we then propose a semantics-based evaluation method, which is proved to mitigate the inflated numbers of BLEU and provides fairer evaluation. Last, we show that training on the layman's terms dataset encourages models to focus on the semantics of the reports, as opposed to overfitting to learning the report templates. We reveal a promising scaling law between the number of training examples and semantics gain provided by our dataset, compared to the inverse pattern brought by the original formats. Our code is available at \url{https://github.com/hegehongcha/LaymanRRG}.