Uncertainty-aware report generation for chest X-rays by variational topic inference

• Published in: Medical image analysis 2022-11, Vol.82, p.102603-102603, Article 102603
• Main Authors: Najdenkoska, Ivona, Zhen, Xiantong, Worring, Marcel, Shao, Ling
• Format: Article
• Language: English
• Subjects: Chest X-ray; Latent variable model; Radiology report; Variational inference
• ISSN: 1361-8415; 1361-8423
• DOI: 10.1016/j.media.2022.102603

Automating report generation for medical imaging promises to minimize labor and aid diagnosis in clinical practice. Deep learning algorithms have recently been shown to be capable of captioning natural photos. However, doing a similar thing for medical data, is difficult due to the variety in reports written by different radiologists with fluctuating levels of knowledge and experience. Current methods for automatic report generation tend to merely copy one of the training samples in the created report. To tackle this issue, we propose variational topic inference, a probabilistic approach for automatic chest X-ray report generation. Specifically, we introduce a probabilistic latent variable model where a latent variable defines a single topic. The topics are inferred in a conditional variational inference framework by aligning vision and language modalities in a latent space, with each topic governing the generation of one sentence in the report. We further adopt a visual attention module that enables the model to attend to different locations in the image while generating the descriptions. We conduct extensive experiments on two benchmarks, namely Indiana U. Chest X-rays and MIMIC-CXR. The results demonstrate that our proposed variational topic inference method can generate reports with novel sentence structure, rather than mere copies of reports used in training, while still achieving comparable performance to state-of-the-art methods in terms of standard language generation criteria. •Definition of the sentence generator net as a Transformer decoder and a fully Transformer-based definition of the VTI model.•A thorough explanation of the base VTI method, with additional derivation steps and algorithms.•Redesigned figures of the architecture for a better illustration of the building blocks of the model.•Additional ablation study and experiments for the new formulation of the model.•Discussion about the probabilistic formulation of the task and trade-off between different decoders for generating medical text.