Generating multi-pathological and multi-modal images and labels for brain MRI

• Published in: Medical image analysis 2024-10, Vol.97, p.103278, Article 103278
• Main Authors: Fernandez, Virginia, Pinaya, Walter Hugo Lopez, Borges, Pedro, Graham, Mark S., Tudosiu, Petru-Daniel, Vercauteren, Tom, Cardoso, M. Jorge
• Format: Article
• Language: English
• Subjects: Algorithms; Brain - diagnostic imaging; Brain MRI; Deep Learning; Generative modelling; Humans; Image Interpretation, Computer-Assisted - methods; Image Processing, Computer-Assisted - methods; Imaging, Three-Dimensional - methods; Magnetic Resonance Imaging - methods; Medical imaging segmentation; Multimodal Imaging - methods
• ISSN: 1361-8415; 1361-8423; 1361-8423
• DOI: 10.1016/j.media.2024.103278

The last few years have seen a boom in using generative models to augment real datasets, as synthetic data can effectively model real data distributions and provide privacy-preserving, shareable datasets that can be used to train deep learning models. However, most of these methods are 2D and provide synthetic datasets that come, at most, with categorical annotations. The generation of paired images and segmentation samples that can be used in downstream, supervised segmentation tasks remains fairly uncharted territory. This work proposes a two-stage generative model capable of producing 2D and 3D semantic label maps and corresponding multi-modal images. We use a latent diffusion model for label synthesis and a VAE-GAN for semantic image synthesis. Synthetic datasets provided by this model are shown to work in a wide variety of segmentation tasks, supporting small, real datasets or fully replacing them while maintaining good performance. We also demonstrate its ability to improve downstream performance on out-of-distribution data. •2D and 3D paired labels and multi-modal images are synthesised end-to-end.•Conditioning enables synthesis of lesions and extrapolation to unseen combinations.•Synthetic datasets can be used to supplement small real datasets for segmentation.•Synthetic datasets improve generalisability to other domains and unseen lesions.•Privacy analysis shows no memorisation of training datasets.