Swin transformer-based GAN for multi-modal medical image translation

• Published in: Frontiers in oncology 2022-08, Vol.12, p.942511
• Main Authors: Yan, Shouang, Wang, Chengyan, Chen, Weibo, Lyu, Jun
• Format: Article
• Language: English
• Subjects: generative adversarial network; magnetic resonance imaging; medical image translation frontiers; multi-modal; Oncology; Swin Transformer
• ISSN: 2234-943X; 2234-943X
• DOI: 10.3389/fonc.2022.942511

Medical image-to-image translation is considered a new direction with many potential applications in the medical field. The medical image-to-image translation is dominated by two models, including supervised Pix2Pix and unsupervised cyclic-consistency generative adversarial network (GAN). However, existing methods still have two shortcomings: 1) the Pix2Pix requires paired and pixel-aligned images, which are difficult to acquire. Nevertheless, the optimum output of the cycle-consistency model may not be unique. 2) They are still deficient in capturing the global features and modeling long-distance interactions, which are critical for regions with complex anatomical structures. We propose a Swin Transformer-based GAN for Multi-Modal Medical Image Translation, named MMTrans. Specifically, MMTrans consists of a generator, a registration network, and a discriminator. The Swin Transformer-based generator enables to generate images with the same content as source modality images and similar style information of target modality images. The encoder part of the registration network, based on Swin Transformer, is utilized to predict deformable vector fields. The convolution-based discriminator determines whether the target modality images are similar to the generator or from the real images. Extensive experiments conducted using the public dataset and clinical datasets showed that our network outperformed other advanced medical image translation methods in both aligned and unpaired datasets and has great potential to be applied in clinical applications.