Transformers in medical image analysis

• Published in: Intelligent medicine 2023-02, Vol.3 (1), p.59-78
• Main Authors: He, Kelei, Gan, Chen, Li, Zhuoyuan, Rekik, Islem, Yin, Zihao, Ji, Wen, Gao, Yang, Wang, Qian, Zhang, Junfeng, Shen, Dinggang
• Format: Article
• Language: English
• Subjects: Deep learning; Diagnosis; Image synthesis; Medical image analysis; Multi-modal learning; Multi-task learning; Registration; Segmentation; Transformer; Weakly-supervised learning
• ISSN: 2667-1026; 2667-1026
• DOI: 10.1016/j.imed.2022.07.002

Transformers have dominated the field of natural language processing and have recently made an impact in the area of computer vision. In the field of medical image analysis, transformers have also been successfully used in to full-stack clinical applications, including image synthesis/reconstruction, registration, segmentation, detection, and diagnosis. This paper aimed to promote awareness of the applications of transformers in medical image analysis. Specifically, we first provided an overview of the core concepts of the attention mechanism built into transformers and other basic components. Second, we reviewed various transformer architectures tailored for medical image applications and discuss their limitations. Within this review, we investigated key challenges including the use of transformers in different learning paradigms, improving model efficiency, and coupling with other techniques. We hope this review would provide a comprehensive picture of transformers to readers with an interest in medical image analysis.