JOINEDTrans: Prior guided multi-task transformer for joint optic disc/cup segmentation and fovea detection

• Published in: Computers in biology and medicine 2024-07, Vol.177, p.108613-108613, Article 108613
• Main Authors: He, Huaqing, Qiu, Jiaming, Lin, Li, Cai, Zhiyuan, Cheng, Pujin, Tang, Xiaoying
• Format: Article
• Language: English
• Subjects: Biology; Biomarkers; Coarse-to-fine; Computer engineering; Computers; Deep learning; Electrical engineering; Eye diseases; Fovea; Fovea detection; Glaucoma; Image processing; Image quality; Image segmentation; Lesions; Localization; Multi-task learning; Neural networks; Optic disc and cup segmentation; Prior; Retina; Science; Transformers; Vessels
• ISSN: 0010-4825; 1879-0534
• DOI: 10.1016/j.compbiomed.2024.108613

Deep learning-based image segmentation and detection models have largely improved the efficiency of analyzing retinal landmarks such as optic disc (OD), optic cup (OC), and fovea. However, factors including ophthalmic disease-related lesions and low image quality issues may severely complicate automatic OD/OC segmentation and fovea detection. Most existing works treat the identification of each landmark as a single task, and take into account no prior information. To address these issues, we propose a prior guided multi-task transformer framework for joint OD/OC segmentation and fovea detection, named JOINEDTrans. JOINEDTrans effectively combines various spatial features of the fundus images, relieving the structural distortions induced by lesions and other imaging issues. It contains a segmentation branch and a detection branch. To be noted, we employ an encoder with prior-learning in a vessel segmentation task to effectively exploit the positional relationship among vessel, OD/OC, and fovea, successfully incorporating spatial prior into the proposed JOINEDTrans framework. There are a coarse stage and a fine stage in JOINEDTrans. In the coarse stage, OD/OC coarse segmentation and fovea heatmap localization are obtained through a joint segmentation and detection module. In the fine stage, we crop regions of interest for subsequent refinement and use predictions obtained in the coarse stage to provide additional information for better performance and faster convergence. Experimental results demonstrate that JOINEDTrans outperforms existing state-of-the-art methods on the publicly available GAMMA, REFUGE, and PALM fundus image datasets. We make our code available at https://github.com/HuaqingHe/JOINEDTrans. •Vessels can guide optic cup/disc segmentation, macula localization.•Simultaneously performing both tasks enhances robustness.•Vessel pretraining encoder proving superior to a normal retinal encoder.•Star-shape loss expedites convergence, correcting erroneous segmentation results.