Reducing annotating load: Active learning with synthetic images in surgical instrument segmentation

• Published in: Medical image analysis 2024-10, Vol.97, p.103246, Article 103246
• Main Authors: Peng, Haonan, Lin, Shan, King, Daniel, Su, Yun-Hsuan, Abuzeid, Waleed M., Bly, Randall A., Moe, Kris S., Hannaford, Blake
• Format: Article
• Language: English
• Subjects: Active deep learning; Deep Learning; Endoscopy; Humans; Image Processing, Computer-Assisted - methods; Medical image synthesis; Minimally Invasive Surgical Procedures; Neural Networks, Computer; Robot instrument segmentation; Surgical Instruments
• ISSN: 1361-8415; 1361-8423; 1361-8423
• DOI: 10.1016/j.media.2024.103246

Accurate instrument segmentation in the endoscopic vision of minimally invasive surgery is challenging due to complex instruments and environments. Deep learning techniques have shown competitive performance in recent years. However, deep learning usually requires a large amount of labeled data to achieve accurate prediction, which poses a significant workload. To alleviate this workload, we propose an active learning-based framework to generate synthetic images for efficient neural network training. In each active learning iteration, a small number of informative unlabeled images are first queried by active learning and manually labeled. Next, synthetic images are generated based on these selected images. The instruments and backgrounds are cropped out and randomly combined with blending and fusion near the boundary. The proposed method leverages the advantage of both active learning and synthetic images. The effectiveness of the proposed method is validated on two sinus surgery datasets and one intraabdominal surgery dataset. The results indicate a considerable performance improvement, especially when the size of the annotated dataset is small. All the code is open-sourced at: https://github.com/HaonanPeng/active_syn_generator. [Display omitted] •Actively generated synthetic images reduce the annotating workload.•Image syntheses make better utilization of active learning selected images.•Neural networks trained by synthetic images have competitive performance.•Multi-blending method further improves segmentation near tissue–instrument boundary.•External backgrounds are efficient when few labeled images are used.