Depth over RGB: automatic evaluation of open surgery skills using depth camera

• Published in: International journal for computer assisted radiology and surgery 2024, Vol.19 (7), p.1349-1357
• Main Authors: Zuckerman, Ido, Werner, Nicole, Kouchly, Jonathan, Huston, Emma, DiMarco, Shannon, DiMusto, Paul, Laufer, Shlomi
• Format: Article
• Language: English
• Subjects: Annotations; Cameras; Clinical Competence; Computer Imaging; Computer Science; Data compression; Evaluation; Hand tools; Health Informatics; Humans; Imaging; Imaging, Three-Dimensional - methods; Medicine; Medicine & Public Health; Object recognition; Original; Original Article; Pattern Recognition and Graphics; Privacy; Radiology; Segmentation; Simulators; Skills; Surgeons; Surgery; Suture Techniques - education; Suture Techniques - instrumentation; Sutures; Vision
• ISSN: 1861-6429; 1861-6410; 1861-6429
• DOI: 10.1007/s11548-024-03158-3

Purpose In this paper, we present a novel approach to the automatic evaluation of open surgery skills using depth cameras. This work is intended to show that depth cameras achieve similar results to RGB cameras, which is the common method in the automatic evaluation of open surgery skills. Moreover, depth cameras offer advantages such as robustness to lighting variations, camera positioning, simplified data compression, and enhanced privacy, making them a promising alternative to RGB cameras. Methods Experts and novice surgeons completed two simulators of open suturing. We focused on hand and tool detection and action segmentation in suturing procedures. YOLOv8 was used for tool detection in RGB and depth videos. Furthermore, UVAST and MSTCN++ were used for action segmentation. Our study includes the collection and annotation of a dataset recorded with Azure Kinect. Results We demonstrated that using depth cameras in object detection and action segmentation achieves comparable results to RGB cameras. Furthermore, we analyzed 3D hand path length, revealing significant differences between experts and novice surgeons, emphasizing the potential of depth cameras in capturing surgical skills. We also investigated the influence of camera angles on measurement accuracy, highlighting the advantages of 3D cameras in providing a more accurate representation of hand movements. Conclusion Our research contributes to advancing the field of surgical skill assessment by leveraging depth cameras for more reliable and privacy evaluations. The findings suggest that depth cameras can be valuable in assessing surgical skills and provide a foundation for future research in this area.