A Dataset and Benchmarks for Segmentation and Recognition of Gestures in Robotic Surgery

• Published in: IEEE transactions on biomedical engineering 2017-09, Vol.64 (9), p.2025-2041
• Main Authors: Ahmidi, Narges, Tao, Lingling, Sefati, Shahin, Gao, Yixin, Lea, Colin, Haro, Benjamin Bejar, Zappella, Luca, Khudanpur, Sanjeev, Vidal, Rene, Hager, Gregory D.
• Format: Article
• Language: English
• Subjects: Activity recognition; Approximation; Automation; benchmark robotic dataset; Benchmark testing; Benchmarking - methods; Benchmarking - standards; Benchmarks; Classification; Clinical Competence - standards; Clinical Competence - statistics & numerical data; Conditional random fields; Data analysis; Data processing; Databases, Factual; Datasets; Evaluation; Gestures; Hidden Markov models; Humans; Imaging, Three-Dimensional - standards; Imaging, Three-Dimensional - statistics & numerical data; Kinematics; kinematics and video; Markov chains; Measurement; Medical personnel; Needles; Pattern Recognition, Automated - methods; Robotic surgery; Robotic Surgical Procedures - standards; Robotic Surgical Procedures - statistics & numerical data; Robots; Segmentation; State of the art; Surgery; surgical motion; United States
• ISSN: 0018-9294; 1558-2531
• DOI: 10.1109/TBME.2016.2647680

Objective: State-of-the-art techniques for surgical data analysis report promising results for automated skill assessment and action recognition. The contributions of many of these techniques, however, are limited to study-specific data and validation metrics, making assessment of progress across the field extremely challenging. Methods: In this paper, we address two major problems for surgical data analysis: First, lack of uniform-shared datasets and benchmarks, and second, lack of consistent validation processes. We address the former by presenting the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), a public dataset that we have created to support comparative research benchmarking. JIGSAWS contains synchronized video and kinematic data from multiple performances of robotic surgical tasks by operators of varying skill. We address the latter by presenting a well-documented evaluation methodology and reporting results for six techniques for automated segmentation and classification of time-series data on JIGSAWS. These techniques comprise four temporal approaches for joint segmentation and classification: hidden Markov model, sparse hidden Markov model (HMM), Markov semi-Markov conditional random field, and skip-chain conditional random field; and two feature-based ones that aim to classify fixed segments: bag of spatiotemporal features and linear dynamical systems. Results: Most methods recognize gesture activities with approximately 80% overall accuracy under both leave-one-super-trial-out and leave-one-user-out cross-validation settings. Conclusion: Current methods show promising results on this shared dataset, but room for significant progress remains, particularly for consistent prediction of gesture activities across different surgeons. Significance: The results reported in this paper provide the first systematic and uniform evaluation of surgical activity recognition techniques on the benchmark database.