Multi-Modal Learning from Video, Eye Tracking, and Pupillometry for Operator Skill Characterization in Clinical Fetal Ultrasound

• Main Authors: Sharma, Harshita, Drukker, Lior, Papageorghiou, Aris T., Noble, J. Alison
• Format: Conference Proceeding
• Language: English
• Subjects: Biological system modeling; Brain modeling; convolutional neural networks; eye tracking; Heart; Multi-modal learning; Network architecture; Performance evaluation; pupillometry; Tensors; Ultrasonic imaging; ultrasound
• ISSN: 1945-7928; 1945-8452
• DOI: 10.1109/ISBI48211.2021.9433863

This paper presents a novel multi-modal learning approach for automated skill characterization of obstetric ultrasound operators using heterogeneous spatio-temporal sensory cues, namely, scan video, eye-tracking data, and pupillometric data, acquired in the clinical environment. We address pertinent challenges such as combining heterogeneous, small-scale and variable-length sequential datasets, to learn deep convolutional neural networks in real-world scenarios. We propose spatial encoding for multi-modal analysis using sonography standard plane images, spatial gaze maps, gaze trajectory images, and pupillary response images. We present and compare five multi-modal learning network architectures using late, intermediate, hybrid, and tensor fusion. We build models for the Heart and the Brain scanning tasks, and performance evaluation suggests that multi-modal learning networks outperform uni-modal networks, with the best-performing model achieving accuracies of 82.4% (Brain task) and 76.4% (Heart task) for the operator skill classification problem.