A Wearable Augmented Reality Navigation System for Surgical Telementoring Based on Microsoft HoloLens

This paper reports a new type of augmented reality (AR) system that integrates a Microsoft HoloLens device with a three-dimensional (3D) point tracking module for medical training and telementored surgery. In this system, a stereo camera is used to track the 3D position of a scalpel and transfer its...

Full description

Saved in:
Bibliographic Details
Published in:Annals of biomedical engineering 2021, Vol.49 (1), p.287-298
Main Authors: Liu, Peng, Li, Chenmeng, Xiao, Changlin, Zhang, Zeshu, Ma, Junqi, Gao, Jian, Shao, Pengfei, Valerio, Ian, Pawlik, Timothy M., Ding, Chengbiao, Yilmaz, Alper, Xu, Ronald
Format: Article
Language:English
Subjects:
Annotations
Augmented reality
Biochemistry
Biological and Medical Physics
Biomedical and Life Sciences
Biomedical Engineering and Bioengineering
Biomedicine
Biophysics
Classical Mechanics
Navigation systems
Original Article
Skin grafts
Surgery
Training
Virtual reality
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper reports a new type of augmented reality (AR) system that integrates a Microsoft HoloLens device with a three-dimensional (3D) point tracking module for medical training and telementored surgery. In this system, a stereo camera is used to track the 3D position of a scalpel and transfer its coordinates wirelessly to a HoloLens device. In the scenario of surgical training, a virtual surgical scene with pre-recorded surgical annotations is superimposed with the actual surgical scene so that the surgical trainee is able to operate following virtual instructions. In the scenario of telementored surgery, the virtual surgical scene is co-registered with the actual surgical scene so that the virtual scalpel remotely mentored by an experienced surgeon provides the AR guidance for the inexperienced on-site operator. The performance characteristics of the proposed AR telementoring system are verified by benchtop experiments. The clinical applicability of the proposed system in telementored skin grafting surgery and fasciotomy is validated in a New Zealand rabbit model. Our benchtop and in vivo experiments demonstrate the potential to improve surgical performance and reduce healthcare disparities in remote areas with limited resources.
ISSN:0090-6964
1573-9686
DOI:10.1007/s10439-020-02538-5