Loading…
Visual servo control of endoscope-holding robot based on multi-objective optimization: System modeling and instrument tracking
In routine laparoscopic surgeries, the fatigue operation and improper chief-assistant coordination of the assistant surgeon both reduce the stability of the endoscopic field of view (EFOV) and the surgical efficiency. In contrast, the assistant endoscope-holding robot (AEHR) can stably hold the endo...
Saved in:
Published in: | Measurement : journal of the International Measurement Confederation 2023-04, Vol.211, p.112658, Article 112658 |
---|---|
Main Authors: | , , , , |
Format: | Article |
Language: | English |
Subjects: | |
Citations: | Items that this one cites Items that cite this one |
Online Access: | Get full text |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Summary: | In routine laparoscopic surgeries, the fatigue operation and improper chief-assistant coordination of the assistant surgeon both reduce the stability of the endoscopic field of view (EFOV) and the surgical efficiency. In contrast, the assistant endoscope-holding robot (AEHR) can stably hold the endoscope and adjust the pose of endoscopic image sensor. However, the traditional method simplifies the tracking of surgical instrument tips (SITs) to the tracking of feature points, and lacks the depth optimization of monocular endoscopic image senor. Based on them, this paper proposes a surgical instrument tracking control method based on visual tracking features (VTFs) and hand-eye coordination (HEC) with feedback of multiple sensors, which can control the depth and axial rotational degrees of freedom (DOFs) of endoscope while tracking SITs. Firstly, the motion mapping Jacobian matrix between the endoscopic image sensor and the endoscope-holding manipulator (EHM) end-effector is constructed with joint encoder feedbacks, and the VTFs of endoscopic camera feedback are extracted based on the deep segmentation model of SITs. Secondly, the position adjustment model based on VTFs and the attitude adjustment model based on HEC are established. Thirdly, a complete set of visual servo control (VSC) software and hardware system for AEHRs is designed. Finally, the correctness, effectiveness and operability of the proposed method are comprehensively verified through the simulation and experiment of surgical instrument tracking. The visual tracking distance and radius error of endoscopic image sensor frames are reduced by no less than 34.6% and 42.59% under different experimental backgrounds, respectively.
[Display omitted]
•An endoscopic pose control method based on VTFs and HEC is established.•An endoscopic position adjustment model is derived.•A VSC experimental platform and simulation system for AEHR is reconstructed.•An open-source semantic segmentation data set for surgical instrument tips is provided.•The system modeling and instrument tracking performance are verified. |
---|---|
ISSN: | 0263-2241 1873-412X |
DOI: | 10.1016/j.measurement.2023.112658 |