Citrus pose estimation under complex orchard environment for robotic harvesting

• Published in: European journal of agronomy 2025-01, Vol.162, p.127418, Article 127418
• Main Authors: Zhang, Guanming, Li, Li, Zhang, Yunfeng, Liang, Jiyuan, Chun, Changpin
• Format: Article
• Language: English
• Subjects: agronomy; algorithms; cameras; Citrus; data collection; Loss minimization; orchards; Pose estimation; RANSAC-LM; Robotic harvesting; robots; system optimization
• ISSN: 1161-0301
• DOI: 10.1016/j.eja.2024.127418

The growth poses of citrus on trees are diverse. To ensure minimal loss during citrus harvesting, accurately estimating the pose of citrus is particularly important. To solve this problem, this research developed a real-time citrus pose estimation system based on neural networks and point cloud processing algorithms. Specifically, this method uses neural networks to identify citrus. After constructing the citrus point cloud, it is input into the Random Sample Consensus with Levenberg-Marquardt (RANSAC-LM) point cloud processing algorithm to obtain the citrus coordinates. Combined with citrus growth information, the pose is output. By analyzing the distribution of citrus poses, citrus poses convenient for end- effector harvesting are defined. To enhance the camera's ability to obtain information about citrus, a camera observation model is constructed to dynamically adjust the camera position. Through experiments, the appropriate deep learning target detection framework YOLO V5 is selected for citrus object detection. The precision (P), recall rate (R), and mean average precision (mAP) are 92.3 %, 79.1 %, and 88.5 % respectively. This network can handle detection tasks in real orchard environments. The original Random Sample Consensus (RANSAC) is improved by using the Levenberg-Marquardt (LM) nonlinear optimization method. Experimental results show that RANSAC-LM reduces the citrus center coordinate precision error from (0.2, 0.2, 2.3) mm to (0.1, 0.2, 1.4) mm, reduces the accuracy Spherical Error Probable (SEP) from 2.77 to 1.61, and finally reduces the citrus pose error from 5.72° to 2.43°. The efficiency of the proposed citrus pose estimation algorithm is 0.24 s. Deployed on a citrus picking robot, it verifies the feasibility of the algorithm and provides a new solution for the pose estimation problem of citrus harvesting robots. [Display omitted] •A proposed method for estimating citrus pose from a single RGB-D image.•Propose observation models to improve information collection for citrus.•Proposed RANSAC-LM algorithm to increase the calculation accuracy.•The time for estimation of a citrus pose does not exceed 0.3 s.