A real-time branch detection and reconstruction mechanism for harvesting robot via convolutional neural network and image segmentation

• Published in: Computers and electronics in agriculture 2022-01, Vol.192, p.106609, Article 106609
• Main Authors: Wan, Hao, Fan, Zeming, Yu, Xiaojun, Kang, Meilin, Wang, Pengbo, Zeng, Xilei
• Format: Article
• Language: English
• Subjects: Artificial neural networks; Branches; CNN; Detection; Frames per second; Harvesting; Image processing; Image segmentation; Neural networks; Polynomials; Real time; Recall; Reconstruction; Robots
• ISSN: 0168-1699; 1872-7107
• DOI: 10.1016/j.compag.2021.106609

•Proposed a real-time detection and reconstruction scheme for obscured branches.•The method was applied to harvesting robots for fruit picking.•Combined image processing and CNNs for improving branch localization accuracy.•Branch reconstruction constraints are proposed according to the growth state.•The branch reconstruction speed reaches up to 22.7 FPS. To alleviate the burden of fruit harvesting imposed by rising costs and decreasing labor supply, intelligent robots are highly desired in modern farms. A major problem, however, is how to detect and locate the tree branches for the robots to plan their arm movements during harvesting process. This study addresses the obscured branch detection and reconstruction problem, and proposes a real-time branch detection and reconstruction (RBDR) mechanism using convolutional neural networks (CNNs) and image processing techniques. Firstly, we build a Branch-CNN framework for detecting the bare branches and complete their rough localization, and then, realize the background segmentation in HSV space to obtain the precise branch regions. Finally, with the distance and angle constraints considered, a polynomial fit is conducted onto the precise boxes of the same branch to fill in the obscured areas. The proposed RBDR mechanism is applied onto a harvesting robot platform, and experiments with both lab simulated orchard environment and real pomegranate tree environment are conducted to verify its feasibility. Results show that under the simulation environment, at an Intersection over Union (IOU) threshold of 0.5, Branch-CNN achieves the best overall performance, with the average detection precision, recall rate, and F1-Score being 90.98%, 92%, and 91%, respectively, and the average reconstruction accuracy of RBDR is 88.76%. Under the real pomegranate tree environment, Branch-CNN achieves 90.7% detection precision, 89% recall rate, and 90% F1-Score, respectively. The overall reconstruction speed of RBDR is 22.7 frames per second (FPS) on image with a resolution of 960*720. Such results fully demonstrate the rationality and effectiveness of RBDR.