A multi-arm robot system for efficient apple harvesting: Perception, task plan and control

• Published in: Computers and electronics in agriculture 2023-08, Vol.211, p.107979, Article 107979
• Main Authors: Li, Tao, Xie, Feng, Zhao, Zhuoqun, Zhao, Hui, Guo, Xin, Feng, Qingchun
• Format: Article
• Language: English
• Subjects: agriculture; algorithms; apples; computer software; computer vision; Depth camera; electronics; farm labor; fruits; Harvesting robot; industry; Multiple robotic arms; Object detection; orchards; Robotics; robots; Task planning; vision
• ISSN: 0168-1699; 1872-7107
• DOI: 10.1016/j.compag.2023.107979

Robot harvesting has emerged as an urgent need for apple industry due to a sharp decline in agricultural labor. In the field of harvesting robots, the use of multiple robotic arms to improve operational efficiency and promote industrial applications has gained significant attention. Despite the significant progress made in multi-arm harvesting robots in recent years, their widespread application in orchard production is hindered by insufficient efficiency of operation and the accuracy of fruit positioning. This paper focuses on the precise perception and multi-arm collaborative control issues of harvesting robots, and proposes a multi-arm apple harvesting robot system. Firstly, the paper introduces the hardware and software integration method and kinematic configuration of the robot, and presents its workspace division and asynchronous sequential operation mode. Secondly, the paper proposes a stereo vision fruit recognition and localization algorithm based on multi-task deep learning to enhance the accuracy of apple fruit positioning and a method of combining multiple perspectives to acquire a global fruit map is introduced. Finally, the paper presents a multi-arm task planning method based on the Markov game framework to optimize the target harvesting order of each arm and improve the collaboration efficiency. The effectiveness of the robot and its perception and control methods are verified through multiple field experiments in orchards. The field trials showed that the proposed vision system reduces the median locating error of the robot system by up to 44.43%; the proposed task planning algorithm can reduce the average cycle time by 33.3% compared to the heuristic-based algorithm, and time taken for optimizing task planning ranged from 1.14 s to 1.21 s; and the robot’s harvest success rate varied from 71.28% to 80.45%, and the average cycle time ranged from 5.8 s to 6.7 s. •A visual recognition and localization algorithm by multiple stereo cameras.•A point-cloud-processing pipeline by a frustum-based method.•A task planning scheme based on multi-agent reinforcement learning framework.•A highly integrated four-arm apple harvesting robotic system.