Segmentation and motion parameter estimation for robotic Medjoul-date thinning

• Published in: Precision agriculture 2022-04, Vol.23 (2), p.514-537
• Main Authors: Shoshan, Tal, Bechar, Avital, Cohen, Yuval, Sadowsky, Avraham, Berman, Sigal
• Format: Article
• Language: English
• Subjects: Agriculture; Artificial neural networks; Atmospheric Sciences; Automation; Biomedical and Life Sciences; Chemistry and Earth Sciences; Computer Science; Errors; Fruits; Image processing; Image quality; Image segmentation; Life Sciences; Motion planning; Neural networks; Orientation; Parameter estimation; Parameter identification; Physics; Remote Sensing/Photogrammetry; Soil Science & Conservation; Statistics for Engineering; Thinning
• ISSN: 1385-2256; 1573-1618
• DOI: 10.1007/s11119-021-09847-2

Laborious fruit thinning is required for attaining high-quality Medjoul dates. Thinning automation can significantly reduce labor and improve efficiency. An image processing apparatus developed for robotic Medjoul thinning is presented. Instance segmentation based on Mask R-CNN was applied to identify the fruit bunch components: spikelets and rachis. Motion planning parameters were extracted using the derived masks: rachis center point (RCP), rachis orientation angle, and spikelets remaining length. RCP and rachis orientation angle were computed geometrically, spikelets remaining length was estimated with a convolutional neural network (CNN) and a deep neural network (DNN). Instance segmentation results were accurate, especially for spikelets, for low intersection over union (IoU) (0.3 IoU, fruit determined for thinning identification, spikelets: 98%, rachises: 73%). However, only 66% of the rachises were correctly matched to spikelets. The segmentation of all spikelets and rachises in the images was of medium quality for low IoU (0.3 IoU, F1, spikelets: 0.67, rachis: 0.77), where both precision and recall dropped for higher IoUs. RCP and rachis orientation angle were accurately estimated (0.3 IoU, error, RCP: 2.2 cm, rachis orientation angle: 5.0°). Spikelets remaining length estimation using CNN resulted in better performance than DNN (0.3 IoU, error, CNN: 19.7%, DNN: 24.6%). Spikelets segmentation results are suitable for thinning automation. However, rachis segmentation and matching the rachis and spikelets may still require human intervention during run-time. RCP and rachis orientation angle estimation errors are acceptable, while spikelets remaining length estimation errors are acceptable only for preliminary motion planning and mandate additional tuning during motion execution.