Visual recognition of cherry tomatoes in plant factory based on improved deep instance segmentation

• Published in: Computers and electronics in agriculture 2022-06, Vol.197, p.106991, Article 106991
• Main Authors: Xu, Penghui, Fang, Nan, Liu, Na, Lin, Fengshan, Yang, Shuqin, Ning, Jifeng
• Format: Article
• Language: English
• Subjects: Automatic picking; Cherry tomatoes; Data integration; Deep instance segmentation; Factories; Fruits; Image segmentation; Industrial plants; Instance segmentation; Plant factory; Production costs; Recognition; Stems; Tomatoes; Visual recognition
• ISSN: 0168-1699; 1872-7107
• DOI: 10.1016/j.compag.2022.106991

•The integral constraint between fruit and stem eliminates the interference of branch.•Multi-classification prediction subnetwork with the balanced loss accurately detects fruit and stem.•Adaptive feature pooling fuses the bottom features and enhances the segmentation accuracy of stem.•This work provides a path for accurately locating cherry tomatoes in plant factory. Accurate recognition of cherry tomatoes is a key issue for the automatic picking system in plant factories, which helps to improve picking efficiency and reduce production costs. By using the depth information and considering the prior adjacent constraint between the fruit and the stem of cherry tomatoes, this paper proposes an improved Mask R-CNN for visual recognition of cherry tomatoes. Firstly, the input layer of the network is modified to achieve dual-mode data fusion of RGB and depth images. Secondly, by constructing the corresponding region generation network to indicate the integral constraint between the fruit and the stem, false recognition of branches is reduced. Thirdly, a multi-class prediction subnetwork is used to decouple the pixel-level category predictions of fruit and stem. Meanwhile, multi-task loss balance and adaptive feature pooling are adopted to overcome the limitation caused by the size difference between fruit and stem. The experimental results show that the improved Mask R-CNN achieved an accuracy of 93.76% for fruit recognition, which is 11.53% and 15.5% higher than that of the standard Mask R-CNN and YOLACT, and it achieves an accuracy of 89.34% for stem recognition, which is 13.91% and 19.7% higher than that of the standard Mask R-CNN and YOLACT, respectively. Besides, the recall rate of the proposed method for stem recognition is 94.47%, which is 11.53% and 8.3% higher than that of YOLACT and Mask R-CNN, respectively. In addition, the proposed method takes only 0.04 s to process a single image, providing an efficient approach for automatically picking cherry tomatoes in plant factories.