Phenotyping of individual apple tree in modern orchard with novel smartphone-based heterogeneous binocular vision and YOLOv5s

• Published in: Computers and electronics in agriculture 2023-06, Vol.209, p.107814, Article 107814
• Main Authors: Zhao, Guanao, Yang, Ruizhe, Jing, Xudong, Zhang, Haosen, Wu, Zhenchao, Sun, Xiaoming, Jiang, Hanhui, Li, Rui, Wei, Xiaofeng, Fountas, Spyros, Zhang, Huijun, Fu, Longsheng
• Format: Article
• Language: English
• Subjects: Android smartphone; Deep learning; Modern apple orchard; Phenotyping; Smartphone-based heterogeneous binocular vision
• ISSN: 0168-1699; 1872-7107
• DOI: 10.1016/j.compag.2023.107814

•The feasibility of phenotype measurement with only a smartphone is proved.•The use of multi-camera of smartphone to obtain depth has been confirmed.•Virtual focal solved inconsistency of heterogeneous binocular images in some sorts.•Pinhole camera model provided physical sizes or phenotypes of apple tree.•The phenotyping performance of smartphone is similar to existing researches. Phenotyping plays a significant role in the breeding of apple tree. However, existing researches mainly relied on instruments, such as LiDAR, RGB-D camera or UAV (unmanned aerial vehicle) embedded with depth sensor, etc., which requires additional costs for users and also inconvenient. Therefore, a novel method of smartphone-based heterogeneous binocular vision was developed to fulfill low-cost automated phenotyping for apple tree. In this study, a pair of cameras on multi-camera smartphone was selected to obtain heterogeneous binocular camera. After that, a so-called virtual focal method was developed to generate standard binocular images from heterogeneous binocular images of smartphone. A well-known YOLOv5s object detection model was trained on a four-class dataset to detect fruits, grafts, trunks and whole trees. Then, the model was simplified to fit the deployment on smartphone. Finally, five phenotypes (trunk diameter, ground diameter, tree height, fruit vertical diameter, and fruit horizontal diameter) of individual apple tree were obtained by pinhole camera model and standard binocular vision. After evaluation of phenotyping manually and by smartphone, our method shows MAPE (mean average percentage error) ranging from 6.00 % to 13.73 % for the five phenotypes. Compared with the existing studies, our method has reached a close or even better phenotyping accuracy with only a smartphone. As more and more smartphones have multi-camera, our method is probably the lowest cost phenotyping method for most of the potential users. Results indicated that the approach could be utilized to phenotyping of apple tree.