Plant growth information measurement based on object detection and image fusion using a smart farm robot

• Published in: Computers and electronics in agriculture 2023-04, Vol.207, p.107703, Article 107703
• Main Authors: Cho, Songhee, Kim, Taehyeong, Jung, Dae-Hyun, Park, Soo Hyun, Na, Yunseong, Ihn, Yong Seok, Kim, KangGeon
• Format: Article
• Language: English
• Subjects: agriculture; Data augmentation; Deep learning in agriculture; electronics; farms; fruits; humans; Image fusion; plant growth; Plant growth information measurement; precision agriculture; Smart farm robot; uncertainty; vegetables
• ISSN: 0168-1699; 1872-7107
• DOI: 10.1016/j.compag.2023.107703

•A smart farm robot accurately performs repeatable tasks in real time.•Automatically measure plant growth information with a deep learning-based approach.•Data augmentation based on fused RGB-D images contributes to precise measurement.•Solve problems in real-world high-plant-density environments. Traditionally, vegetable and fruit production has relied on empirical and ambiguous decisions made by human farmers. To overcome this uncertainty in agriculture, smart farm robots have been widely studied in recent years. However, measuring growth information with robots remains a challenge because of the similarity in the appearance of the target plant and those around it. In this study, we propose a smart farm robot that accurately measures the growth information of a target plant based on object detection, image fusion, and data augmentation with fused images. The proposed smart farm robot uses an end-to-end real-time deep learning-based object detector that shows state-of-the-art performances. To distinguish the target plant from other plants with a higher accuracy and improved robustness than those of existing methods, we exploited image fusion using both RGB and depth images. In particular, the data augmentation, based on the fused RGB, and depth information, contributes to the precise measurement of growth information from smart farms, regardless of the high density of vegetables and fruits in these farms. We propose and evaluate a real-time measurement system to obtain precise target-plant growth information in precision agriculture. The code and models are publicly available on Github: https://github.com/kistvision/Plant_growth_measurement.