Transfer and zero-shot learning for scalable weed detection and classification in UAV images

• Published in: Knowledge-based systems 2024-05, Vol.292, p.111586, Article 111586
• Main Authors: Belissent, Nicolas, Peña, José M., Mesías-Ruiz, Gustavo A., Shawe-Taylor, John, Pérez-Ortiz, María
• Format: Article
• Language: English
• Subjects: Computer vision; Machine learning; Precision agriculture; Residual networks; Site-specific weed management; Unmanned aerial vehicles
• ISSN: 0950-7051; 1872-7409
• DOI: 10.1016/j.knosys.2024.111586

In an effort to reduce pesticide use, agronomists and computer scientists have joined forces to develop site-specific weed detection and classification systems. These systems aim to recognize and locate weed species within a crop field, using precision equipment to apply required herbicides timely and only where needed, with the objective of reducing the sprayable surface required to eliminate the given weed and protect the crop, with both economic and environmental benefits. Yet, with climate change on the rise, common weeds are expected to undergo some changes to adapt to their environment, possibly with new or invasive weeds spreading to areas where they did not exist before. These changes (often morphological) as well as new invasions need to be taken into account by future classifiers and detection algorithms to ensure system robustness and adaptation to new habitats/climate dynamics. This paper proposes a set of experiments evaluating the use of transfer learning and zero-shot learning for weed classification using our novel TomatoWeeds dataset. Residual networks of variable depth, pretrained on the Imagenet and/or DeepWeeds datasets were evaluated. A ResNet50 pretrained on both datasets and fine-tuned on the TomatoWeeds dataset performed best, returning a holdout set accuracy of 77.8%, showing the advantageous use of transfer learning in this domain. Zero-shot learning, using both embeddings of images and morphological and habitat text-based descriptions, is implemented to test the ability of machine learning pipelines of recognizing unseen classes at test time (which may arise e.g. due to changing climate dynamics), a learning task in which the field (and our experiments) are still far from satisfactory results. Further research could benefit from larger weed-specific datasets for transfer learning as well as deeper network architectures to improve model performance. The projection-based ZSL could also benefit from larger datasets and new zero-shot learning architectures in hope that unseen classes are accurately projected. •Use of the TomatoWeed dataset; containing UAV images of tomato fields.•Three individual weed species are labeled using single pixel annotation.•Different transfer learning configuration were tested.•Resnet50 pretrained on Imagenet and DeepWeeds was deemed best performing.•Zero-shot learning approach built to detect unseen weed classes at testing.