Real-time evaluation of object detection models across open world scenarios

• Published in: Applied soft computing 2024-09, Vol.163, p.111921, Article 111921
• Main Authors: Goswami, Puneet, Aggarwal, Lakshita, Kumar, Arun, Kanwar, Rahul, Vasisht, Urvi
• Format: Article
• Language: English
• Subjects: Computer vision; DETR; Evaluation metrices; Faster R-CNN; Model comparison; Object detection; Resnet 101; Resnet 50; YOLO
• ISSN: 1568-4946
• DOI: 10.1016/j.asoc.2024.111921

Object detection models have been experiencing significant improvements over the years due to advancements in deep learning techniques, increased availability of large-scale annotated datasets, and computational resources. Different object detection models have varying levels of accuracy, speed, and robustness. With the increasing complexity and diversity of object detection models, it becomes a problem for researchers and practitioners to choose the most suitable model for their specific needs. This research paper outlines the escalating demand for robust comparison of object detection models in response to rapidly advancing technology. This evaluation helps in identifying the strengths and weaknesses of these models and selecting the most suitable one for a specific task. This highlights a significant challenge stemming from the lack of recent comparative studies on object detection models across various image qualities, object sizes, and training data sizes. The above challenges are tackled by a meticulous evaluation of three state-of-the-art object detection models: YOLO-v8, Faster R-CNN with ResNet 50 and 101 backbones, and End-to-End Object Detection Transformers (DETR) utilizing ResNet 50 and 101 backbones by employing a rigorous assessment framework encompassing mean Average Precision (mAP), accuracy, and inference speed. This study focuses on thoroughly examining how well the models perform across three different datasets: TACO, PlastOPol, and TACO 4.5. These datasets consist of open-world images captured in real-time from various locations. They include 1500, 2500, and 6500 images respectively, depicting real-world environments with varying lighting conditions and complex backgrounds. The results identify YOLOv8 as the superior model for high and medium-quality images, while Faster R-CNN performs better for low-quality images. However, DETR's accuracy falls short compared to other models. The paper fills a crucial gap in understanding model performance across varying image qualities and object sizes and helps in taking informed decisions in object detection systems. [Display omitted] •Qualitative and Quantitative Comparison of object detection models.•Assessment of model performance under real-world challenges in environmental scenes.•Insights into model suitability for environmental monitoring, using detection models.•Evaluating object detection models over the different sizes (1500, 2500, and 6500 images) and evaluating it’s accuracy.