Why Accuracy is Not Enough: The Need for Consistency in Object Detection

Object detectors are vital to many modern computer vision applications. However, even state-of-the-art object detectors are not perfect. On two images that look similar to human eyes, the same detector can make different predictions because of small image distortions like camera sensor noise and lig...

Full description

Saved in:
Bibliographic Details
Published in:IEEE multimedia 2022-07, Vol.29 (3), p.8-16
Main Authors: Tung, Caleb, Goel, Abhinav, Bordwell, Fischer, Eliopoulos, Nick, Hu, Xiao, Lu, Yung-Hsiang, Thiruvathukal, George K.
Format: Magazinearticle
Language:English
Subjects:
Accuracy
Behavioral sciences
Cameras
Computer vision
Consistency
Detectors
Distortion
Frames (data processing)
Image compression
Measurement
Multiple target tracking
Neural networks
Object detection
Object recognition
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Object detectors are vital to many modern computer vision applications. However, even state-of-the-art object detectors are not perfect. On two images that look similar to human eyes, the same detector can make different predictions because of small image distortions like camera sensor noise and lighting changes. This problem is called inconsistency. Existing accuracy metrics do not properly account for inconsistency, and similar work in this area only targets improvements on artificial image distortions. Therefore, we propose a method to use nonartificial video frames to measure object detection consistency over time, across frames. Using this method, we show that the consistency of modern object detectors ranges from 83.2% to 97.1% on different video datasets from the multiple object tracking challenge. We conclude by showing that applying image distortion corrections such as WEBP Image Compression and Unsharp Masking can improve consistency by as much as 5.1%, with no loss in accuracy.
ISSN:1070-986X
1941-0166
DOI:10.1109/MMUL.2022.3175239