Structure-Measure: A New Way to Evaluate Foreground Maps

Foreground map evaluation is crucial for gauging the progress of object segmentation algorithms, in particular in the field of salient object detection where the purpose is to accurately detect and segment the most salient object in a scene. Several measures (e.g., area-under-the-curve, F1-measure,...

Full description

Saved in:
Bibliographic Details
Published in:International journal of computer vision 2021-09, Vol.129 (9), p.2622-2638
Main Authors: Cheng, Ming-Ming, Fan, Deng-Ping
Format: Article
Language:English
Subjects:
Algorithms
Analysis
Artificial Intelligence
Computer Imaging
Computer Science
Evaluation
Gaging
Image Processing and Computer Vision
Image segmentation
Object recognition
Pattern Recognition
Pattern Recognition and Graphics
Salience
Similarity
Vision
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Foreground map evaluation is crucial for gauging the progress of object segmentation algorithms, in particular in the field of salient object detection where the purpose is to accurately detect and segment the most salient object in a scene. Several measures (e.g., area-under-the-curve, F1-measure, average precision, etc.) have been used to evaluate the similarity between a foreground map and a ground-truth map. The existing measures are based on pixel-wise errors and often ignore the structural similarities. Behavioral vision studies, however, have shown that the human visual system is highly sensitive to structures in scenes. Here, we propose a novel, efficient (0.005 s per image), and easy to calculate measure known as S-measure (structural measure) to evaluate foreground maps. Our new measure simultaneously evaluates region-aware and object-aware structural similarity between a foreground map and a ground-truth map. We demonstrate superiority of our measure over existing ones using 4 meta-measures on 5 widely-used benchmark datasets. Furthermore, we conduct a behavioral judgment study over a new database. Data from 45 subjects shows that on average they preferred the saliency maps chosen by our measure over the saliency maps chosen by the state-of-the-art measures. Our experimental results offer new insights into foreground map evaluation where current measures fail to truly examine the strengths and weaknesses of models. Code: https://github.com/DengPingFan/S-measure .
ISSN:0920-5691
1573-1405
DOI:10.1007/s11263-021-01490-8