Reputation Systems Evaluation Survey

Various reputation systems have been proposed for a broad range of distributed applications, such as peer-to-peer, ad-hoc, and multiagent systems. Their evaluation has been mostly based on proprietary methods due to the lack of widely acceptable evaluation measures and methodologies. Differentiating...

Full description

Saved in:
Bibliographic Details
Published in:ACM computing surveys 2016-02, Vol.48 (3), p.1-28
Main Authors: Koutrouli, Eleni, Tsalgatidou, Aphrodite
Format: Article
Language:English
Subjects:
Acceptability
Benchmarking
Benchmarks
Dynamic tests
Dynamical systems
Dynamics
Information systems
Peer to peer computing
Proposals
Reputations
Resilience
Social networks
Studies
Vocabularies & taxonomies
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Various reputation systems have been proposed for a broad range of distributed applications, such as peer-to-peer, ad-hoc, and multiagent systems. Their evaluation has been mostly based on proprietary methods due to the lack of widely acceptable evaluation measures and methodologies. Differentiating factors in various evaluation approaches include the evaluation metrics, the consideration of the dynamic behavior of peers, the use of social networks, or the study of resilience to specific threat scenarios. The lack of a generally accepted common evaluation framework hinders the objective evaluation and comparison of different reputation systems. Aiming at narrowing the gap in the research area of objective evaluation of reputation systems, in this article, we study the various approaches to evaluating and comparing reputation systems, present them in a taxonomy, and analyze their strengths and limitations, with special focus on works suggesting a Common Evaluation Framework (CEF). We identify the challenges for a widely accepted CEF that enables testing and benchmarking of reputation systems, and we present the required properties for such a CEF; we also present an analysis of current CEF-related works in the context of the identified properties and our related proposals.
ISSN:0360-0300
1557-7341
DOI:10.1145/2835373