SCiMet: Stable, sCalable and reliable Metric-based framework for quality assessment in collaborative content generation systems

• Published in: Journal of informetrics 2021-05, Vol.15 (2), p.101127, Article 101127
• Main Authors: Allahbakhsh, Mohammad, Amintoosi, Haleh, Behkamal, Behshid, Beheshti, Amin, Bertino, Elisa
• Format: Article
• Language: English
• Subjects: Attack-resilient; Collaborative content; Quality assessment; Quality metric; Scientometrics
• ISSN: 1751-1577; 1875-5879
• DOI: 10.1016/j.joi.2020.101127

•Existing metrics for evaluating artifacts such as research articles are not comprehensive.•Existing metrics such as h-index or impact factor mostly rely on citation count.•Quality of artifacts (articles), contributors (authors) and venues are interrelated.•Comprehensive, interrelated, iteratively computed metrics are harder to manipulate•SCiMet (proposed method) is strong against manipulation and also applicable to other collaborative content generation systems. In collaborative content generation (CCG), such as publishing scientific articles, a group of contributors collaboratively generates artifacts available through a venue. The main concern in such systems is the quality. A remarkable range of research considers quality metrics partially when dealing with the quality of artifacts, contributors, and venues. However, such approaches have several drawbacks. One of the most notable ones is that they are not comprehensive in terms of the metrics to evaluate all entities, including artifacts, contributors, and venues. Also, they are vulnerable to potential attacks. In this paper, we propose a novel iterative definition in which the quality of artifacts, collaborators, and venues are defined interconnectedly. In our framework, the quality of an artifact is defined based on the quality of its contributors, venue, references, and citations. The quality of a contributor is defined based on the quality of his artifacts, collaborators, and the venues. Quality of a venue is defined based on both quality of artifacts and contributors. We propose a data model, formulations, and an algorithm for the proposed approach. We also compare the robustness of our approach against malicious manipulations with two well-known related approaches. The comparison results show the superiority of our method over other related approaches.