Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors

We propose a robustness testing approach for software systems that process large amounts of data. Our method uses metamorphic relations to check software output for erroneous input in the absence of a tangible test oracle. We use this technique to test two major citation database systems: Scopus and...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on software engineering 2021-06, Vol.47 (6), p.1164-1183
Main Authors: Zhou, Zhi Quan, Tse, T. H., Witheridge, Matt
Format: Article
Language:English
Subjects:
citation count
Citation indexes
Database systems
fault-based testing
Fuzzing
Google
Google Scholar
journal impact factor
Metamorphic robustness testing
metamorphic testing
oracle problem
Robustness
Scopus
Software
Software engineering
software robustness
Software testing
verification and validation
Web of Science
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We propose a robustness testing approach for software systems that process large amounts of data. Our method uses metamorphic relations to check software output for erroneous input in the absence of a tangible test oracle. We use this technique to test two major citation database systems: Scopus and the Web of Science. We report a surprising finding that the inclusion of hyphens in paper titles impedes citation counts, and that this is a result of the lack of robustness of the citation database systems in handling hyphenated paper titles. Our results are valid for the entire literature as well as for individual fields such as chemistry. We further find a strong and significant negative correlation between the journal impact factor (JIF) of IEEE Transactions on Software Engineering (TSE) and the percentage of hyphenated paper titles published in TSE. Similar results are found for ACM Transactions on Software Engineering and Methodology. A software engineering field-wide study reveals that the higher JIF-ranked journals are publishing a lower percentage of papers with hyphenated titles. Our results challenge the common belief that citation counts and JIFs are reliable measures of the impact of papers and journals, as they can be distorted simply by the presence of hyphens in paper titles.
ISSN:0098-5589
1939-3520
DOI:10.1109/TSE.2019.2915065