What is meaningful research and how should we measure it?

• Published in: Scientometrics 2020-10, Vol.125 (1), p.153-169
• Main Authors: Helmer, Sven, Blumenthal, David B., Paschen, Kathrin
• Format: Article
• Language: English
• Subjects: Computer Science; Evaluation; Feedback loops; Information Storage and Retrieval; Library Science
• ISSN: 0138-9130; 1588-2861
• DOI: 10.1007/s11192-020-03649-5

We discuss the trend towards using quantitative metrics for evaluating research. We claim that, rather than promoting meaningful research, purely metric-based research evaluation schemes potentially lead to a dystopian academic reality, leaving no space for creativity and intellectual initiative. After sketching what the future could look like if quantitative metrics are allowed to proliferate, we provide a more detailed discussion on why research is so difficult to evaluate and outline approaches for avoiding such a situation. In particular, we characterize meaningful research as an essentially contested concept and argue that quantitative metrics should always be accompanied by operationalized instructions for their proper use and continuously evaluated via feedback loops. Additionally, we analyze a dataset containing information about computer science publications and their citation history and indicate how quantitative metrics could potentially be calibrated via alternative evaluation methods such as test of time awards. Finally, we argue that, instead of over-relying on indicators, research environments should primarily be based on trust and personal responsibility.