Ranking versus rating in peer review of research grant applications

The allocation of public funds for research has been predominantly based on peer review where reviewers are asked to rate an application on some form of ordinal scale from poor to excellent. Poor reliability and bias of peer review rating has led funding agencies to experiment with different approac...

Full description

Saved in:
Bibliographic Details
Published in:PloS one 2023-10, Vol.18 (10), p.e0292306-e0292306
Main Authors: Tamblyn, Robyn, Girard, Nadyne, Hanley, James, Habib, Bettina, Mota, Adrian, Khan, Karim M, Ardern, Clare L
Format: Article
Language:English
Subjects:
Bias
Careers
Comparative analysis
Decision making
Evaluation
Funding
Grants-in-aid
Peer review
People and Places
Productivity
Quality assessment
Ranking
Ratings & rankings
Reliability
Research and Analysis Methods
Research grants
Research institutes
Reviews
Science Policy
Sex discrimination
Social Sciences
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The allocation of public funds for research has been predominantly based on peer review where reviewers are asked to rate an application on some form of ordinal scale from poor to excellent. Poor reliability and bias of peer review rating has led funding agencies to experiment with different approaches to assess applications. In this study, we compared the reliability and potential sources of bias associated with application rating with those of application ranking in 3,156 applications to the Canadian Institutes of Health Research. Ranking was more reliable than rating and less susceptible to the characteristics of the review panel, such as level of expertise and experience, for both reliability and potential sources of bias. However, both rating and ranking penalized early career investigators and favoured older applicants. Sex bias was only evident for rating and only when the applicant's H-index was at the lower end of the H-index distribution. We conclude that when compared to rating, ranking provides a more reliable assessment of the quality of research applications, is not as influenced by reviewer expertise or experience, and is associated with fewer sources of bias. Research funding agencies should consider adopting ranking methods to improve the quality of funding decisions in health research.
ISSN:1932-6203
1932-6203
DOI:10.1371/journal.pone.0292306