Grant Peer Review: Improving Inter-Rater Reliability with Training

This study developed and evaluated a brief training program for grant reviewers that aimed to increase inter-rater reliability, rating scale knowledge, and effort to read the grant review criteria. Enhancing reviewer training may improve the reliability and accuracy of research grant proposal scorin...

Full description

Saved in:
Bibliographic Details
Published in:PloS one 2015-06, Vol.10 (6), p.e0130450-e0130450
Main Authors: Sattler, David N, McKnight, Patrick E, Naney, Linda, Mathis, Randy
Format: Article
Language:English
Subjects:
Accuracy
Adult
Bias
Cognitive biases
Colleges & universities
Criteria
Decision making
Female
Funding
GENERAL AND MISCELLANEOUS
Grants
Health sciences
Humans
Knowledge
Male
National Institutes of Health (U.S.)
Noise
Peer review
Peer Review, Research
Psychologists
Public health
R&D
Reliability analysis
Reliability aspects
Research & development
Research facilities
Research programs
Research Support as Topic
Reviews
Studies
Surveys and Questionnaires
Systematic review
Teaching
Training
United States
Validity
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This study developed and evaluated a brief training program for grant reviewers that aimed to increase inter-rater reliability, rating scale knowledge, and effort to read the grant review criteria. Enhancing reviewer training may improve the reliability and accuracy of research grant proposal scoring and funding recommendations. Seventy-five Public Health professors from U.S. research universities watched the training video we produced and assigned scores to the National Institutes of Health scoring criteria proposal summary descriptions. For both novice and experienced reviewers, the training video increased scoring accuracy (the percentage of scores that reflect the true rating scale values), inter-rater reliability, and the amount of time reading the review criteria compared to the no video condition. The increase in reliability for experienced reviewers is notable because it is commonly assumed that reviewers--especially those with experience--have good understanding of the grant review rating scale. The findings suggest that both experienced and novice reviewers who had not received the type of training developed in this study may not have appropriate understanding of the definitions and meaning for each value of the rating scale and that experienced reviewers may overestimate their knowledge of the rating scale. The results underscore the benefits of and need for specialized peer reviewer training.
ISSN:1932-6203
1932-6203
DOI:10.1371/journal.pone.0130450