Reliability of Editors' Subjective Quality Ratings of Peer Reviews of Manuscripts

CONTEXT.— Quality of reviewers is crucial to journal quality, but there are usually too many for editors to know them all personally. A reliable method of rating them (for education and monitoring) is needed. OBJECTIVE.— Whether editors' quality ratings of peer reviewers are reliable and how th...

Full description

Saved in:
Bibliographic Details
Published in:JAMA : the journal of the American Medical Association 1998-07, Vol.280 (3), p.229-231
Main Authors: Callaham, Michael L, Baxt, William G, Waeckerle, Joseph F, Wears, Robert L
Format: Article
Language:English
Subjects:
Biological and medical sciences
Medical research
Medical sciences
Peer review
Peer Review - standards
Prospective Studies
Public health. Hygiene
Public health. Hygiene-occupational medicine
Publishing - standards
Quality Control
Reliability
Teaching. Deontology. Ethics. Legislation
Citations: Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:CONTEXT.— Quality of reviewers is crucial to journal quality, but there are usually too many for editors to know them all personally. A reliable method of rating them (for education and monitoring) is needed. OBJECTIVE.— Whether editors' quality ratings of peer reviewers are reliable and how they compare with other performance measures. DESIGN.— A 3.5-year prospective observational study. SETTING.— Peer-reviewed journal. PARTICIPANTS.— All editors and peer reviewers who reviewed at least 3 manuscripts. MAIN OUTCOME MEASURES.— Reviewer quality ratings, individual reviewer rate of recommendation for acceptance, congruence between reviewer recommendation and editorial decision (decision congruence), and accuracy in reporting flaws in a masked test manuscript. INTERVENTIONS.— Editors rated the quality of each review on a subjective 1 to 5 scale. RESULTS.— A total of 4161 reviews of 973 manuscripts by 395 reviewers were studied. The within-reviewer intraclass correlation was 0.44 (P
ISSN:0098-7484
1538-3598
DOI:10.1001/jama.280.3.229