Philosophy of Science 79 (5):859-870 (2012)

Authors
Carole J. Lee
University of Washington
Abstract
Psychometrically oriented researchers construe low inter-rater reliability measures for expert peer reviewers as damning for the practice of peer review. I argue that this perspective overlooks different forms of normatively appropriate disagreement among reviewers. Of special interest are Kuhnian questions about the extent to which variance in reviewer ratings can be accounted for by normatively appropriate disagreements about how to interpret and apply evaluative criteria within disciplines during times of normal science. Until these empirical-cum-philosophical analyses are done, it will remain unclear the extent to which low inter-rater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers.
Keywords peer review  social epistemology  science of science
Categories (categorize this paper)
DOI 10.1086/667841
Options
Edit this record
Mark as duplicate
Export citation
Find it on Scholar
Request removal from index
Revision history

Download options

PhilArchive copy


Upload a copy of this paper     Check publisher's policy     Papers currently archived: 72,564
Through your library

References found in this work BETA

Studies in the Way of Words.Paul Grice - 1989 - Philosophy 65 (251):111-113.
Logic and Conversation.H. Paul Grice - 1989 - In Studies in the Way of Words. Harvard University Press. pp. 22-40.
Bias in Peer Review.Carole J. Lee, Cassidy R. Sugimoto, Guo Zhang & Blaise Cronin - 2013 - Journal of the American Society for Information Science and Technology 64 (1):2-17.

View all 16 references / Add more references

Citations of this work BETA

Is Peer Review a Good Idea?Remco Heesen & Liam Kofi Bright - 2021 - British Journal for the Philosophy of Science 72 (3):635-663.
Bias in Peer Review.Carole J. Lee, Cassidy R. Sugimoto, Guo Zhang & Blaise Cronin - 2013 - Journal of the American Society for Information Science and Technology 64 (1):2-17.
The Social Dimensions of Scientific Knowledge.Helen Longino - 2008 - Stanford Encyclopedia of Philosophy.
Commensuration Bias in Peer Review.Carole J. Lee - 2015 - Philosophy of Science 82 (5):1272-1283,.
The Reference Class Problem for Credit Valuation in Science.Carole J. Lee - 2020 - Philosophy of Science 87 (5):1026-1036.

View all 12 citations / Add more citations

Similar books and articles

Bias in Peer Review.Carole J. Lee, Cassidy R. Sugimoto, Guo Zhang & Blaise Cronin - 2013 - Journal of the American Society for Information Science and Technology 64 (1):2-17.
The Principles and Practices of Peer Review.Ronald N. Kostoff - 1997 - Science and Engineering Ethics 3 (1):19-34.
Evidence for the Effectiveness of Peer Review.Robert H. Fletcher & Suzanne W. Fletcher - 1997 - Science and Engineering Ethics 3 (1):35-50.
Using a Dialectical Scientific Brief in Peer Review.Arthur E. Stamps - 1997 - Science and Engineering Ethics 3 (1):85-98.
‘Peer Review’ Culture.Malcolm Atkinson - 2001 - Science and Engineering Ethics 7 (2):193-204.

Analytics

Added to PP index
2012-07-24

Total views
64 ( #182,240 of 2,533,567 )

Recent downloads (6 months)
3 ( #199,858 of 2,533,567 )

How can I increase my downloads?

Downloads

My notes