To arrive at their final evaluation of a manuscript or grant proposal, reviewers must convert a submission’s strengths and weaknesses for heterogeneous peer review criteria into a single metric of quality or merit. I identify this process of commensuration as the locus for a new kind of peer review bias. Commensuration bias illuminates how the systematic prioritization of some peer review criteria over others permits and facilitates problematic patterns of publication and funding in science. Commensuration bias also foregrounds a range (...) of structural strategies for realigning peer review practices and institutions with the aims of science. (shrink)

Those who comment on modern scientific institutions are often quick to praise institutional structures that leave scientists to their own devices. These comments reveal an underlying presumption that scientists do best when left alone—when they operate in what we call the ‘scientific state of nature’. Through computer simulation, we challenge this presumption by illustrating an inefficiency that arises in the scientific state of nature. This inefficiency suggests that one cannot simply presume that science is most efficient when institutional control is (...) absent. In some situations, actively encouraging unpopular, risky science would improve scientific outcomes. 1 Introduction2 Scientists and Bandits3 Choosing an ϵ4 Structure of Communication5 Discussion. (shrink)

Should editors of scientific journals practice triple-anonymous reviewing? I consider two arguments in favor. The first says that insofar as editors’ decisions are affected by information they would not have had under triple-anonymous review, an injustice is committed against certain authors. I show that even well-meaning editors would commit this wrong and I endorse this argument. The second argument says that insofar as editors’ decisions are affected by information they would not have had under triple-anonymous review, it will negatively affect (...) the quality of published papers. I distinguish between two kinds of biases that an editor might have. I show that one of them has a positive effect on quality and the other a negative one, and that the combined effect could be either positive or negative. Thus I do not endorse the second argument in general. However, I do endorse this argument for certain fields, for which I argue that the positive effect does not apply. (shrink)

Peer reviewers at many funding agencies and scientific journals are asked to score submissions both on individual criteria and overall. The overall scores should be some kind of aggregate of the criteria scores. Carole Lee identifies this as a potential locus for bias to enter the peer review process, which she calls commensuration bias. Here I view the aggregation of scores through the lens of social choice theory. I argue that, when reviewing grant proposals, it is in many cases impossible (...) to avoid commensuration bias. (shrink)

Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available evidence, they are either positive or neutral. (...) We conclude that on present evidence abolishing peer review weakly dominates the status quo. (shrink)

It is valuable for inquiry to have researchers who are committed advocates of their own theories. However, in light of pervasive disagreement, such a commitment is not well explained by the idea that researchers believe their theories. Instead, this commitment, the rational attitude to take toward one’s favored theory during the course of inquiry, is what I call endorsement. Endorsement is a doxastic attitude, but one which is governed by a different type of epistemic rationality. This inclusive epistemic rationality is (...) sensitive to reasons beyond those to think the particular proposition in question is true. Instead, it includes extrinsic epistemic reasons, which concern the health of inquiry more generally. Such extrinsic reasons include the distribution of cognitive labor that a researcher will contribute to by endorsing a particular theory. Recognizing endorsement and inclusive epistemic rationality thus allows us to smooth a tension between individual rationality and collective rationality. It does so by showing how it can be epistemically rational to endorse a theory on the basis of the way this endorsement will benefit collective inquiry. I provide a decision theoretic treatment for inclusive epistemic rationality and endorsement which illustrates how this can be accomplished. (shrink)

Intellectual courage requires acting to promote epistemic goods despite significant risk of harm. Courage is distinguished from recklessness and cowardice because the expected epistemic benefit of a courageous action outweighs (in some sense) the threatened harm. Sometimes, however, inquirers pursue theories that are not best supported by their current evidence. For these inquirers, the expected epistemic benefit of their actions cannot be explained by appeal to their evidence alone. The probability of pursuing the true theory cannot contribute enough to the (...) expected epistemic benefit for the action to count as courageous rather than reckless. Thus, there must be some other epistemic consideration which favors their action, besides evidence for their theory. I argue that the proper account of intellectual courage requires recognition of inquisitive reasons: a distinct category of epistemic reasons which concern successful inquiry. This category includes reasons to think a theory itself is promising, e.g., that the theory suggests potentially fruitful new research. It also includes social epistemic reasons, e.g., that pursuing a theory will improve the distribution of cognitive labor. Inquisitive reasons help explain why researchers who pursue improbable theories count as intellectually courageous, rather than reckless. The expected epistemic benefit of an action is partially determined by the inquisitive reasons in its favor. On my account, intellectually courageous inquiry requires acting in a way that is sensitive to inquisitive reasons. (shrink)

A part of the scientific literature consists of intermediate results within a longer project. Scientists often publish a first result in the course of their work, while aware that they should soon achieve a more advanced result from this preliminary result. Should they follow the proverb “a bird in the hand is worth two in the bush”, and publish any intermediate result they get? This is the normative question addressed in this paper. My aim is to clarify, to refine, and (...) to assess informal arguments about the choice whether to publish intermediate results. To this end, I adopt a rational decision framework, supposing some utility or preferences, and I propose a formal model. The best publishing strategy turns out to depend on the research situation. In some simple circumstances, even selfish and short-minded scientists should publish their intermediate results, and should thus behave like their altruistic peers, i. e. like society would like them to behave. In other research situations, with inhomogeneous reward or difficulty profiles, the best strategy is opposite. These results suggest qualified philosophical morals. (shrink)

There are now several proposals for introducing random elements into the process of funding allocation for research, and some initial implementation of this policy by funding bodies. The proposals have been supported on efficiency grounds, with models, including social epistemology models, showing random allocation could increase the generation of significant truths in a community of scientists when compared to funding by peer review. The models in the literature are, however, fairly abstract. This paper introduces some of the considerations that are (...) required to build on the modelling work towards a fully-fledged policy proposal, including issues of cost and fairness. (shrink)

There is an increasing push by journals to ensure that data and products related to published papers are shared as part of a cultural move to promote transparency, reproducibility, and trust in the scientific literature. Yet few journals commit to evaluating their effectiveness in implementing reporting standards aimed at meeting those goals (1, 2). Similarly, though the vast majority of journals endorse peer review as an approach to ensure trust in the literature, few make their peer review data available to (...) evaluate effectiveness toward achieving concrete measures of quality, including consistency and completeness in meeting reporting standards. Remedying these apparent disconnects is critical for closing the gap between guidance recommendations and actual reporting behavior. We see this as a collective action problem requiring leadership and investment by publishers, who can be incentivized through mechanisms that allow them to manage reputational risk and through continued innovation in journal assessment. (shrink)