This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the (...) scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. (shrink)

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the (...) scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. (shrink)

Two important aspects of the relationship between peer review and innovation includes the acceptance of articles for publication in journals and the assessment of applications for grants for the funding of research work. While there are well-known examples of the rejection by journals of first choice of many papers that have radically changed the way we think about the world outside ourselves, such papers do get published eventually, however tortuous the process required. With grant applications the situation differs in that (...) the refusal of a grant necessarily curtails the possible research that may be attempted. Here there are many reasons for conservatism and reservation as to the ability of a grant allocation process based on peer review to deliver truly innovative investigations. Other methods are needed; although such methods need not be applied across the board, they should constitute the methods whereby some 10–20% of the grant monies are assigned. The nomination of prizes for specific accomplishments is one way of achieving innovation although this presumes that investigators or institution already have available the money necessary to effect the innovations; otherwise it is a question of the selection and funding of particular individuals or institutions and requiring them to solve particular problems that are set in the broadest of terms. (shrink)

A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since they make decisions that have a (...) significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making. (shrink)

Recently, high media visibility was reached by an experiment that involved “hoaxlike deception” of journals within humanities and social sciences. Its aim was to provide evidence of “inadequate” quality standards especially within gender studies. The article discusses the project in the context of both previous systematic studies of peer reviewing and scientific hoaxes and analyzes its possible empirical outcomes. Despite claims to the contrary, the highly political, both ethically and methodologically flawed “experiment” failed to provide the evidence it sought. The (...) experiences can be summed up as follows: journals with higher impact factors were more likely to reject papers submitted as part of the project; the chances were better, if the manuscript was allegedly based on empirical data; peer reviews can be an important asset in the process of revising a manuscript; and when the project authors, with academic education from neighboring disciplines, closely followed the reviewers’ advice, they were able to learn relatively quickly what is needed for writing an acceptable article. The boundary between a seriously written paper and a “hoax” gradually became blurred. Finally, the way the project ended showed that in the long run, the scientific community will uncover fraudulent practices. (shrink)

The operative nucleus of peer review processes has largely remained a ‘black box’ to analytical empirical research. There is a lack of direct insights into the communicative machinery of peer review, i.e., into ‘gatekeeping in action’. This article attempts to fill a small part of this huge research gap. It is based on an ethnographic case study about peer review communication in a sociological journal. It looks at the final phase of the peer review process: the decisions taken in the (...) oral communication of editors’ meetings. The article describes this meeting as an instrumented talk, supported by written tools and constrained by necessary procedural outcomes. It analyzes examples of interactive negotiations of manuscripts and, in the end, it discusses the procedural rationality of peer review as a public sphere for professional judgment. (shrink)

Epistemological differences fuel continuous and frequently divisive debates in the social sciences and the humanities. Sociologists have yet to consider how such differences affect peer evaluation. The empirical literature has studied distributive fairness, but neglected how epistemological differences affect perception of fairness in decision making. The normative literature suggests that evaluators should overcome their epistemological differences by ‘‘translating’’ their preferred standards into general criteria of evaluation. However, little is known about how procedural fairness actually operates. Drawing on eighty-one interviews with (...) panelists serving on five multidisciplinary fellowship competitions in the social sciences and the humanities, we show that Evaluators generally draw on four epistemological styles to make arguments in favor of and against proposals. These are the constructivist, comprehensive, positivist, and utilitarian styles; and Peer reviewers define a fair decision-making process as one in which panelists engage in ‘‘cognitive contextualization,’’ that is, use epistemological styles most appropriate to the field or discipline of the proposal under review. (shrink)

A relatively high incidence of unsatisfactory review decisions is widely recognised and acknowledged as ‘the peer review problem’. Factors contributing to this problem are identified and examined. Specific examples of unreasonable rejection are considered. It is concluded that weaknesses of the ‘peer review’ system are significant and that they are well known or readily recognisable but that necessary counter-measures are not always enforced. Careful management is necessary to discount hollow opinion or error in review comment. Review and referee functions should (...) be quite separate. (shrink)

A relatively high incidence of unsatisfactory review decisions is widely recognised and acknowledged as ‘the peer review problem’. Factors contributing to this problem are identified and examined. Specific examples of unreasonable rejection are considered. It is concluded that weaknesses of the ‘peer review’ system are significant and that they are well known or readily recognisable but that necessary counter-measures are not always enforced. Careful management is necessary to discount hollow opinion or error in review comment. Review and referee functions should (...) be quite separate. (shrink)

Contemporary debates about scientific institutions and practice feature many proposed reforms. Most of these require increased efforts from scientists. But how do scientists’ incentives for effort interact? How can scientific institutions encourage scientists to invest effort in research? We explore these questions using a game-theoretic model of publication markets. We employ a base game between authors and reviewers, before assessing some of its tendencies by means of analysis and simulations. We compare how the effort expenditures of these groups interact in (...) our model under a variety of settings, such as double-blind and open review systems. We make a number of findings, including that open review can increase the effort of authors in a range of circumstances and that these effects can manifest in a policy-relevant period of time. However, we find that open review’s impact on authors’ efforts is sensitive to the strength of several other influences. (shrink)

This brief article presents the introduction and draft of the fundamental ideas developed at length in the book of the same title, which gives a challenging point of view about science and its history/philosophy/sociology. Science is in decline. After centuries of great achievements, the exhaustion of new forms and fatigue have reached our culture in all of its manifestations including the pure sciences. Our society is saturated with knowledge which does not offer people any sense in their lives. There is (...) a loss of ideals in the search for great truths and a shift towards an anodyne specialized industry. (shrink)