Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available evidence, they are (...) either positive or neutral. We conclude that on present evidence abolishing peer review weakly dominates the status quo. (shrink)

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we (...) selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated. (shrink)

At present, peer review is the most common method used by funding agencies to make decisions about resource allocation. But how reliable, efficient, and fair is it in practice? The ex ante evaluation of scientific novelty is a fundamentally uncertain endeavor; bias and chance are embedded in the final outcome. In the current study, I will examine some of the most central problems of peer review and highlight the possible benefits of using a lottery as an (...) alternative decision-making mechanism. Lotteries are driven by chance, not reason. The argument made in the study is that the epistemic landscape could benefit in several respects by using a lottery, thus avoiding all types of bias, disagreement, and other limitations associated with the peer review process. Funding agencies could form a pool of funding applicants who have minimal qualification levels and then select randomly within that pool. The benefits of a lottery would not only be that it saves time and resources, but also that it contributes to a more dynamic selection process and increases the epistemic diversity, fairness, and impartiality within academia. (shrink)

This Open Access book about the Zadeh Project demonstrates and explores a core question in clinical ethics: how can ethics consultants be accountable in the face of a robust plurality of ethical standpoints, especially those that underwrite practices and methods for doing ethics consultation as well as those viewpoints and values encountered in daily clinical ethics practice? Underscoring this question is the recognition that the field of clinical ethics consultation has arrived at a crucial point in its maturation. Many efforts (...) are underway to more formally “professionalize” the field, with most aimed toward stabilizing a specific set of institutional considerations. Stretched between these institutional and practical initiatives resides a crucial set of of ethical considerations, chief among them the meaning and scope of responsibility for clinical ethics consultants. Developed around a long-form case scenario, the Zadeh Project provides a multi-layered series of “peer-reviews”: critique of the actions of the case scenario’s ethics consultant; reflection on clinical ethics method; examination of the many ways that commitments to method and practice can, and do, intersect, overlap, and alter one another. The design and format of this book thus models a key element for clinical ethics practice: the need and ability to provide careful and thoughtful explanation of core moral considerations that emerge among diverse standpoints. Specifically designed for those studying to become and those who are ethics consultants, this book, with its innovative and multi-layered approach, allows readers to share a peer-review-like experience that shows accountability to be what it is, an ethical, not merely procedural or administrative, undertaking. (shrink)

Peer review and publication is one of the factors proposed in Daubert v. Merrell Dow Pharmaceuticals, Inc. as indicia of the reliability of scientific testimony. This Article traces the origins of the peer-review system, the process by which it became standard at scientific and medical journals, and the many roles it now plays. Additionally, the Author articulates the epistemological rationale for pre-publication peer-review and the inherent limitations of the system as a scientific quality-control mechanism. (...) The Article explores recent changes in science, in scientific publishing, and in the academy that have put the system under strain. The Author argues that Justice Blackmun's advice to courts - that peer-reviewed publication is relevant, but is not dispositive - is of little practical help. Instead, the Author suggests questions that courts should ask in assessing the significance of the fact that testimony is, or is not, based on peer-reviewed publication and illustrates with reference to another Bendectin case, Blum v. Merrell Dow Pharmaceuticals, Inc., where some of these questions were asked. (shrink)

Peer review has long been regarded as the gold standard of scientific publication, essential to the integrity of science itself. But, as any publishing scientist knows, peer review has its downside, including long delays and reviewer bias. Until the coming of the Internet, there appeared to be no alternative. Now, articles appear online as preprints almost immediately upon submission. But they lack peer review and thus their scientific standing can be questioned. Post-publication discussion platforms (...) such as PubPeer have proven useful, but are no substitute for pre-publication peer review. Nevertheless, some may be tempted to believe that peer review can now be done without. This article challenges that view by analyzing a recent, non-peer-reviewed article in Skeptical Enquirer, a magazine published by the Committee for Skeptical Enquiry (CSI). The article, “Sodom Meteor Strike Claims Should Be Taken With a Pillar of Salt,” casts doubt on one of the most widely read scientific articles of the last decade and provides a stern warning of the cost of abandoning peer review. (shrink)

Pre-publication certification through peer review stands in need of philosophical examination. In this paper, philosopher-psychologist Steven James Bartlett recalls the arguments marshalled four hundred years ago by English poet John Milton against restraint of publication by the "gatekeepers of publication," AKA today's peer reviewers.

Previous research has found that funding disparities are driven by applications’ final impact scores and that only a portion of the black/white funding gap can be explained by bibliometrics and topic choice. Using National Institutes of Health R01 applications for council years 2014–2016, we examine assigned reviewers’ preliminary overall impact and criterion scores to evaluate whether racial disparities in impact scores can be explained by application and applicant characteristics. We hypothesize that differences in commensuration—the process of combining criterion scores into (...) overall impact scores—disadvantage black applicants. Using multilevel models and matching on key variables including career stage, gender, and area of science, we find little evidence for racial disparities emerging in the process of combining preliminary criterion scores into preliminary overall impact scores. Instead, preliminary criterion scores fully account for racial disparities—yet do not explain all of the variability—in preliminary overall impact scores. (shrink)

Peer-review is neither reliable, fair, nor a valid basis for predicting ‘impact’: as quality control, peer-review is not fit for purpose. Endorsing the consensus, I offer a reframing: while a normative social process, peer-review also shapes the writing of a scientific paper. In so far as ‘cognition’ describes enabling conditions for flexible behavior, the practices of peer-review thus constrain knowledge-making. To pursue cognitive functions of peer-review, however, manuscripts must be seen (...) as ‘symbolizations’, replicable patterns that use technologically enabled activity. On this bio-cognitive view, peer-review constrains knowledge-making by writers, editors, reviewers. Authors are prompted to recursively re-aggregate symbolizations to present what are deemed acceptable knowledge claims. How, then, can recursive re-embodiment be explored? In illustration, I sketch how the paper’s own content came to be re-aggregated: agonistic review drove reformatting of argument structure, changes in rhetorical ploys and careful choice of wordings. For this reason, the paper’s knowledge-claims can be traced to human activity that occurs in distributed cognitive systems. Peer-review is on the frontline in the knowledge sector in that it delimits what can count as knowing. Its systemic nature is therefore crucial to not only discipline-centered ‘real’ science but also its ‘post-academic’ counterparts. (shrink)

Two important aspects of the relationship between peer review and innovation includes the acceptance of articles for publication in journals and the assessment of applications for grants for the funding of research work. While there are well-known examples of the rejection by journals of first choice of many papers that have radically changed the way we think about the world outside ourselves, such papers do get published eventually, however tortuous the process required. With grant applications the situation differs (...) in that the refusal of a grant necessarily curtails the possible research that may be attempted. Here there are many reasons for conservatism and reservation as to the ability of a grant allocation process based on peer review to deliver truly innovative investigations. Other methods are needed; although such methods need not be applied across the board, they should constitute the methods whereby some 10–20% of the grant monies are assigned. The nomination of prizes for specific accomplishments is one way of achieving innovation although this presumes that investigators or institution already have available the money necessary to effect the innovations; otherwise it is a question of the selection and funding of particular individuals or institutions and requiring them to solve particular problems that are set in the broadest of terms. (shrink)

Peer review is a practice used in the evaluation of scientific and scholarly papers in order to select papers for publication in scholarly journals. The practice has also been extended to other domains, such as the evaluation of grant proposals, medical practice, book publication, and even to such areas as teaching evaluation. The primary area that has been of interest to sociologists, however, has been publication in scientific journals. The practice is usually understood to have begun in the (...) seventeenth century in the Royal Society in London, but it has also been claimed that there have been precursors to this practice. In the popular mind, peer review in science is a means of “bullet‐proofing” research, as a Wall Street Journal article once put it, that is to say as a guarantee of quality. But within science it has a very different meaning. (shrink)

Peer review process helps in evaluating and validating of research that is published in the journals. U.S. Office of Research Integrity reported that data fraudulence was found to be involved in 94% cases of misconduct from 228 identified articles between 1994–2012. If fraud in published article are significantly as high as reported, the question arise in mind, were these articles peer reviewed? Another report said that the reviewers failed to detect 16 cases of fabricated article of Jan (...) Hendrick Schon. Superficial peer reviewing process does not reveals suspicion of misconduct. Lack of knowledge of systemic review process not only demolish the academic integrity in publication but also loss the trust of the people of the institution, the nation, and the world. The aim of this review article is to aware stakeholders specially novice reviewers about the peer review system. Beginners will understand how to review an article and they can justify better action choices in dealing with reviewing an article. (shrink)

A relatively high incidence of unsatisfactory review decisions is widely recognised and acknowledged as ‘the peer review problem’. Factors contributing to this problem are identified and examined. Specific examples of unreasonable rejection are considered. It is concluded that weaknesses of the ‘peer review’ system are significant and that they are well known or readily recognisable but that necessary counter-measures are not always enforced. Careful management is necessary to discount hollow opinion or error in review (...) comment. Review and referee functions should be quite separate. (shrink)

This paper reviews the published empirical evidence concerning journal peer review consisting of 68 papers, all but three published since 1975. Peer review improves quality, but its use to screen papers has met with limited success. Current procedures to assure quality and fairness seem to discourage scientific advancement, especially important innovations, because findings that conflict with current beliefs are often judged to have defects. Editors can use procedures to encourage the publication of papers with innovative findings (...) such as invited papers, early-acceptance procedures, author nominations of reviewers, structured rating sheets, open peer review, results-blind review, and, in particular, electronic publication. Some journals are currently using these procedures. The basic principle behind the proposals is to change the decision from whether to publish a paper to how to publish it. (shrink)

BackgroundThere is increasing need for peer reviewers as the scientific literature grows. Formal education in biostatistics and research methodology during residency training is lacking. In this pilot study, we addressed these issues by evaluating a novel method of teaching residents about biostatistics and research methodology using peer review of standardized manuscripts. We hypothesized that mentored peer review would improve resident knowledge and perception of these concepts more than non-mentored peer review, while improving (...) class='Hi'>review quality.MethodsA partially blinded, randomized, controlled multi-center study was performed. Seventy-eight neurology residents from nine US neurology programs were randomized to receive mentoring from a local faculty member or not. Within a year, residents reviewed a baseline manuscript and four subsequent manuscripts, all with introduced errors designed to teach fundamental review concepts. In the mentored group, mentors discussed completed reviews with residents. Primary outcome measure was change in knowledge score between pre- and post-tests, measuring epidemiology and biostatistics knowledge. Secondary outcome measures included level of confidence in the use and interpretation of statistical concepts before and after intervention, and RQI score for baseline and final manuscripts.ResultsSixty-four residents (82%) completed initial review with gradual decline in completion on subsequent reviews. Change in primary outcome, the difference between pre- and post-test knowledge scores, did not differ between mentored (−8.5%) and non-mentored (−13.9%) residents (p = 0.48). Significant differences in secondary outcomes (using 5-point Likert scale, 5 = strongly agree) included mentored residents reporting enhanced understanding of research methodology (3.69 vs 2.61; p = 0.001), understanding of manuscripts (3.73 vs 2.87; p = 0.006), and application of study results to clinical practice (3.65 vs 2.78; p = 0.005) compared to non-mentored residents. There was no difference between groups in level of interest in peer review (3.00 vs 3.09; p = 0.72) or the quality of manuscript review assessed by the Review Quality Instrument (RQI) (3.25 vs 3.06; p = 0.50).ConclusionsWe used mentored peer review of standardized manuscripts to teach biostatistics and research methodology and introduce the peer review process to residents. Though knowledge level did not change, mentored residents had enhanced perception in their abilities to understand research methodology and manuscripts and apply study results to clinical practice. (shrink)

Open Peer Review: Educational Philosophy and Theory (EPAT)Michael A. Peters, Beijing Normal University, PR ChinaIn 2016 EPAT started experimenting with open peer review for articles that were part...

BackgroundOpen peer review practices are increasing in medicine and life sciences, but in social sciences and humanities they are still rare. We aimed to map out how editors of respected SSH journals perceive open peer review, how they balance policy, ethics, and pragmatism in the review processes they oversee, and how they view their own power in the process.MethodsWe conducted 12 pre-registered semi-structured interviews with editors of respected SSH journals. Interviews consisted of 21 questions and (...) lasted an average of 67 min. Interviews were transcribed, descriptively coded, and organized into code families.ResultsSSH editors saw anonymized peer review benefits to outweigh those of open peer review. They considered anonymized peer review the “gold standard” that authors and editors are expected to follow to respect institutional policies; moreover, anonymized review was also perceived as ethically superior due to the protection it provides, and more pragmatic due to eased seeking of reviewers. Finally, editors acknowledged their power in the publication process and reported strategies for keeping their work as unbiased as possible.ConclusionsEditors of SSH journals preferred the benefits of anonymized peer review over open peer and acknowledged the power they hold in the publication process during which authors are almost completely disclosed to editorial bodies. We recommend journals to communicate the transparency elements of their manuscript review processes by listing all bodies who contributed to the decision on every review stage. (shrink)

Peer review of journal submissions has become one of the most important pillars of quality management in academic publishing. Because of growing concerns with the quality and effectiveness of the system, a host of enthusiastic innovators has proposed and experimented with new procedures and technologies. However, little is known about whether these innovations manage to convince other journal editors. This paper will address open questions regarding the implementation of new review procedures, the occurrence rate of various (...) class='Hi'>peer review procedures and their distribution over scientific disciplines or academic publishers, as well as the motivations for editors or publishers to engage in novel review procedures. It shows that in spite of enthusiastic innovation, the adoption of new peer review procedures is in fact very slow, with the exception of text similarity scanners. For now, peer review innovations appear to be restricted to specific niches in academic publishing. Analysing these niches, the article concludes with a reflection on the circumstances in which innovations might be more widely implemented. (shrink)

While much literature has sprouted on peer review, this is the first book-length, wide-ranging study that utilizes methods and resources of contemporary philosophy. It covers the tension between peer review and the liberal notion that truth emerges when ideas proliferate in the marketplace of ideas; arguments for and against blind review of submissions; the alleged conservatism of peer review; the anomalous nature of book reviewing; the status of non-peer-reviewed publications; and the future (...) of peer review. (shrink)

BackgroundThe aim of this survey was to determine the level of awareness and understanding of peer review and peer review models amongst junior hospital doctors and whether this influences clinical decision-making.MethodsA 30-question online anonymous survey was developed aimed at determining awareness of peer review models and the purpose of peer review, perceived trustworthiness of different peer review models and the role of peer review in clinical decision-making. It was (...) sent to 800 trainee doctors in medical specialties on the University College London Partners trainee database.ResultsThe response rate was (178/800) 22%. Most respondents were specialist registrars. Checking that research is conducted correctly (152/178, 85%) and the data interpreted correctly (148/178, 83%) were viewed as the most important purposes of peer review. Most respondents were aware of open (133/178, 75%), double-blind (125/178, 70%) and single-blind peer review (121/178, 68%). 101/178 (57%) had heard of collaborative, 87/178 (49%) of post publication and 29/178 (16%) of decoupled peer review. Of those who were aware of double-blind, single-blind open and collaborative peer review, 85 (68%), 82 (68%), 74 (56%) and 24 (24%), respectively, understood how they worked. The NEJM, Lancet and The BMJ were deemed to have most trustworthy peer review, 137/178 (77%), 129/178 (72%) and 115/178 (65%), respectively. That peer review had taken place was important for a journal content to be used for clinical decision-making 152/178 (85%), but the ability to see peer review reports was not as important 22/178 (12%). Most felt there was a need for peer review training and that this should be at the specialist registrar stage of training.ConclusionsJunior hospital doctors view peer review to be important as a means of quality control, but do not value the ability to scrutinize peer review themselves. The unquestioning acceptance of peer review as final validation in the field of medicine emphasises not only the responsibility held by medical journals to ensure peer review is done well but also the need to raise awareness amongst the medical community of the limitations of the current peer review process. (shrink)

Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look at the empirical and methodological limitations of research on bias raises questions about the existence and extent of many hypothesized forms of bias. In addition, the notion of bias is predicated on an implicit ideal that, once articulated, raises questions about the (...) normative implications of research on bias in peer review. This review provides a brief description of the function, history, and scope of peer review; articulates and critiques the conception of bias unifying research on bias in peer review; characterizes and examines the empirical, methodological, and normative claims of bias in peer review research; and assesses possible alternatives to the status quo. We close by identifying ways to expand conceptions and studies of bias to countenance the complexity of social interactions among actors involved directly and indirectly in peer review. (shrink)

In a recent article in the British Journal for the Philosophy of Science, Heesen and Bright argue that prepublication peer review should be abolished and replaced with postpublication peer review (provided the matter is judged purely on epistemic grounds). In this article, I show that there are three problems with their argument. First, it fails to consider the epistemic cost of implementing the change to postpublication peer review. Second, it fails to consider some potential (...) epistemic benefits of prepublication peer review, which involve avoiding bias. Third, it fails to consider some potential epistemic disadvantages of postpublication peer review, which stem from the greater number of papers that would be published under that system. (shrink)

An error in the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, which wrongly predicted the disappearance of Himalayan glaciers by 2035, fueled doubts about the authority, honesty and rigor of the IPCC as a leading institution in climate science and, correspondingly, raised questions about whether global warming is anything more than a hoax put forward by environmentalists. The late and confusing reaction of the IPCC to these allegations only worsened the matter. By comparing assessment reports issued by (...) the IPCC, this paper asks the question: ‘Why, despite extensive peer review, did the Himalayan glacier melting rate error get published?’ I chronicle exactly what happened, consider why it happened as it did, and show how these answers require us to rethink the concept of ‘peer review’ in scientific practice. I also identify several future directions for peer review if it wants to stay ahead of the game, bearing in mind the forthcoming IPCC Assessment Reports. (shrink)

Background: Editorial peer review is universally used but little studied. We examined the relationship between external reviewers' recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine. Methodology/Principal Findings: We examined reviewer recommendations and editors' decisions at JGIM between 2004 and 2008. For manuscripts undergoing peer review, we calculated chance-corrected agreement among reviewers on recommendations to reject versus accept or revise. Using mixed effects logistic regression models, we (...) estimated intra-class correlation coefficients at the reviewer and manuscript level. Finally, we examined the probability of rejection in relation to reviewer agreement and disagreement. The 2264 manuscripts sent for external review during the study period received 5881 reviews provided by 2916 reviewers; 28% of reviews recommended rejection. Chance corrected agreement on rejection among reviewers was 0.11. In mixed effects models adjusting for study year and manuscript type, the reviewer-level ICC was 0.23 and the manuscript-level ICC was 0.17. The editors' overall rejection rate was 48%: 88% when all reviewers for a manuscript agreed on rejection and 20% when all reviewers agreed that the manuscript should not be rejected. Conclusions/Significance: Reviewers at JGIM agreed on recommendations to reject vs. accept/revise at levels barely beyond chance, yet editors placed considerable weight on reviewers' recommendations. Efforts are needed to improve the reliability of the peer-review process while helping editors understand the limitations of reviewers' recommendations. © 2010 Kravitz et al. (shrink)

BackgroundThe Patient-Centered Outcomes Research Institute is obligated to peer review and to post publicly “Final Research Reports” of all funded projects. PCORI peer review emphasizes adherence to PCORI’s Methodology Standards and principles of ethical scientific communication. During the peer review process, reviewers and editors seek to ensure that results are presented objectively and interpreted appropriately, e.g., free of spin.MethodsTwo independent raters assessed PCORI peer review feedback sent to authors. We calculated the proportion (...) of reports in which spin was identified during peer review, and the types of spin identified. We included reports submitted by April 2018 with at least one associated journal article. The same raters then assessed whether authors addressed reviewers’ comments about spin. The raters also assessed whether spin identified during PCORI peer review was present in related journal articles.ResultsWe included 64 PCORI-funded projects. Peer reviewers or editors identified spin in 55/64 submitted research reports. Types of spin included reporting bias, inappropriate interpretation, inappropriate extrapolation of results, and inappropriate attribution of causality. Authors addressed comments about spin related to 47/55 of the reports.Of 110 associated journal articles, PCORI comments about spin were potentially applicable to 44/110 articles, of which 27/44 contained the same spin that was identified in the PCORI research report. The proportion of articles with spin was similar for articles accepted before and after PCORI peer review.DiscussionJust as spin is common in journal articles and press releases, we found that most reports submitted to PCORI included spin. While most spin was mitigated during the funder’s peer review process, we found no evidence that review of PCORI reports influenced spin in journal articles. Funders could explore interventions aimed at reducing spin in published articles of studies they support. (shrink)

Recent science funding policy scholars and practitioners have advocated for the use of lotteries, or elements of random chance, as supplementations of traditional peer review for evaluating grant applications. One of the primary motivations for lotteries is their purported openness to innovative research. The purpose of this paper is to argue that current proponents of funding science by lottery overestimate the viability of peer review and thus unduly restrict the scope of lotteries in science funding practice. (...) I further show how this analysis suggests a different way of introducing lotteries into science funding policy. (shrink)

There is a paucity of studies addressing the nature of the social relations of production prevailing in academia prior to the commodification of academic research. By filling that gap, this paper enables us to better understand the historical presuppositions from which the process of knowledge commodification in academia has evolved. Our theoretically informed analysis will focus on peer review, given that it is one of the few academic practices where traces of that historical past can still be found. (...) On the basis of Marx’s exposition of the main features of pre-capitalist social relations of production in the Grundrisse, it will be concluded that peer review reveals that social relations of production in academia were of a pre-capitalist, or natural-like, nature: peer review is labour of a direct nature; is the practice through which the academic community mediates its own reproduction, determining how – and, sometimes, if – its members enter into contact with the objective conditions of production; finally, in this latter capacity, peer review presupposes that the academic community relates to the conditions of knowledge production as though they were their collective property. These findings are employed to account for the underlying logics of the ongoing ‘functional transformation’ of peer review. (shrink)

Peer review is a widely accepted instrument for raising the quality of science. Peer review limits the enormous unstructured influx of information and the sheer amount of dubious data, which in its absence would plunge science into chaos. In particular, peer review offers the benefit of eliminating papers that suffer from poor craftsmanship or methodological shortcomings, especially in the experimental sciences. However, we believe that peer review is not always appropriate for the (...) evaluation of controversial hypothetical science. We argue that the process of peer review can be prone to bias towards ideas that affirm the prior convictions of reviewers and against innovation and radical new ideas. Innovative hypotheses are thus highly vulnerable to being “filtered out” or made to accord with conventional wisdom by the peer review process. Consequently, having introduced peer review, the Elsevier journal Medical Hypotheses may be unable to continue its tradition as a radical journal allowing discussion of improbable or unconventional ideas. Hence we conclude by asking the publisher to consider re-introducing the system of editorial review to Medical Hypotheses. (shrink)

Peer review is an opportunity to perform an unlawful censorship which ensures that no apostate notion ever get published in mainstream journals. Or such peer review censorship is an opportunity to steal any content and to claim afterward the priority of first publication. And last but not least, the peer review is an academic tool to promote the mainstream pseudoscience.

Peer review is often taken to be the main form of quality control on academic research. Usually journals carry this out. However, parts of maths and physics appear to have a parallel, crowd-sourced model of peer review, where papers are posted on the arXiv to be publicly discussed. In this paper we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting papers by quality. Our argument (...) rests on two key claims. First, crowd-sourced peer review will lead on average to more reviewers per paper than journal-solicited peer review. Second, due to the wisdom of the crowds, more reviewers will tend to make better judgments than fewer. We make the second claim precise by looking at the Condorcet Jury Theorem as well as two related jury theorems developed specifically to apply to peer review. (shrink)

Peer review is a topic of considerable concern to many researchers, and there is a correspondingly large body of research on the topic. This issue of Science and Engineering Ethics presents recent work on peer review that is both grounded in empirical science and is applicable to policy decisions. This research raises two basic questions; (a) how does current peer review operate, and (b) how can it be improved? Topics addressed include descriptions of how (...) peer review is used in Federal agencies. whether peer review leads to better manuscripts, demographic characteristics of authors or reviewers (status or institutional affiliation), blinding of reviewers, authors, or results, reliability and consistency of reviews, accepting a paper before the study is done, simultaneous submission, and use of dispute resolution procedures such as scientific dialectical and pleading protocols. (shrink)

Supporting young researchers to peer-review and edit journal submissions will also accelerate their training, says Quan-Hoang Vuong. -/- *Citation: -/- Quan-Hoang Vuong. (2022, Sept. 11). Early career researchers can help fix peer review delays. Times Higher Education.

The paper offers a critical review of Roberto Farneti’s paper a minor philosophy. The state of the art of philosophical scholarship in Italy , recently published in Philosophia. It is argued that overall the status and interest of philosophy as practiced nowadays in Italy is less disappointing than Farneti makes out. It is also maintained that submitting papers to peer-refereed international journals can help cure the moral and sociological disease that besets the Italian academia, but that, as such, (...) it is less likely to improve the scientific quality of contributions in philosophy than Farneti claims. In passing, a few recommendations both to the philosophical community at large and to the Italian Government are put forward. (shrink)

An important function in all scholarly and academic activities is the participation in the peer review system. One aspect of this peer review evaluation is service on committees judging candidates for important awards, prizes and fellowships. Some reflective observations on this process are made in which a number of factors determining the final choice are identified. It is pointed out that the decisions of such committees are based not only on relevant and objective criteria but are (...) also influenced by a number of irrelevant criteria; caution must be exercised to minimize the effect of the latter factors in order to maintain the highest ethical standards in the selection process. The ranking practices of awards committees, national academies and learned societies are briefly reviewed by pointing out some ethical pitfalls, anecdotal incidents, and the ways to avoid the tarnishing of the selection process. (shrink)

Blind review is ubiquitous in contemporary science, but there is no consensus among stakeholders and researchers about when or how much or why blind review should be done. In this essay, we explain why blinding enhances the impartiality and credibility of science while also defending a norm according to which blind review is a baseline presumption in scientific peer review.

Mission-oriented climate change research is often unverifiable. Therefore, many stakeholders look to peer-reviewed climate change research for trustworthy information about deeply uncertain and impactful phenomena. This is because peer-review signals that research has been vetted for scientific standards like reproducibility and replicability. Here we evaluate the transparency of research methodologies in mission-oriented computational climate research. We find that only five percent of our sample meets the minimal standard of fully open data and code required for reproducibility and (...) replicability. The widespread lack of minimal openness standards in peer-reviewed climate change research jeopardizes scientific credibility and risks poor societal outcomes as we manage growing climate risks. The scientific community can and must do better. We, the authors, outline necessary openness standards that we commit to as researchers, reviewers, and readers. We invite you to join us. (shrink)

Retractions of COVID-19 literature in both preprints and the peer-reviewed literature serve as a reminder that there are still challenging issues underlying the integrity of the biomedical literature. The risks to academia become larger when such retractions take place in high-ranking biomedical journals. In some cases, retractions result from unreliable or nonexistent data, an issue that could easily be avoided by having open data policies, but there have also been retractions due to oversight in peer review and (...) editorial verification. As COVID-19 continues to affect academics and societies around the world, failures in peer review might also constitute a public health risk. The effectiveness by which COVID-19 literature is corrected, including through retractions, depends on the stringency of measures in place to detect errors and to correct erroneous literature. It also relies on the stringent implementation of open data policies. (shrink)

This article examines the evolution of peer review and the modern editorial processes of scholarly journals by analyzing a novel data set derived from the Royal Society’s archives and covering 1865-1965, that is, the historical period in which refereeing became firmly established. Our analysis reveals how the Royal Society’s editorial processes coped with both an increasing reliance on refereeing and a growth in submissions, while maintaining collective responsibility and minimizing research waste. By engaging more of its fellows in (...) editorial activity, the society was able to establish an equilibrium of number of submissions per reviewer that was relatively stable over time. Nevertheless, our analysis shows that the distribution of editorial work was significantly uneven. Our findings reveal interesting parallels with current concerns about the scale and distribution of peer review work and suggest the strategic importance of the management of the editorial process to achieve a creative mix of community commitment and professional responsibility that is essential in contemporary journals. (shrink)

IntroductionAllocation of research funds relies on peer review to support funding decisions, and these processes can be susceptible to biases and inefficiencies. The aim of this work was to determine which past interventions to peer review and decision-making have worked to improve research funding practices, how they worked, and for whom.MethodsRealist synthesis of peer-review publications and grey literature reporting interventions in peer review for research funding.ResultsWe analysed 96 publications and 36 website sources. (...) Sixty publications enabled us to extract stakeholder-specific context-mechanism-outcomes configurations for 50 interventions, which formed the basis of our synthesis. Shorter applications, reviewer and applicant training, virtual funding panels, enhanced decision models, institutional submission quotas, applicant training in peer review and grant-writing reduced interrater variability, increased relevance of funded research, reduced time taken to write and review applications, promoted increased investment into innovation, and lowered cost of panels.ConclusionsReports of 50 interventions in different areas of peer review provide useful guidance on ways of solving common issues with the peer review process. Evidence of the broader impact of these interventions on the research ecosystem is still needed, and future research should aim to identify processes that consistently work to improve peer review across funders and research contexts. (shrink)

Anyone who labors at academic scholarship knows vividly—perhaps even painfully—how dependent that enterprise is on a system of peer review. A scholar submits a work to a journal, press, or conference committee, or sends a proposal to a foundation; the submission is then evaluated by other professionals. The judgment of these referees determines whether the work is published by the target journal or press, appears on the conference program, or is funded by the desired institution. In many fields (...) the overwhelming majority of submissions are rejected. Careers are thus often made or destroyed by the process. (shrink)

Open peer review (OPR) is moving into the mainstream, but it is often poorly understood and surveys of researcher attitudes show important barriers to implementation. As more journals move to implement and experiment with the myriad of innovations covered by this term, there is a clear need for best practice guidelines to guide implementation. This brief article aims to address this knowledge gap, reporting work based on an interactive stakeholder workshop to create best-practice guidelines for editors and journals (...) who wish to transition to OPR. Although the advice is aimed mainly at editors and publishers of scientific journals, since this is the area in which OPR is at its most mature, many of the principles may also be applicable for the implementation of OPR in other areas (e.g., books, conference submissions). (shrink)

To arrive at their final evaluation of a manuscript or grant proposal, reviewers must convert a submission’s strengths and weaknesses for heterogeneous peer review criteria into a single metric of quality or merit. I identify this process of commensuration as the locus for a new kind of peer review bias. Commensuration bias illuminates how the systematic prioritization of some peer review criteria over others permits and facilitates problematic patterns of publication and funding in science. (...) Commensuration bias also foregrounds a range of structural strategies for realigning peer review practices and institutions with the aims of science. (shrink)

Peer review is widely recognized as a mechanism for quality control of academic content. This research article aims at comparing the review reports and decisions of reviewers who are members of the editorial board of the European Scientific Journal with those reviewers suggested by the authors and who are not affiliated with the journal. 457 review reports on 378 papers submitted to the ESJ in the period of October–December 2017 were analysed. Statistical methods including OLS and (...) Wilcoxon rank-sum test were applied based on the score approach toward the reviewers’ assessments of the papers and their characteristics related to the country, gender, and time of revisions. Results show the difference between the decisions these two groups of reviewers made. Even though editor-suggested and author-suggested reviewers need equal time to review a paper, the former are less favourable towards the authors of the papers. It is also concluded that factors such as time and country of the reviewers influence their decisions. In this regard, the editors should avoid relying their decisions solely on review reports received from reviewers suggested by the authors. However, further research with larger sample sizes should be conducted. (shrink)

A relatively high incidence of unsatisfactory review decisions is widely recognised and acknowledged as ‘the peer review problem’. Factors contributing to this problem are identified and examined. Specific examples of unreasonable rejection are considered. It is concluded that weaknesses of the ‘peer review’ system are significant and that they are well known or readily recognisable but that necessary counter-measures are not always enforced. Careful management is necessary to discount hollow opinion or error in review (...) comment. Review and referee functions should be quite separate. (shrink)