In 2014 Jennifer Cramblett, a white lesbian, filed a Complaint for Wrongful Birth alleging that the Midwest Sperm Bank mistakenly provided sperm from an African–American donor. In this article, we trace the complex and overlapping lines of legal and social inheritance that have conditioned not only the possibility of such a lawsuit, but also the legal language and arguments within the Complaint itself. First, we trace the racial politics of homonormativity, which set the conditions of possibility for an out, white (...) lesbian to bring this case forward. Second, we trace the inheritance of wrongful birth tort law, reviewing its prior race and disability-related uses, and its basis in feminist reproductive rights. Third, we trace how disability, race and sexuality interlock within the eugenic inheritance of both ‘wrongful birth’ and reproductive technologies. Finally, we follow traces of racial inheritance, namely, the loss of white property and proximity to whiteness. (shrink)

Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available evidence, they are (...) either positive or neutral. We conclude that on present evidence abolishing peer review weakly dominates the status quo. (shrink)

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we (...) selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated. (shrink)

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.

This paper reviews the published empirical evidence concerning journal peer review consisting of 68 papers, all but three published since 1975. Peer review improves quality, but its use to screen papers has met with limited success. Current procedures to assure quality and fairness seem to discourage scientific advancement, especially important innovations, because findings that conflict with current beliefs are often judged to have defects. Editors can use procedures to encourage the publication of papers with innovative findings (...) such as invited papers, early-acceptance procedures, author nominations of reviewers, structured rating sheets, open peer review, results-blind review, and, in particular, electronic publication. Some journals are currently using these procedures. The basic principle behind the proposals is to change the decision from whether to publish a paper to how to publish it. (shrink)

Peer review and publication is one of the factors proposed in Daubert v. Merrell Dow Pharmaceuticals, Inc. as indicia of the reliability of scientific testimony. This Article traces the origins of the peer-review system, the process by which it became standard at scientific and medical journals, and the many roles it now plays. Additionally, the Author articulates the epistemological rationale for pre-publication peer-review and the inherent limitations of the system as a scientific quality-control mechanism. (...) The Article explores recent changes in science, in scientific publishing, and in the academy that have put the system under strain. The Author argues that Justice Blackmun's advice to courts - that peer-reviewed publication is relevant, but is not dispositive - is of little practical help. Instead, the Author suggests questions that courts should ask in assessing the significance of the fact that testimony is, or is not, based on peer-reviewed publication and illustrates with reference to another Bendectin case, Blum v. Merrell Dow Pharmaceuticals, Inc., where some of these questions were asked. (shrink)

Pre-publication certification through peer review stands in need of philosophical examination. In this paper, philosopher-psychologist Steven James Bartlett recalls the arguments marshalled four hundred years ago by English poet John Milton against restraint of publication by the "gatekeepers of publication," AKA today's peer reviewers.

Two important aspects of the relationship between peer review and innovation includes the acceptance of articles for publication in journals and the assessment of applications for grants for the funding of research work. While there are well-known examples of the rejection by journals of first choice of many papers that have radically changed the way we think about the world outside ourselves, such papers do get published eventually, however tortuous the process required. With grant applications the situation differs (...) in that the refusal of a grant necessarily curtails the possible research that may be attempted. Here there are many reasons for conservatism and reservation as to the ability of a grant allocation process based on peer review to deliver truly innovative investigations. Other methods are needed; although such methods need not be applied across the board, they should constitute the methods whereby some 10–20% of the grant monies are assigned. The nomination of prizes for specific accomplishments is one way of achieving innovation although this presumes that investigators or institution already have available the money necessary to effect the innovations; otherwise it is a question of the selection and funding of particular individuals or institutions and requiring them to solve particular problems that are set in the broadest of terms. (shrink)

Peer review is a widely accepted instrument for raising the quality of science. Peer review limits the enormous unstructured influx of information and the sheer amount of dubious data, which in its absence would plunge science into chaos. In particular, peer review offers the benefit of eliminating papers that suffer from poor craftsmanship or methodological shortcomings, especially in the experimental sciences. However, we believe that peer review is not always appropriate for the (...) evaluation of controversial hypothetical science. We argue that the process of peer review can be prone to bias towards ideas that affirm the prior convictions of reviewers and against innovation and radical new ideas. Innovative hypotheses are thus highly vulnerable to being “filtered out” or made to accord with conventional wisdom by the peer review process. Consequently, having introduced peer review, the Elsevier journal Medical Hypotheses may be unable to continue its tradition as a radical journal allowing discussion of improbable or unconventional ideas. Hence we conclude by asking the publisher to consider re-introducing the system of editorial review to Medical Hypotheses. (shrink)

Peer review has long been regarded as the gold standard of scientific publication, essential to the integrity of science itself. But, as any publishing scientist knows, peer review has its downside, including long delays and reviewer bias. Until the coming of the Internet, there appeared to be no alternative. Now, articles appear online as preprints almost immediately upon submission. But they lack peer review and thus their scientific standing can be questioned. Post-publication discussion platforms (...) such as PubPeer have proven useful, but are no substitute for pre-publication peer review. Nevertheless, some may be tempted to believe that peer review can now be done without. This article challenges that view by analyzing a recent, non-peer-reviewed article in Skeptical Enquirer, a magazine published by the Committee for Skeptical Enquiry (CSI). The article, “Sodom Meteor Strike Claims Should Be Taken With a Pillar of Salt,” casts doubt on one of the most widely read scientific articles of the last decade and provides a stern warning of the cost of abandoning peer review. (shrink)

At present, peer review is the most common method used by funding agencies to make decisions about resource allocation. But how reliable, efficient, and fair is it in practice? The ex ante evaluation of scientific novelty is a fundamentally uncertain endeavor; bias and chance are embedded in the final outcome. In the current study, I will examine some of the most central problems of peer review and highlight the possible benefits of using a lottery as an (...) alternative decision-making mechanism. Lotteries are driven by chance, not reason. The argument made in the study is that the epistemic landscape could benefit in several respects by using a lottery, thus avoiding all types of bias, disagreement, and other limitations associated with the peer review process. Funding agencies could form a pool of funding applicants who have minimal qualification levels and then select randomly within that pool. The benefits of a lottery would not only be that it saves time and resources, but also that it contributes to a more dynamic selection process and increases the epistemic diversity, fairness, and impartiality within academia. (shrink)

In a recent article in the British Journal for the Philosophy of Science, Heesen and Bright argue that prepublication peer review should be abolished and replaced with postpublication peer review (provided the matter is judged purely on epistemic grounds). In this article, I show that there are three problems with their argument. First, it fails to consider the epistemic cost of implementing the change to postpublication peer review. Second, it fails to consider some potential (...) epistemic benefits of prepublication peer review, which involve avoiding bias. Third, it fails to consider some potential epistemic disadvantages of postpublication peer review, which stem from the greater number of papers that would be published under that system. (shrink)

Peer review process helps in evaluating and validating of research that is published in the journals. U.S. Office of Research Integrity reported that data fraudulence was found to be involved in 94% cases of misconduct from 228 identified articles between 1994–2012. If fraud in published article are significantly as high as reported, the question arise in mind, were these articles peer reviewed? Another report said that the reviewers failed to detect 16 cases of fabricated article of Jan (...) Hendrick Schon. Superficial peer reviewing process does not reveals suspicion of misconduct. Lack of knowledge of systemic review process not only demolish the academic integrity in publication but also loss the trust of the people of the institution, the nation, and the world. The aim of this review article is to aware stakeholders specially novice reviewers about the peer review system. Beginners will understand how to review an article and they can justify better action choices in dealing with reviewing an article. (shrink)

While much literature has sprouted on peer review, this is the first book-length, wide-ranging study that utilizes methods and resources of contemporary philosophy. It covers the tension between peer review and the liberal notion that truth emerges when ideas proliferate in the marketplace of ideas; arguments for and against blind review of submissions; the alleged conservatism of peer review; the anomalous nature of book reviewing; the status of non-peer-reviewed publications; and the future (...) of peer review. (shrink)

Mission-oriented climate change research is often unverifiable. Therefore, many stakeholders look to peer-reviewed climate change research for trustworthy information about deeply uncertain and impactful phenomena. This is because peer-review signals that research has been vetted for scientific standards like reproducibility and replicability. Here we evaluate the transparency of research methodologies in mission-oriented computational climate research. We find that only five percent of our sample meets the minimal standard of fully open data and code required for reproducibility and (...) replicability. The widespread lack of minimal openness standards in peer-reviewed climate change research jeopardizes scientific credibility and risks poor societal outcomes as we manage growing climate risks. The scientific community can and must do better. We, the authors, outline necessary openness standards that we commit to as researchers, reviewers, and readers. We invite you to join us. (shrink)

Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look at the empirical and methodological limitations of research on bias raises questions about the existence and extent of many hypothesized forms of bias. In addition, the notion of bias is predicated on an implicit ideal that, once articulated, raises questions about the (...) normative implications of research on bias in peer review. This review provides a brief description of the function, history, and scope of peer review; articulates and critiques the conception of bias unifying research on bias in peer review; characterizes and examines the empirical, methodological, and normative claims of bias in peer review research; and assesses possible alternatives to the status quo. We close by identifying ways to expand conceptions and studies of bias to countenance the complexity of social interactions among actors involved directly and indirectly in peer review. (shrink)

Background: Editorial peer review is universally used but little studied. We examined the relationship between external reviewers' recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine. Methodology/Principal Findings: We examined reviewer recommendations and editors' decisions at JGIM between 2004 and 2008. For manuscripts undergoing peer review, we calculated chance-corrected agreement among reviewers on recommendations to reject versus accept or revise. Using mixed effects logistic regression models, we (...) estimated intra-class correlation coefficients at the reviewer and manuscript level. Finally, we examined the probability of rejection in relation to reviewer agreement and disagreement. The 2264 manuscripts sent for external review during the study period received 5881 reviews provided by 2916 reviewers; 28% of reviews recommended rejection. Chance corrected agreement on rejection among reviewers was 0.11. In mixed effects models adjusting for study year and manuscript type, the reviewer-level ICC was 0.23 and the manuscript-level ICC was 0.17. The editors' overall rejection rate was 48%: 88% when all reviewers for a manuscript agreed on rejection and 20% when all reviewers agreed that the manuscript should not be rejected. Conclusions/Significance: Reviewers at JGIM agreed on recommendations to reject vs. accept/revise at levels barely beyond chance, yet editors placed considerable weight on reviewers' recommendations. Efforts are needed to improve the reliability of the peer-review process while helping editors understand the limitations of reviewers' recommendations. © 2010 Kravitz et al. (shrink)

A relatively high incidence of unsatisfactory review decisions is widely recognised and acknowledged as ‘the peer review problem’. Factors contributing to this problem are identified and examined. Specific examples of unreasonable rejection are considered. It is concluded that weaknesses of the ‘peer review’ system are significant and that they are well known or readily recognisable but that necessary counter-measures are not always enforced. Careful management is necessary to discount hollow opinion or error in review (...) comment. Review and referee functions should be quite separate. (shrink)

That there is a “crisis of peer review” at the moment is not in dispute, but sufficient attention has not yet been paid to the normative potential that lies in current calls for reform. In contrast to approaches to “fixing” the problems in peer review, which tend to maintain the status quo in terms of professionalising opportunities, this paper addresses the needs of philosophers and how peer‐review reform can be an opportunity to improve the (...) academic discipline of philosophy, whereby progress is understood as making the discipline more fair to the global academic community and more conducive to the flourishing of academic philosophers. The paper evaluates recent categories of relevant norms and correlating reforms. In conclusion, it recommends that philosophy pursue the norms of transparency and democracy explicitly when proposing peer‐review reform and suggest that proposals for forum‐based models of peer review are most likely to support those norms. (shrink)

Recent science funding policy scholars and practitioners have advocated for the use of lotteries, or elements of random chance, as supplementations of traditional peer review for evaluating grant applications. One of the primary motivations for lotteries is their purported openness to innovative research. The purpose of this paper is to argue that current proponents of funding science by lottery overestimate the viability of peer review and thus unduly restrict the scope of lotteries in science funding practice. (...) I further show how this analysis suggests a different way of introducing lotteries into science funding policy. (shrink)

Retractions of COVID-19 literature in both preprints and the peer-reviewed literature serve as a reminder that there are still challenging issues underlying the integrity of the biomedical literature. The risks to academia become larger when such retractions take place in high-ranking biomedical journals. In some cases, retractions result from unreliable or nonexistent data, an issue that could easily be avoided by having open data policies, but there have also been retractions due to oversight in peer review and (...) editorial verification. As COVID-19 continues to affect academics and societies around the world, failures in peer review might also constitute a public health risk. The effectiveness by which COVID-19 literature is corrected, including through retractions, depends on the stringency of measures in place to detect errors and to correct erroneous literature. It also relies on the stringent implementation of open data policies. (shrink)

Previous research has found that funding disparities are driven by applications’ final impact scores and that only a portion of the black/white funding gap can be explained by bibliometrics and topic choice. Using National Institutes of Health R01 applications for council years 2014–2016, we examine assigned reviewers’ preliminary overall impact and criterion scores to evaluate whether racial disparities in impact scores can be explained by application and applicant characteristics. We hypothesize that differences in commensuration—the process of combining criterion scores into (...) overall impact scores—disadvantage black applicants. Using multilevel models and matching on key variables including career stage, gender, and area of science, we find little evidence for racial disparities emerging in the process of combining preliminary criterion scores into preliminary overall impact scores. Instead, preliminary criterion scores fully account for racial disparities—yet do not explain all of the variability—in preliminary overall impact scores. (shrink)

This Open Access book about the Zadeh Project demonstrates and explores a core question in clinical ethics: how can ethics consultants be accountable in the face of a robust plurality of ethical standpoints, especially those that underwrite practices and methods for doing ethics consultation as well as those viewpoints and values encountered in daily clinical ethics practice? Underscoring this question is the recognition that the field of clinical ethics consultation has arrived at a crucial point in its maturation. Many efforts (...) are underway to more formally “professionalize” the field, with most aimed toward stabilizing a specific set of institutional considerations. Stretched between these institutional and practical initiatives resides a crucial set of of ethical considerations, chief among them the meaning and scope of responsibility for clinical ethics consultants. Developed around a long-form case scenario, the Zadeh Project provides a multi-layered series of “peer-reviews”: critique of the actions of the case scenario’s ethics consultant; reflection on clinical ethics method; examination of the many ways that commitments to method and practice can, and do, intersect, overlap, and alter one another. The design and format of this book thus models a key element for clinical ethics practice: the need and ability to provide careful and thoughtful explanation of core moral considerations that emerge among diverse standpoints. Specifically designed for those studying to become and those who are ethics consultants, this book, with its innovative and multi-layered approach, allows readers to share a peer-review-like experience that shows accountability to be what it is, an ethical, not merely procedural or administrative, undertaking. (shrink)

Peer review is an opportunity to perform an unlawful censorship which ensures that no apostate notion ever get published in mainstream journals. Or such peer review censorship is an opportunity to steal any content and to claim afterward the priority of first publication. And last but not least, the peer review is an academic tool to promote the mainstream pseudoscience.

An error in the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, which wrongly predicted the disappearance of Himalayan glaciers by 2035, fueled doubts about the authority, honesty and rigor of the IPCC as a leading institution in climate science and, correspondingly, raised questions about whether global warming is anything more than a hoax put forward by environmentalists. The late and confusing reaction of the IPCC to these allegations only worsened the matter. By comparing assessment reports issued by (...) the IPCC, this paper asks the question: ‘Why, despite extensive peer review, did the Himalayan glacier melting rate error get published?’ I chronicle exactly what happened, consider why it happened as it did, and show how these answers require us to rethink the concept of ‘peer review’ in scientific practice. I also identify several future directions for peer review if it wants to stay ahead of the game, bearing in mind the forthcoming IPCC Assessment Reports. (shrink)

A relatively high incidence of unsatisfactory review decisions is widely recognised and acknowledged as ‘the peer review problem’. Factors contributing to this problem are identified and examined. Specific examples of unreasonable rejection are considered. It is concluded that weaknesses of the ‘peer review’ system are significant and that they are well known or readily recognisable but that necessary counter-measures are not always enforced. Careful management is necessary to discount hollow opinion or error in review (...) comment. Review and referee functions should be quite separate. (shrink)

Peer review of journal submissions has become one of the most important pillars of quality management in academic publishing. Because of growing concerns with the quality and effectiveness of the system, a host of enthusiastic innovators has proposed and experimented with new procedures and technologies. However, little is known about whether these innovations manage to convince other journal editors. This paper will address open questions regarding the implementation of new review procedures, the occurrence rate of various (...) class='Hi'>peer review procedures and their distribution over scientific disciplines or academic publishers, as well as the motivations for editors or publishers to engage in novel review procedures. It shows that in spite of enthusiastic innovation, the adoption of new peer review procedures is in fact very slow, with the exception of text similarity scanners. For now, peer review innovations appear to be restricted to specific niches in academic publishing. Analysing these niches, the article concludes with a reflection on the circumstances in which innovations might be more widely implemented. (shrink)

As philosophers we should have as one of our aims to produce as much philosophical knowledge as possible. A lot of potential philosophical knowledge is lost because of the flaws of the peer review system, and so a lot of philosophical knowledge would be gained were the system improved. Accordingly, as authors we should write papers about how to fix peer review, and as editors we should accept such papers if they are good. This paper presents (...) some familiar problems with peer review, elaborates on and motivates the argument just given, and replies to some objections to it, making the case that fixing peer review is both a philosophical problem and one that admits of a solution. (shrink)

Peer review is often taken to be the main form of quality control on academic research. Usually journals carry this out. However, parts of maths and physics appear to have a parallel, crowd-sourced model of peer review, where papers are posted on the arXiv to be publicly discussed. In this paper we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting papers by quality. Our argument (...) rests on two key claims. First, crowd-sourced peer review will lead on average to more reviewers per paper than journal-solicited peer review. Second, due to the wisdom of the crowds, more reviewers will tend to make better judgments than fewer. We make the second claim precise by looking at the Condorcet Jury Theorem as well as two related jury theorems developed specifically to apply to peer review. (shrink)

Anyone who labors at academic scholarship knows vividly—perhaps even painfully—how dependent that enterprise is on a system of peer review. A scholar submits a work to a journal, press, or conference committee, or sends a proposal to a foundation; the submission is then evaluated by other professionals. The judgment of these referees determines whether the work is published by the target journal or press, appears on the conference program, or is funded by the desired institution. In many fields (...) the overwhelming majority of submissions are rejected. Careers are thus often made or destroyed by the process. (shrink)

To arrive at their final evaluation of a manuscript or grant proposal, reviewers must convert a submission’s strengths and weaknesses for heterogeneous peer review criteria into a single metric of quality or merit. I identify this process of commensuration as the locus for a new kind of peer review bias. Commensuration bias illuminates how the systematic prioritization of some peer review criteria over others permits and facilitates problematic patterns of publication and funding in science. (...) Commensuration bias also foregrounds a range of structural strategies for realigning peer review practices and institutions with the aims of science. (shrink)

BackgroundThere is increasing need for peer reviewers as the scientific literature grows. Formal education in biostatistics and research methodology during residency training is lacking. In this pilot study, we addressed these issues by evaluating a novel method of teaching residents about biostatistics and research methodology using peer review of standardized manuscripts. We hypothesized that mentored peer review would improve resident knowledge and perception of these concepts more than non-mentored peer review, while improving (...) class='Hi'>review quality.MethodsA partially blinded, randomized, controlled multi-center study was performed. Seventy-eight neurology residents from nine US neurology programs were randomized to receive mentoring from a local faculty member or not. Within a year, residents reviewed a baseline manuscript and four subsequent manuscripts, all with introduced errors designed to teach fundamental review concepts. In the mentored group, mentors discussed completed reviews with residents. Primary outcome measure was change in knowledge score between pre- and post-tests, measuring epidemiology and biostatistics knowledge. Secondary outcome measures included level of confidence in the use and interpretation of statistical concepts before and after intervention, and RQI score for baseline and final manuscripts.ResultsSixty-four residents (82%) completed initial review with gradual decline in completion on subsequent reviews. Change in primary outcome, the difference between pre- and post-test knowledge scores, did not differ between mentored (−8.5%) and non-mentored (−13.9%) residents (p = 0.48). Significant differences in secondary outcomes (using 5-point Likert scale, 5 = strongly agree) included mentored residents reporting enhanced understanding of research methodology (3.69 vs 2.61; p = 0.001), understanding of manuscripts (3.73 vs 2.87; p = 0.006), and application of study results to clinical practice (3.65 vs 2.78; p = 0.005) compared to non-mentored residents. There was no difference between groups in level of interest in peer review (3.00 vs 3.09; p = 0.72) or the quality of manuscript review assessed by the Review Quality Instrument (RQI) (3.25 vs 3.06; p = 0.50).ConclusionsWe used mentored peer review of standardized manuscripts to teach biostatistics and research methodology and introduce the peer review process to residents. Though knowledge level did not change, mentored residents had enhanced perception in their abilities to understand research methodology and manuscripts and apply study results to clinical practice. (shrink)

We examine the criticisms and subsequent changes that arise in the course of peer review. Fifty-two scholars who had recently published in Administrative Science Quarterly were surveyed regarding their peer review experience and how their article changed from initial journal submission to eventual publication. Papers that challenged theoretical perspectives faced distinctively high levels of criticism and change, particularly with attention to methodology, while those that offered a new perspective or that extended or combined established perspectives were (...) less criticized and changed. The number of challenge-oriented publications was small as well, suggesting that either few such submissions survive the review process or few are submitted in the first place. Overall, peer review appears open to expansion of the variety of theoretical argument but does little to aid in the winnowing out of established perspectives. (shrink)

Peer review is a practice used in the evaluation of scientific and scholarly papers in order to select papers for publication in scholarly journals. The practice has also been extended to other domains, such as the evaluation of grant proposals, medical practice, book publication, and even to such areas as teaching evaluation. The primary area that has been of interest to sociologists, however, has been publication in scientific journals. The practice is usually understood to have begun in the (...) seventeenth century in the Royal Society in London, but it has also been claimed that there have been precursors to this practice. In the popular mind, peer review in science is a means of “bullet‐proofing” research, as a Wall Street Journal article once put it, that is to say as a guarantee of quality. But within science it has a very different meaning. (shrink)

As universities in the United Kingdom gear themselves up for the next Research Excellence Framework, REF2021, with peer review at its core, we critically re-visit the idea of peer review as a gold standard proxy for research excellence. We question the premise that anonymous peer review is a necessary and enabling condition for impartial, expert judgement. We argue that the intentions and supposed benefits underlying peer review and its associated concepts have become (...) congealed in received discourse about research quality. Hence we explore the key conceptual issues raised by the nested assumptions and concepts that come into play in peer review as currently practised: primarily those of secrecy, anonymity, legitimacy, trust, impartiality and openness. After delineating the benefits attributed to peer review, we contrast its declared virtues with its problematic features. We locate peer review in an audit culture in which the reviewer is an academic labourer. Drawing on recent trends in moral and political philosophy, we question the usefulness of the ideal of impartiality when tied to secrecy. Then we raise more deliberative, intersubjective possibilities for a revised understanding of peer review in the context of an academic community. Finally, we suggest ways in which the academic community could pursue quality in research by recasting peer review to be less secret and more open. (shrink)

This chapter critically reflects on the critiques, reviews, and many proposals presented in Parts Two, Three, and Four, and provides a summary conclusion for the entire Zadeh Project. Obvious differences between experience and reporting on experience are highlighted, with particular attention to the ways such differences are detailed by the Zadeh Scenario and in our colleagues’ responses to it. In addition, we discuss a key challenge associated with clinical ethics practice and the peer review of such practice: identifying (...) what actually matters for those engaged in these kinds of activities. In each case, we argue, the work of discovering what matters to the individuals actually involved cannot be overlooked. Furthermore, that work may even be the primary ingredient to responsible conduct for both consultation and peer review. Building on this idea that responding responsibly to that which is actually encountered in clinical practice serves as a primary activity of, and thus grounds responsibility in, clinical ethics practice, we then argue that ethical analysis of Finder’s activities in the Zadeh Scenario, and for clinical ethics peer review more generally, has at least three layers: that which addresses the immediate interactions between an ethics consultant and those whom the consultant engages; that which addresses the ethics consultant’s own sense of self-responsibility; that which addresses peer reviewers’ own frames for evaluating another’s clinical ethics practice. We conclude that the Zadeh Project exemplifies what is at stake in ethics consultation and peer review. Taken as a whole, then, the Zadeh Project reveals how peer review, in acknowledgement of the responsibility to create a context for peer education, rests upon a commitment to model – and demand critical engagement with – what is held to be most worthwhile for clinical ethics practice and the field of clinical ethics consultation as a whole. (shrink)

The main goal of publicly funded biomedical research is to generate social value through the creation and application of knowledge that can improve the well-being of current and future people. Prioritising research with the greatest potential social value is crucial for good stewardship of limited public resources and ensuring ethical involvement of research participants. At the National Institutes of Health (NIH), peer reviewers hold the expertise and responsibility for social value assessment and resulting prioritisation at the project level. However, (...) previous research has shown that peer reviewers place more emphasis on a study’s methods (‘Approach’) than on its potential social value (best approximated by the criterion of ‘Significance’). Lower weighting of Significance may be due to reviewers’ views on the relative importance of social value, their belief that social value is evaluated at other stages of the research priority-setting process or the lack of guidance on how to approach the challenging task of assessing expected social value. The NIH is currently revising its review criteria and how these criteria contribute to overall scores. To elevate the role of social value in priority setting, the agency should support empirical research on how peer reviewers approach the assessment of social value, provide more specific guidance for reviewing social value and experiment with alternative reviewer assignment strategies. These recommendations would help ensure that funding priorities align with the NIH’s mission and the obligation of taxpayer-funded research to contribute to the public good. (shrink)

BackgroundThe aim of this survey was to determine the level of awareness and understanding of peer review and peer review models amongst junior hospital doctors and whether this influences clinical decision-making.MethodsA 30-question online anonymous survey was developed aimed at determining awareness of peer review models and the purpose of peer review, perceived trustworthiness of different peer review models and the role of peer review in clinical decision-making. It was (...) sent to 800 trainee doctors in medical specialties on the University College London Partners trainee database.ResultsThe response rate was (178/800) 22%. Most respondents were specialist registrars. Checking that research is conducted correctly (152/178, 85%) and the data interpreted correctly (148/178, 83%) were viewed as the most important purposes of peer review. Most respondents were aware of open (133/178, 75%), double-blind (125/178, 70%) and single-blind peer review (121/178, 68%). 101/178 (57%) had heard of collaborative, 87/178 (49%) of post publication and 29/178 (16%) of decoupled peer review. Of those who were aware of double-blind, single-blind open and collaborative peer review, 85 (68%), 82 (68%), 74 (56%) and 24 (24%), respectively, understood how they worked. The NEJM, Lancet and The BMJ were deemed to have most trustworthy peer review, 137/178 (77%), 129/178 (72%) and 115/178 (65%), respectively. That peer review had taken place was important for a journal content to be used for clinical decision-making 152/178 (85%), but the ability to see peer review reports was not as important 22/178 (12%). Most felt there was a need for peer review training and that this should be at the specialist registrar stage of training.ConclusionsJunior hospital doctors view peer review to be important as a means of quality control, but do not value the ability to scrutinize peer review themselves. The unquestioning acceptance of peer review as final validation in the field of medicine emphasises not only the responsibility held by medical journals to ensure peer review is done well but also the need to raise awareness amongst the medical community of the limitations of the current peer review process. (shrink)

Open Peer Review: Educational Philosophy and Theory (EPAT)Michael A. Peters, Beijing Normal University, PR ChinaIn 2016 EPAT started experimenting with open peer review for articles that were part...

BackgroundThe Patient-Centered Outcomes Research Institute is obligated to peer review and to post publicly “Final Research Reports” of all funded projects. PCORI peer review emphasizes adherence to PCORI’s Methodology Standards and principles of ethical scientific communication. During the peer review process, reviewers and editors seek to ensure that results are presented objectively and interpreted appropriately, e.g., free of spin.MethodsTwo independent raters assessed PCORI peer review feedback sent to authors. We calculated the proportion (...) of reports in which spin was identified during peer review, and the types of spin identified. We included reports submitted by April 2018 with at least one associated journal article. The same raters then assessed whether authors addressed reviewers’ comments about spin. The raters also assessed whether spin identified during PCORI peer review was present in related journal articles.ResultsWe included 64 PCORI-funded projects. Peer reviewers or editors identified spin in 55/64 submitted research reports. Types of spin included reporting bias, inappropriate interpretation, inappropriate extrapolation of results, and inappropriate attribution of causality. Authors addressed comments about spin related to 47/55 of the reports.Of 110 associated journal articles, PCORI comments about spin were potentially applicable to 44/110 articles, of which 27/44 contained the same spin that was identified in the PCORI research report. The proportion of articles with spin was similar for articles accepted before and after PCORI peer review.DiscussionJust as spin is common in journal articles and press releases, we found that most reports submitted to PCORI included spin. While most spin was mitigated during the funder’s peer review process, we found no evidence that review of PCORI reports influenced spin in journal articles. Funders could explore interventions aimed at reducing spin in published articles of studies they support. (shrink)

The major challenge facing today’s biomedical researchers is the increasing competition for available funds. The competitive review process, through which the National Institutes of Health (NIH) awards grants, is built upon review by a committee of expert scientists. The NIH is firmly committed to ensuring that its peer review system is fair and objective.

The demand for greater public accountability is changing the nature of ex ante peer review at public science agencies worldwide. Based on a four year research project, this essay examines these changes through an analysis of the process of grant proposal review at two US public science agencies, the National Institutes of Health (NIH) and the National Science Foundation (NSF). Weaving historical and conceptual narratives with analytical accounts, we describe the ways in which these two agencies struggle (...) with the question of incorporating considerations of societal impact into the process of peer review. We use this comparative analysis to draw two main conclusions. First, evaluation of broader societal impacts is not different in kind from evaluation of intellectual merit. Second, the scientific community may actually bolster its autonomy by taking a broader range of considerations into its peer review processes. (shrink)

In this chapter we reflect on the relevance of peer review for assessing the quality of clinical ethics consultation. We contend that peer review in the narrative form as presented in this book provides an alternative to the formal clinical ethics consultation review procedures typically found in the clinical ethics literature. We elaborate on peer review as a reflection on clinical ethics consultation practice, the elements which a story should contain in order to (...) provide a basis for peer review, and the differences between the assessments of the peer reviewers. We argue that a narrative approach to assessing the quality of ethics consultation can be further developed by allowing all stakeholders who are involved in the clinical ethics practice to actively take part in the evaluation process, following a “responsive evaluation” approach. An example of this is creating a Community of Practice, the aim of which is to foster a joint learning process of all parties involved. At the end of this chapter, we draw conclusions on peer review as a dialogical tool for evaluating quality of clinical ethics consultation. (shrink)