To arrive at their final evaluation of a manuscript or grant proposal, reviewers must convert a submission’s strengths and weaknesses for heterogeneous peer review criteria into a single metric of quality or merit. I identify this process of commensuration as the locus for a new kind of peer review bias. Commensuration bias illuminates how the systematic prioritization of some peer review criteria over others permits and facilitates problematic patterns of publication and funding (...) in science. Commensuration bias also foregrounds a range of structural strategies for realigning peer review practices and institutions with the aims of science. (shrink)

Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look at the empirical and methodological limitations of research on bias raises questions about the existence and extent of many hypothesized forms of bias. In addition, the notion of bias is predicated on an implicit ideal that, once articulated, (...) raises questions about the normative implications of research on bias in peer review. This review provides a brief description of the function, history, and scope of peer review; articulates and critiques the conception of bias unifying research on bias in peer review; characterizes and examines the empirical, methodological, and normative claims of bias in peer review research; and assesses possible alternatives to the status quo. We close by identifying ways to expand conceptions and studies of bias to countenance the complexity of social interactions among actors involved directly and indirectly in peer review. (shrink)

Peer reviews of 84 organic farming grant applications from Sweden were analyzed to determine whether the reviewers’ affiliation to one of two types of agriculture (i.e., organic and conventional) influenced their reviews. Fifteen reviewers were divided into three groups: (1) scientists with experience in organic farming research; (2) scientists with no experience in organic farming research; and (3) users of organic farming research. The two groups of scientists assessed the societal relevance and scientific quality of the grant applications based (...) on three criteria (i.e., presentation, methodology, qualifications), whereas the user group only assessed societal relevance. The analysis showed that the two groups of scientists provided very different reviews. Scientist reviewers with experience in organic farming research agreed more with the user group on research relevance than did scientist reviewers without such experience, and the assessment of relevance was closely correlated to the assessment of scientific quality within both scientific groups. As both scientific groups did not clearly distinguish between societal relevance and scientific quality, the idea of an objective science is challenged. The contextual values associated with the norms of good agriculture were not clearly distinguished from the constitutive values of science associated with the traditional norms of good science. This raises the question of whether organic and conventional grant applications should be mixed for review regardless of the reviewers. (shrink)

Peer reviewers at many funding agencies and scientific journals are asked to score submissions both on individual criteria and overall. The overall scores should be some kind of aggregate of the criteria scores. Carole Lee identifies this as a potential locus for bias to enter the peer review process, which she calls commensuration bias. Here I view the aggregation of scores through the lens of social choice theory. I argue that, when reviewing grant proposals, it (...) is in many cases impossible to avoid commensuration bias. (shrink)

Implicit biases involve associations outside conscious awareness that lead to a negative evaluation of a person on the basis of irrelevant characteristics such as race or gender. This review examines the evidence that healthcare professionals display implicit biases towards patients. PubMed, PsychINFO, PsychARTICLE and CINAHL were searched for peer-reviewed articles published between 1st March 2003 and 31st March 2013. Two reviewers assessed the eligibility of the identified papers based on precise content and quality criteria. The references of eligible (...) papers were examined to identify further eligible studies. Forty two articles were identified as eligible. Seventeen used an implicit measure, to test the biases of healthcare professionals. Twenty five articles employed a between-subjects design, using vignettes to examine the influence of patient characteristics on healthcare professionals’ attitudes, diagnoses, and treatment decisions. The second method was included although it does not isolate implicit attitudes because it is recognised by psychologists who specialise in implicit cognition as a way of detecting the possible presence of implicit bias. Twenty seven studies examined racial/ethnic biases; ten other biases were investigated, including gender, age and weight. Thirty five articles found evidence of implicit bias in healthcare professionals; all the studies that investigated correlations found a significant positive relationship between level of implicit bias and lower quality of care. The evidence indicates that healthcare professionals exhibit the same levels of implicit bias as the wider population. The interactions between multiple patient characteristics and between healthcare professional and patient characteristics reveal the complexity of the phenomenon of implicit bias and its influence on clinician-patient interaction. The most convincing studies from our review are those that combine the IAT and a method measuring the quality of treatment in the actual world. Correlational evidence indicates that biases are likely to influence diagnosis and treatment decisions and levels of care in some circumstances and need to be further investigated. Our review also indicates that there may sometimes be a gap between the norm of impartiality and the extent to which it is embraced by healthcare professionals for some of the tested characteristics. Our findings highlight the need for the healthcare profession to address the role of implicit biases in disparities in healthcare. More research in actual care settings and a greater homogeneity in methods employed to test implicit biases in healthcare is needed. (shrink)

BackgroundThe process of peer-review in academia has attracted criticism surrounding issues of bias, fairness, and professionalism; however, frequency of occurrence of such comments is unknown.MethodsWe evaluated 1491 sets of reviewer comments from the fields of “Ecology and Evolution” and “Behavioural Medicine,” of which 920 were retrieved from the online review repository Publons and 571 were obtained from six early career investigators. Comment sets were coded for the occurrence of “unprofessional comments” and “incomplete, inaccurate or unsubstantiated critiques” (...) using an a-prior rubric based on our published research. Results are presented as absolute numbers and percentages.ResultsOverall, 12% of comment sets included at least one unprofessional comment towards the author or their work, and 41% contained incomplete, inaccurate of unsubstantiated critiques.ConclusionsThe large number of unprofessional comments, and IIUCs observed could heighten psychological distress among investigators, particularly those at an early stage in their career. We suggest that development and adherence to a universally agreed upon reviewer code of conduct is necessary to improve the quality and professional experience of peer review. (shrink)

Peer review is a widely accepted instrument for raising the quality of science. Peer review limits the enormous unstructured influx of information and the sheer amount of dubious data, which in its absence would plunge science into chaos. In particular, peer review offers the benefit of eliminating papers that suffer from poor craftsmanship or methodological shortcomings, especially in the experimental sciences. However, we believe that peer review is not always appropriate for the (...) evaluation of controversial hypothetical science. We argue that the process of peer review can be prone to bias towards ideas that affirm the prior convictions of reviewers and against innovation and radical new ideas. Innovative hypotheses are thus highly vulnerable to being “filtered out” or made to accord with conventional wisdom by the peer review process. Consequently, having introduced peer review, the Elsevier journal Medical Hypotheses may be unable to continue its tradition as a radical journal allowing discussion of improbable or unconventional ideas. Hence we conclude by asking the publisher to consider re-introducing the system of editorial review to Medical Hypotheses. (shrink)

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.

BackgroundThe Patient-Centered Outcomes Research Institute is obligated to peer review and to post publicly “Final Research Reports” of all funded projects. PCORI peer review emphasizes adherence to PCORI’s Methodology Standards and principles of ethical scientific communication. During the peer review process, reviewers and editors seek to ensure that results are presented objectively and interpreted appropriately, e.g., free of spin.MethodsTwo independent raters assessed PCORI peer review feedback sent to authors. We calculated the proportion (...) of reports in which spin was identified during peer review, and the types of spin identified. We included reports submitted by April 2018 with at least one associated journal article. The same raters then assessed whether authors addressed reviewers’ comments about spin. The raters also assessed whether spin identified during PCORI peer review was present in related journal articles.ResultsWe included 64 PCORI-funded projects. Peer reviewers or editors identified spin in 55/64 submitted research reports. Types of spin included reporting bias, inappropriate interpretation, inappropriate extrapolation of results, and inappropriate attribution of causality. Authors addressed comments about spin related to 47/55 of the reports.Of 110 associated journal articles, PCORI comments about spin were potentially applicable to 44/110 articles, of which 27/44 contained the same spin that was identified in the PCORI research report. The proportion of articles with spin was similar for articles accepted before and after PCORI peer review.DiscussionJust as spin is common in journal articles and press releases, we found that most reports submitted to PCORI included spin. While most spin was mitigated during the funder’s peer review process, we found no evidence that review of PCORI reports influenced spin in journal articles. Funders could explore interventions aimed at reducing spin in published articles of studies they support. (shrink)

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials (...) we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated. (shrink)

At present, peer review is the most common method used by funding agencies to make decisions about resource allocation. But how reliable, efficient, and fair is it in practice? The ex ante evaluation of scientific novelty is a fundamentally uncertain endeavor; bias and chance are embedded in the final outcome. In the current study, I will examine some of the most central problems of peer review and highlight the possible benefits of using a lottery as (...) an alternative decision-making mechanism. Lotteries are driven by chance, not reason. The argument made in the study is that the epistemic landscape could benefit in several respects by using a lottery, thus avoiding all types of bias, disagreement, and other limitations associated with the peer review process. Funding agencies could form a pool of funding applicants who have minimal qualification levels and then select randomly within that pool. The benefits of a lottery would not only be that it saves time and resources, but also that it contributes to a more dynamic selection process and increases the epistemic diversity, fairness, and impartiality within academia. (shrink)

In a recent article in the British Journal for the Philosophy of Science, Heesen and Bright argue that prepublication peer review should be abolished and replaced with postpublication peer review (provided the matter is judged purely on epistemic grounds). In this article, I show that there are three problems with their argument. First, it fails to consider the epistemic cost of implementing the change to postpublication peer review. Second, it fails to consider some potential (...) epistemic benefits of prepublication peer review, which involve avoiding bias. Third, it fails to consider some potential epistemic disadvantages of postpublication peer review, which stem from the greater number of papers that would be published under that system. (shrink)

Peer review has long been regarded as the gold standard of scientific publication, essential to the integrity of science itself. But, as any publishing scientist knows, peer review has its downside, including long delays and reviewer bias. Until the coming of the Internet, there appeared to be no alternative. Now, articles appear online as preprints almost immediately upon submission. But they lack peer review and thus their scientific standing can be questioned. Post-publication discussion (...) platforms such as PubPeer have proven useful, but are no substitute for pre-publication peer review. Nevertheless, some may be tempted to believe that peer review can now be done without. This article challenges that view by analyzing a recent, non-peer-reviewed article in Skeptical Enquirer, a magazine published by the Committee for Skeptical Enquiry (CSI). The article, “Sodom Meteor Strike Claims Should Be Taken With a Pillar of Salt,” casts doubt on one of the most widely read scientific articles of the last decade and provides a stern warning of the cost of abandoning peer review. (shrink)

BackgroundDescriptive studies examining publication rates and citation counts demonstrate a geographic skew toward high-income countries (HIC), and research from low- or middle-income countries (LMICs) is generally underrepresented. This has been suggested to be due in part to reviewers’ and editors’ preference toward HIC sources; however, in the absence of controlled studies, it is impossible to assert whether there is bias or whether variations in the quality or relevance of the articles being reviewed explains the geographic divide. This study synthesizes (...) the evidence from randomized and controlled studies that explore geographic bias in the peer review process.MethodsA systematic review was conducted to identify research studies that explicitly explore the role of geographic bias in the assessment of the quality of research articles. Only randomized and controlled studies were included in the review. Five databases were searched to locate relevant articles. A narrative synthesis of included articles was performed to identify common findings.ResultsThe systematic literature search yielded 3501 titles from which 12 full texts were reviewed, and a further eight were identified through searching reference lists of the full texts. Of these articles, only three were randomized and controlled studies that examined variants of geographic bias. One study found that abstracts attributed to HIC sources elicited a higher review score regarding relevance of the research and likelihood to recommend the research to a colleague, than did abstracts attributed to LIC sources. Another study found that the predicted odds of acceptance for a submission to a computer science conference were statistically significantly higher for submissions from a “Top University.” Two of the studies showed the presence of geographic bias between articles from “high” or “low” prestige institutions.ConclusionsTwo of the three included studies identified that geographic bias in some form was impacting on peer review; however, further robust, experimental evidence is needed to adequately inform practice surrounding this topic. Reviewers and researchers should nonetheless be aware of whether author and institutional characteristics are interfering in their judgement of research. (shrink)

Peer review is often taken to be the main form of quality control on academic research. Usually journals carry this out. However, parts of maths and physics appear to have a parallel, crowd-sourced model of peer review, where papers are posted on the arXiv to be publicly discussed. In this paper we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting papers by quality. Our argument (...) rests on two key claims. First, crowd-sourced peer review will lead on average to more reviewers per paper than journal-solicited peer review. Second, due to the wisdom of the crowds, more reviewers will tend to make better judgments than fewer. We make the second claim precise by looking at the Condorcet Jury Theorem as well as two related jury theorems developed specifically to apply to peer review. (shrink)

A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since (...) they make decisions that have a significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making. (shrink)

In the past few decades, peer review has come to dominate virtually all professionally respectable academic and scientific publications. However, despite its near-universal acceptance, no code of conduct has been developed to which peer reviewers and their editors are encouraged to adhere. This paper proposes such a code of conduct.

A recent commentary argued for arbitration to resolve authorship disputes within academic research settings explaining that current mechanisms to resolve conflicts result in unclear outcomes and institutional power vested in senior investigators could compromise fairness. We argue here that arbitration is not a suitable means to resolve disputes among researchers in academia because it remains unclear who will assume the costs of arbitration, the rules of evidence do not apply to arbitration, and decisions are binding and very difficult to appeal. (...) Instead of arbitration, we advocate for peer-based approaches involving a peer review committee and research ethics consultation to help resolve authorship disagreements. We describe the composition of an institutional peer review committee to address authorship disputes. Both of these mechanisms are found, or can be formed, within academic institutions and offer several advantages to researchers who are likely to shy away from legalistic processes and gravitate towards those handled by their peers. Peer-based approaches are cheaper than arbitration and the experts involved have knowledge about academic publishing and the culture of research in the specific field. Decisions by knowledgeable and neutral experts could reduce bias, have greater authority, and could be appealed. Not only can peer-based approaches be leveraged to resolve authorship disagreements, but they may also enhance collegiality and promote a healthy team environment. (shrink)

Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes (...) including editorial responsibility, the subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review. (shrink)

Background: In biomedical research, there have been numerous scandals highlighting conflicts of interest (COIs) leading to significant bias in judgment and questionable practices. Academic institutions, journals, and funding agencies have developed and enforced policies to mitigate issues related to COI, especially surrounding financial interests. After a case of editorial COI in a prominent bioethics journal, there is concern that the same level of oversight regarding COIs in the biomedical sciences may not apply to the field of bioethics. In this (...) study, we examined the availability and comprehensiveness of COI policies for authors, peer reviewers, and editors of bioethics journals. Methods: After developing a codebook, we analyzed the content of online COI policies of 63 bioethics journals, along with policy information provided by journal editors that was not publicly available. Results: Just over half of the bioethics journals had COI policies for authors (57%), and only 25% for peer reviewers and 19% for editors. There was significant variation among policies regarding definitions, the types of COIs described, the management mechanisms, and the consequences for noncompliance. Definitions and descriptions centered on financial COIs, followed by personal and professional relationships. Almost all COI policies required disclosure of interests for authors as the primary management mechanism. Very few journals outlined consequences for noncompliance with COI policies or provided additional resources. Conclusion: Compared to other studies of biomedical journals, a much lower percentage of bioethics journals have COI policies and these vary substantially in content. The bioethics publishing community needs to develop robust policies for authors, peer reviewers, and editors and these should be made publicly available to enhance academic and public trust in bioethics scholarship. (shrink)

BackgroundFor a long time, in sports, researchers have tried to understand an expert by comparing them with novices, raising the doubts if the visual search characteristics distinguish experts from novices. Therefore, the aim of the present study was to review and conduct a meta-analysis to evaluate the differences in visual search behavior between experts and novices in team sports athletes.MethodsThis systematic review with meta-analysis followed the PRISMA 2020 and Cochrane's guidelines. Healthy team athletes were included, which engaged in (...) regular practice, from any sex or competitive level, specifically classified a priori as expert or novice in the original research. We considered only research published in peer-reviewed journals, with no limitations regarding date or language. It was considered healthy team sport athletes engaged in regular practice. The scenarios could be in situ or film-based. The databases of EBSCO, PubMed, Scopus, SPORTDiscus, and Web of Science were used to perform the searches. The risk of bias was calculated through the RoBANS tool.ResultsFrom a total of 6,257 records, of which 985 were duplicates, titles and abstracts of 5,272 were screened, and 45 required full-text analysis. Of those, 23 were excluded due to not fulfilling the eligibility criteria regarding participants. In the end, 22 studies were selected, however, as two studies were part of the same trial and were analyzed conjointly.DiscussionExperts showed to be older and with more years of practice. The ability to distinguish experts from novices was not so clear regarding the variables analyzed. This could be due to the strategies chosen in each study, which were specific to each scenario, and when grouping all together, it was lost information within non-representative averages. The distinction between experts and novices was not clear, showing a lot of heterogeneity in the included studies. The expert classification itself may have been the conditioning aspect for these results, retaining the doubt and the need for more studies in the field.Systematic review registrationThe protocol was pre-registered in OSF. (shrink)

Peer review is a topic of considerable concern to many researchers, and there is a correspondingly large body of research on the topic. This issue of Science and Engineering Ethics presents recent work on peer review that is both grounded in empirical science and is applicable to policy decisions. This research raises two basic questions; (a) how does current peer review operate, and (b) how can it be improved? Topics addressed include descriptions of how (...) peer review is used in Federal agencies. whether peer review leads to better manuscripts, demographic characteristics of authors or reviewers (status or institutional affiliation), blinding of reviewers, authors, or results, reliability and consistency of reviews, accepting a paper before the study is done, simultaneous submission, and use of dispute resolution procedures such as scientific dialectical and pleading protocols. (shrink)

Does hindsight knowledge make research seem more ethical and predictable? In line with the notion of hindsight bias, students in 3 experiments knowing the outcome of an animal experiment judged the results as more foreseeable and ethical relative to students who did not know the outcome. Via self to other comparisons, students evaluate themselves more favorably compared to a peer but exhibited hindsight bias in doing so. Uniquely, the findings reveal the possibility that students deem themselves to (...) be more skeptical and objective than their peers. Implications for teaching animal ethics and for Institutional Review Boards are discussed. (shrink)

Prediction markets, which trade contracts based on the results of predictions, have been remarkably successful in predicting the results of political events. A number of proposals have been made to extend prediction markets to scientific questions, and some small-scale science prediction markets have been implemented. Advocates for science prediction markets argue that they could alleviate problems in science such as bias in peer review and epistemically unjustified consensus. I argue that bias in peer review (...) and epistemically unjustified consensuses are genuine problems in science, and that current attempts to solve them fail in practical circumstances. Prediction markets do show some promise for answering scientific questions while avoiding the pitfalls of consensus and peer review. However, there are strong reasons to believe that science prediction markets will not perform nearly so well as election prediction markets, and implementing prediction markets on a large scale could have significant unintended consequences for the organization of scientific research. The promise of prediction markets to solve current problems in assessing scientific claims is therefore largely illusory. (shrink)

Rooted in altruism theory, the purpose of the double-blind academic journal peer-review process is to: assess the quality of scientific research, minimize the potential for nepotism, and; advance the standards of research through high-quality, constructive feedback. However, considering the limited, if any, public recognition and monetary incentives that referees receive for reviewing manuscripts, academics are often reluctant to squander their limited time toward peer reviewing manuscripts. If they do accept such invitations, referees, at times, do not invest (...) the appropriate time needed and, as a result, scantily review manuscripts, which adversely affects the quality of the review. In addition, given that authors’ identities are not blind to journal editors, there is the potential for bias toward well-established academics from highly-ranked institutions. As a result of these issues, the aims of the academic journal review process are currently not being fulfilled. To rectify these issues, several recommendations, namely: single-blind the editors, pay reviewers, standardize the review process, increase the acceptance standards at academic conferences, and provide constructive feedback, are offered. (shrink)

This research examined the experiences of authors of academic journal articles in the educational sector of all eight universities in Finland. The ethical principles of peer review and best and worst review processes were in focus. Data were gathered by electronic questionnaire, which was completed by 121 respondents who represented well the heterogeneity of the staff in the educational sector. Out of nine ethical principles honesty, constructiveness, and impartiality were appreciated but promptness, balance, and diplomacy were criticized. (...) According to two open questions, a third of authors praised and blamed reviewers as experts and non-experts. The accuracy of feedback was more often present in the best rather than in the worst experienced review processes. Journals’ editors and their decision-making called forth more negative than positive accounts. The results were discussed in the context of ethical codes for reviewers and researchers’ findings regarding the ethical responsibilities to promote good science with thorough, appropriate, and honest feedback and feedforward. (shrink)

BackgroundDespite its ubiquity in academic research, the phrase ‘ethical challenge(s)’ appears to lack an agreed definition. A lack of a definition risks introducing confusion or avoidable bias. Conceptual clarity is a key component of research, both theoretical and empirical. Using a rapid review methodology, we sought to review definitions of ‘ethical challenge(s)’ and closely related terms as used in current healthcare research literature.MethodsRapid review to identify peer-reviewed reports examining ‘ethical challenge(s)’ in any context, extracting data (...) on definitions of ‘ethical challenge(s)’ in use, and synonymous use of closely related terms in the general manuscript text. Data were analysed using content analysis. Four databases (MEDLINE, Philosopher’s Index, EMBASE, CINAHL) were searched from April 2016 to April 2021.Results393 records were screened, with 72 studies eligible and included: 53 empirical studies, 17 structured reviews and 2 review protocols. 12/72 (17%) contained an explicit definition of ‘ethical challenge(s), two of which were shared, resulting in 11 unique definitions. Within these 11 definitions, four approaches were identified: definition through concepts; reference to moral conflict, moral uncertainty or difficult choices; definition by participants; and challenges linked to emotional or moral distress. Each definition contained one or more of these approaches, but none contained all four. 68/72 (94%) included studies used terms closely related to synonymously refer to ‘ethical challenge(s)’ within their manuscript text, with 32 different terms identified and between one and eight different terms mentioned per study.ConclusionsOnly 12/72 studies contained an explicit definition of ‘ethical challenge(s)’, with significant variety in scope and complexity. This variation risks confusion and biasing data analysis and results, reducing confidence in research findings. Further work on establishing acceptable definitional content is needed to inform future bioethics research. (shrink)

The questions of scientific choice which were left unresolved when the rapid expansion of academic science in the United States began in the early 1960s have come back to trouble the scientific community. There is now widespread dissatisfaction with the process of review by peers as one of the major systems for the allocation of public funds for research. While earlier criticisms had been brushed off by the assertion—unsupported by facts—that no other systems existed, the present situation cannot be (...) so easily dismissed.A serious examination of other national and international arrangements shows that a wide variety of procedures are in use and there is no research which shows that one system is either more productive scientifically, or more cost-effective in bringing about valuable scientific research. New systems which may be considered should avoid the major defects of the system of peer review as now practised: the enormous waste of scientists' time, the great potential for conflicts of interest, and the inherent bias against innovation.The principal system which I have proposed here combines the best elements of peer review with the simplicity and efficiency of the use of a formula. Moreover, this formula based on peer review of performance incorporates all the elements for which the academic scientific establishment should be accountable to its patron, which is the public treasury. A final virtue of the proposed system is that it provides simple and convenient procedures through the use of numerical weighting factors for the policy-maker to guide the support of scientific research as a whole. (shrink)

High percentages of submitted papers are rejected at editorial levels without offering a second chance to authors by sending their papers for further peer-reviews. In most cases, the rejections are typical quick answers without helpful argumentations related to the content of the rejected material. More surprisingly, some journals vaunt their high rejection rates as a “mark of prestige”!However, journals that reject high percentages of submitted papers have built their prominent positions based on a flawed measure, the impact factor, and (...) from a long and favorable historical context. Their shareholders may think that they are allowed to have a large margin of rejection rates without affecting their sponsorship or funding sources thanks to an extended anchorage since tens, or in some cases hundreds, of years compared to unknown or new journals that struggle to pave a way in the scientific publication world. Historical anchorage of some journals also makes it unfair to compare old and new .. (shrink)

The objective of this article was to review existing research to assess the evidence for predictive processing in sign language, the conditions under which it occurs, and the effects of language mastery on the neural bases of PP. This review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework. We searched peer-reviewed electronic databases and gray literature. We also searched the reference lists of records selected for the review and forward citations to identify all (...) relevant publications. We searched for records based on five criteria. To reduce the risk of bias, the remaining two authors with expertise in sign language processing and a variety of research methods reviewed the results. Disagreements were resolved through extensive discussion. In the final review, 7 records were included, of which 5 were published articles and 2 were dissertations. The reviewed records provide evidence for PP in signing populations, although the underlying mechanism in the visual modality is not clear. The reviewed studies addressed the motor simulation proposals, neural basis of PP, as well as the development of PP. All studies used dynamic sign stimuli. Most of the studies focused on semantic prediction. The question of the mechanism for the interaction between one’s sign language competence and PP in the manual-visual modality remains unclear, primarily due to the scarcity of participants with varying degrees of language dominance. There is a paucity of evidence for PP in sign languages, especially for frequency-based, phonetic, and syntactic prediction. However, studies published to date indicate that Deaf native/native-like L1 signers predict linguistic information during sign language processing, suggesting that PP is an amodal property of language processing.Systematic Review Registration[https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021238911], identifier [CRD42021238911]. (shrink)

Authors endure considerable hardship carrying out biomedical research, from generating ideas to completing their manuscripts and submitting their findings and data (as is increasingly required) to a journal. When researchers submit to journals, they entrust their findings and ideas to editors and peer reviewers who are expected to respect the confidentiality of peer review. Inherent trust in peer review is built on the ethical conduct of authors, editors and reviewers, and on the respect of this (...) confidentiality. If such confidentiality is breached by unethical reviewers who might steal or plagiarize the authors’ ideas, researchers will lose trust in peer review and may resist submitting their findings to that journal. Science loses as a result, scientific and medical advances slow down, knowledge may become scarce, and it is unlikely that increasing bias in the literature will be detected or eliminated. In such a climate, society will ultimately be deprived from scientific and medical advances. Despite a rise in documented cases of abused peer review, there is still a relative lack of qualitative and quantitative studies on reviewer-related misconduct, most likely because evidence is difficult to come by. Our paper presents an assessment of editors’ and reviewers’ responsibilities in preserving the confidentiality of manuscripts during the peer review process, in response to a 2016 case of intellectual property theft by a reviewer. Our main objectives are to propose additional measures that would offer protection of authors’ intellectual ideas from predatory reviewers, and increase researchers’ awareness of the responsible reviewing of journal articles and reporting of biomedical research. (shrink)

BackgroundDouble-blind peer review has been proposed as a possible solution to avoid implicit referee bias in academic publishing. The aims of this study are to analyse the demographics of corresponding authors choosing double-blind peer review and to identify differences in the editorial outcome of manuscripts depending on their review model.MethodsData includes 128,454 manuscripts received between March 2015 and February 2017 by 25 Nature-branded journals. We investigated the uptake of double-blind review in relation to (...) journal tier, as well as gender, country, and institutional prestige of the corresponding author. We then studied the manuscripts’ editorial outcome in relation to review model and author’s characteristics. The gender (male, female, or NA) of the corresponding authors was determined from their first name using a third-party service (Gender API). The prestige of the corresponding author’s institutions was measured from the data of the Global Research Identifier Database (GRID) by dividing institutions in three prestige groups with reference to the 2016 Times Higher Education (THE) ranking. We employed descriptive statistics for data exploration, and we tested our hypotheses using Pearson’s chi-square and binomial tests. We also performed logistic regression modelling with author update, out-to-review, and acceptance as response, and journal tier, author gender, author country, and institution as predictors.ResultsAuthor uptake for double-blind submissions was 12% (12,631 out of 106,373). We found a small but significant association between journal tier and review type (p value < 0.001, Cramer’s V = 0.054, df = 2). We had gender information for 50,533 corresponding authors and found no statistically significant difference in the distribution of peer review model between males and females (p value = 0.6179). We had 58,920 records with normalised institutions and a THE rank, and we found that corresponding authors from the less prestigious institutions are more likely to choose double-blind review (p value < 0.001, df = 2, Cramer’s V = 0.106). In the ten countries with the highest number of submissions, we found a large significant association between country and review type (p value < 0.001, df = 10, Cramer’s V = 0.189). The outcome both at first decision and post review is significantly more negative (i.e. a higher likelihood for rejection) for double-blind than single-blind papers (p value < 0.001, df = 1, Cramer’s V = 0.112 for first decision; p value < 0.001; df = 1, Cramer’s V = 0.082 for post-review decision).ConclusionsThe proportion of authors that choose double-blind review is higher when they submit to more prestigious journals, they are affiliated with less prestigious institutions, or they are from specific countries; the double-blind option is also linked to less successful editorial outcomes. (shrink)

Background Implicit biases are present in the general population and among professionals in various domains, where they can lead to discrimination. Many interventions are used to reduce implicit bias. However, uncertainties remain as to their effectiveness. -/- Methods We conducted a systematic review by searching ERIC, PUBMED and PSYCHINFO for peer-reviewed studies conducted on adults between May 2005 and April 2015, testing interventions designed to reduce implicit bias, with results measured using the Implicit Association Test (IAT) (...) or sufficiently similar methods. -/- Results 30 articles were identified as eligible. Some techniques, such as engaging with others’ perspective, appear unfruitful, at least in short term implicit bias reduction, while other techniques, such as exposure to counterstereotypical exemplars, are more promising. Robust data is lacking for many of these interventions. -/- Conclusions Caution is thus advised when it comes to programs aiming at reducing biases. This does not weaken the case for implementing widespread structural and institutional changes that are multiply justified. (shrink)

Authors endure considerable hardship carrying out biomedical research, from generating ideas to completing their manuscripts and submitting their findings and data to a journal. When researchers submit to journals, they entrust their findings and ideas to editors and peer reviewers who are expected to respect the confidentiality of peer review. Inherent trust in peer review is built on the ethical conduct of authors, editors and reviewers, and on the respect of this confidentiality. If such confidentiality (...) is breached by unethical reviewers who might steal or plagiarize the authors’ ideas, researchers will lose trust in peer review and may resist submitting their findings to that journal. Science loses as a result, scientific and medical advances slow down, knowledge may become scarce, and it is unlikely that increasing bias in the literature will be detected or eliminated. In such a climate, society will ultimately be deprived from scientific and medical advances. Despite a rise in documented cases of abused peer review, there is still a relative lack of qualitative and quantitative studies on reviewer-related misconduct, most likely because evidence is difficult to come by. Our paper presents an assessment of editors’ and reviewers’ responsibilities in preserving the confidentiality of manuscripts during the peer review process, in response to a 2016 case of intellectual property theft by a reviewer. Our main objectives are to propose additional measures that would offer protection of authors’ intellectual ideas from predatory reviewers, and increase researchers’ awareness of the responsible reviewing of journal articles and reporting of biomedical research. (shrink)

Scientific communication takes place at two registers: first, interactions with colleagues in close proximity—members of a network, school of thought or circle; second, depersonalised transactions among a potentially unlimited number of scholars can be involved (e.g., author and readers). The interference between the two registers in the process of peer review produces a drift toward conflict of interest. Three particular cases of peer review are differentiated: journal submissions, grant applications and applications for tenure. The current conflict (...) of interest policies do not cover all these areas. Furthermore, they have a number of flaws, which involves an excessive reliance on scholars’ personal integrity. Conflicts of interest could be managed more efficiently if several elements and rules of the judicial process were accepted in science. The analysis relies on both primary and secondary data with a particular focus on Canada. (shrink)

This paper examines the problem of vituperative feedback from peer reviewers. We argue that such feedback is morally unacceptable, insofar as it humiliates authors and damages their dignity. We draw from social-psychological research to explore those aspects of the peer-review process in general and the anonymity of blind reviewing in particular that contribute to reviewers’ humiliating comments. We then apply Iris Murdoch's ideas about a virtuous consciousness and humility to make the case that peer referees have (...) a moral obligation not to humiliate the authors whose work they review. (shrink)

During the early years of the COVID-19 pandemic, preclinical and clinical research were sped up and scaled up in both the public and private sectors and in partnerships between them. This resulted in some extraordinary advances, but it also raised a range of issues regarding the ethics, rigour, and integrity of scientific research, academic publication, and public communication. Many of the failures of scientific rigour and integrity that occurred during the pandemic were exacerbated by the rush to generate, disseminate, and (...) implement research findings, which not only created opportunities for unscrupulous actors but also compromised the methodological, peer review, and advisory processes that would usually identify sub-standard research and prevent compromised clinical or policy-level decisions. While it would be tempting to attribute these failures of science and its translation solely to the “unprecedented” circumstances of the COVID-19 pandemic, the reality is that they preceded the pandemic and will continue to arise once it is over. Existing strategies for promoting scientific rigour and integrity need to be made more rigorous, better integrated into research training and institutional cultures, and made more sophisticated. They might also need to be modified or supplemented with other strategies that are fit for purpose not only in public health emergencies but in any research that is sped-up and scaled up to address urgent unmet medical needs. (shrink)

The purpose of grant peer review is to identify the most excellent and promising research projects. However, sociologists of science and STS scholars have shown that peer review tends to promote solid low-risk projects at the expense of more original and innovative projects that often come with higher risk. It has also been shown that the review process is affected by significant measures of chance. Against this background, the aim of this study is to theorize (...) the notions of academic judgment and agonistic chance, and to present and analyze situations in which expert reviewers are faced with the challenge of trying to decide which grant proposals to select when there is strong disagreement. The empirical analysis is based on ethnographic observations of ten panel groups at the Swedish Research Council in the areas of natural and engineering sciences. By focusing on disagreement, the study provides a more in-depth understanding of how agonistic chance creeps into the peer review process and becomes part of the consensus that is created. (shrink)

Peer review facilitates quality control and integrity of scientific research. Although publishing policies have adapted to include the use of Artificial Intelligence (AI) tools, such as Chat Generative Pre-trained Transformer (ChatGPT), in the preparation of manuscripts by authors, there is a lack of guidelines or policies on whether peer reviewers can use such tools. The present article highlights the lack of policies on the use of AI tools in the peer review process (PRP) and argues (...) that we need to go beyond policies by creating transparent procedures that will enable journals to investigate allegations of non-compliance and take decisions that will protect the integrity of the peer review system. Reviewers found to violate relevant policies must be excluded from the process to safeguard the integrity of the peer review system. (shrink)

Under the traditional system of peer-reviewed publication, the degree of prestige conferred to authors by successful publication is tied to the degree of the intellectual rigor of its peer review process: ambitious scientists do well professionally by doing well epistemically. As a result, we should expect journal editors, in their dual role as epistemic evaluators and prestige-allocators, to have the power to motivate improved author behavior through the tightening of publication requirements. Contrary to this expectation, I will (...) argue that the publication bias literature in academic medicine demonstrates that editor interventions have had limited effectiveness in improving the health of the publication and trial registration record, suggesting that much stronger interventions are needed. (shrink)

On the surface, developing a social psychology of science seems compelling as a way to understand how individual social cognition – in aggregate – contributes towards individual and group behavior within scientific communities (Kitcher, 2002). However, in cases where the functional input-output profile of psychological processes cannot be mapped directly onto the observed behavior of working scientists, it becomes clear that the relationship between psychological claims and normative philosophy of science should be refined. For example, a robust body of social (...) psychological research demonstrates implicit gender bias in the evaluation of others (e.g., Steinpreis, Anders, & Ritzke, 1999). Many expected implicit bias to be a major cause of women’s underrepresentation in math intensive fields of science; however, quantitative sociological research of hiring and manuscript and grant evaluation has discovered no gender disparity in outcomes (Ceci & Williams, 2011). Why might this be so? This paper will discuss methodological challenges that go beyond classic problems of external validity in extrapolating psychological effects and explanations to scientific communities. These problems include more complex external validity issues raised by the introduction of multi-process models of cognition (e.g., implicit versus explicit social cognition) as well as the reflexive role that folk and experimental theories of social psychology play in guiding the behavior of scientists at the individual and community level. (shrink)

Epistemological differences fuel continuous and frequently divisive debates in the social sciences and the humanities. Sociologists have yet to consider how such differences affect peer evaluation. The empirical literature has studied distributive fairness, but neglected how epistemological differences affect perception of fairness in decision making. The normative literature suggests that evaluators should overcome their epistemological differences by ‘‘translating’’ their preferred standards into general criteria of evaluation. However, little is known about how procedural fairness actually operates. Drawing on eighty-one interviews (...) with panelists serving on five multidisciplinary fellowship competitions in the social sciences and the humanities, we show that Evaluators generally draw on four epistemological styles to make arguments in favor of and against proposals. These are the constructivist, comprehensive, positivist, and utilitarian styles; and Peer reviewers define a fair decision-making process as one in which panelists engage in ‘‘cognitive contextualization,’’ that is, use epistemological styles most appropriate to the field or discipline of the proposal under review. (shrink)

BackgroundArtificial intelligence has been described as the “fourth industrial revolution” with transformative and global implications, including in healthcare, public health, and global health. AI approaches hold promise for improving health systems worldwide, as well as individual and population health outcomes. While AI may have potential for advancing health equity within and between countries, we must consider the ethical implications of its deployment in order to mitigate its potential harms, particularly for the most vulnerable. This scoping review addresses the following (...) question: What ethical issues have been identified in relation to AI in the field of health, including from a global health perspective?MethodsEight electronic databases were searched for peer reviewed and grey literature published before April 2018 using the concepts of health, ethics, and AI, and their related terms. Records were independently screened by two reviewers and were included if they reported on AI in relation to health and ethics and were written in the English language. Data was charted on a piloted data charting form, and a descriptive and thematic analysis was performed.ResultsUpon reviewing 12,722 articles, 103 met the predetermined inclusion criteria. The literature was primarily focused on the ethics of AI in health care, particularly on carer robots, diagnostics, and precision medicine, but was largely silent on ethics of AI in public and population health. The literature highlighted a number of common ethical concerns related to privacy, trust, accountability and responsibility, and bias. Largely missing from the literature was the ethics of AI in global health, particularly in the context of low- and middle-income countries.ConclusionsThe ethical issues surrounding AI in the field of health are both vast and complex. While AI holds the potential to improve health and health systems, our analysis suggests that its introduction should be approached with cautious optimism. The dearth of literature on the ethics of AI within LMICs, as well as in public health, also points to a critical need for further research into the ethical implications of AI within both global and public health, to ensure that its development and implementation is ethical for everyone, everywhere. (shrink)

Peer-review is neither reliable, fair, nor a valid basis for predicting ‘impact’: as quality control, peer-review is not fit for purpose. Endorsing the consensus, I offer a reframing: while a normative social process, peer-review also shapes the writing of a scientific paper. In so far as ‘cognition’ describes enabling conditions for flexible behavior, the practices of peer-review thus constrain knowledge-making. To pursue cognitive functions of peer-review, however, manuscripts must be seen (...) as ‘symbolizations’, replicable patterns that use technologically enabled activity. On this bio-cognitive view, peer-review constrains knowledge-making by writers, editors, reviewers. Authors are prompted to recursively re-aggregate symbolizations to present what are deemed acceptable knowledge claims. How, then, can recursive re-embodiment be explored? In illustration, I sketch how the paper’s own content came to be re-aggregated: agonistic review drove reformatting of argument structure, changes in rhetorical ploys and careful choice of wordings. For this reason, the paper’s knowledge-claims can be traced to human activity that occurs in distributed cognitive systems. Peer-review is on the frontline in the knowledge sector in that it delimits what can count as knowing. Its systemic nature is therefore crucial to not only discipline-centered ‘real’ science but also its ‘post-academic’ counterparts. (shrink)

General science literacy contributes to good public decision‐making about technology and medicine. This essay explores the kinds of science literacy currently developed by public education in the United States of America. It argues that current curricula on “science as inquiry” (formerly the “nature of science”) need to be brought up to date with the inclusion of discussion of social epistemological concepts such as trust and scientific authority, scientific disagreement versus science denialism, the role of ideology and bias in scientific (...) research, and the importance of peer review and responsiveness to criticism. (shrink)

Our recent paper reported that 43% of reviewer comment sets shared with authors contained at least one unprofessional comment or an incomplete, inaccurate of unsubstantiated critique. Publication of this work sparked an online conversation surrounding professionalism in peer review. We collected and analyzed these social media comments as they offered real-time responses to our work and provided insight into the views held by commenters and potential peer-reviewers that would be difficult to quantify using existing empirical tools. Overall, (...) 75% of comments were positive, of which 59% were supportive and 16% shared similar personal experiences. However, a subset of negative comments emerged, that provided potential insight into the reasons underlying unprofessional comments were made during the peer-review process. These comments were classified into three main themes: forced niceness will adversely impact the peer-review process and allow for publication of poor-quality science ; dismissing comments as not offensive to another person because they were not deemed personally offensive to the reader ; and authors brought unprofessional comments upon themselves as they submitted substandard work. Here, we argue against these themes as justifications for directing unprofessional comments towards authors during the peer review process. We argue that it is possible to be both critical and professional, and that no author deserves to be the recipient of demeaning ad hominem attacks regardless of supposed provocation. Suggesting otherwise only serves to propagate a toxic culture within peer review. While we previously postulated that establishing a peer-reviewer code of conduct could help improve the peer-review system, we now posit that priority should be given to repairing the negative cultural zeitgeist that exists in peer-review. (shrink)

Previous research has found that funding disparities are driven by applications’ final impact scores and that only a portion of the black/white funding gap can be explained by bibliometrics and topic choice. Using National Institutes of Health R01 applications for council years 2014–2016, we examine assigned reviewers’ preliminary overall impact and criterion scores to evaluate whether racial disparities in impact scores can be explained by application and applicant characteristics. We hypothesize that differences in commensuration—the process of combining criterion scores into (...) overall impact scores—disadvantage black applicants. Using multilevel models and matching on key variables including career stage, gender, and area of science, we find little evidence for racial disparities emerging in the process of combining preliminary criterion scores into preliminary overall impact scores. Instead, preliminary criterion scores fully account for racial disparities—yet do not explain all of the variability—in preliminary overall impact scores. (shrink)

As universities in the United Kingdom gear themselves up for the next Research Excellence Framework, REF2021, with peer review at its core, we critically re-visit the idea of peer review as a gold standard proxy for research excellence. We question the premise that anonymous peer review is a necessary and enabling condition for impartial, expert judgement. We argue that the intentions and supposed benefits underlying peer review and its associated concepts have become (...) congealed in received discourse about research quality. Hence we explore the key conceptual issues raised by the nested assumptions and concepts that come into play in peer review as currently practised: primarily those of secrecy, anonymity, legitimacy, trust, impartiality and openness. After delineating the benefits attributed to peer review, we contrast its declared virtues with its problematic features. We locate peer review in an audit culture in which the reviewer is an academic labourer. Drawing on recent trends in moral and political philosophy, we question the usefulness of the ideal of impartiality when tied to secrecy. Then we raise more deliberative, intersubjective possibilities for a revised understanding of peer review in the context of an academic community. Finally, we suggest ways in which the academic community could pursue quality in research by recasting peer review to be less secret and more open. (shrink)