While there is a substantive literature on the link between welfare states and individuals’ trust, little is known about the micro-linkage of the conditionality of welfare as a driver of trust. This study presents a unique randomized social experiment investigating this link. Recipients of the regular Dutch social assistance policy are compared to recipients of two alternative schemes inspired by the basic income and based on a more trusting and unconditional approach, testing the main reciprocity argument in the literature: a (...) trusting government will harvest trust from welfare recipients in return. Particularly trust in local government – the level at which the experiment was implemented – increases among recipients of the alternative treatments. Subsequently, we innovatively theorize and test rigorously which mediating mechanisms might explain this increase. Policy evaluation, social integration, and psychological well-being are studied in this respect. Of these, the only underlying mechanism proven to mediate the treatment effect in local political trust, is citizens’ satisfaction with policy. (shrink)

Modern education attaches great importance to interdisciplinary skills, among which computational thinking is a core element, and heralds a new era. IT application has shaped education in the 21st century. Computational thinking has provided further impetus for building an all-encompassing social network and fostering a DIY culture enabled by digital technologies. One empirical study used four apps to test children's development in computational thinking and fluency. The article will help students overcome their fears of coding. Peer reviews provide students (...) with an opportunity to learn from each other and become more motivated. These reviews also serve as feedback for teachers to evaluate students' performance. Experimental design is used in this study, and a peer review system is implemented. Freshmen attending a programming class in a university are used as samples. At the class, students write computer programs with f-Chart, which provides a graphical user interface for students to learn programming logic and design. Zuvio, a cloud-based interactive response system, is used to conduct the peer reviews. The data of this study are analyzed through R. The results show not only an improvement in students' learning performance but also a gap between students' peer review scores and teachers' evaluation scores. Learning feedback and evaluation is crucial to transform education between students and teachers into a sustainable cycle system. (shrink)

The prevention of sexual violence is a major goal of sexual health. In cases of accused sexual offenders, the assessment of diminished criminal responsibility of the accused is one of the most important procedures undertaken by experts in the German legal system. This assessment follows a two-stage method assessing first the severity of a paraphilic disorder and then second criteria for or against diminished capacity. The present study examines the predictive validity of two different sets of criteria for the assessment (...) of criminal responsibility in the context of paraphilic disorders combined with sexual offending. Two exemplary case vignettes of two suspected sexual offenders were developed to assess the criteria. For each participant, one of the two exemplary case vignettes was randomly presented. The presentation of the two different sets of criteria was also randomized, so that each participant was assigned only one of the two criteria sets to rate one of the presented cases. N = 349 participants from different professional backgrounds completed their assessments and were included in the data analysis. The data were evaluated using logistic regression. Results show that the more recently published criteria set predicts both the severity of the disorder as well as the diminished capacity twice as good as the older criteria set of Boetticher et al. currently used regularly for forensic court reports. In preliminary conclusion, the new criteria of Briken and Müller form an empirically based assessment of criminal responsibility. However, the proposed criteria cannot replace an extensive exploration of the accused person and a careful file analysis. Validity and reliability of the results are also limited due to the methodical choice of a vignette study. (shrink)

Children's cooperation with peers undergoes substantial developmental changes between 3 and 10 years of age. Here we stipulate that young children's initial fearfulness of peers' behaviour develops into older children's fearfulness of peers' evaluations of their own behaviour. Cooperation may constitute an adaptive environment in which the expressions of fear and self-conscious emotions regulate the quality of children's peer relationships.

Our recent paper reported that 43% of reviewer comment sets shared with authors contained at least one unprofessional comment or an incomplete, inaccurate of unsubstantiated critique. Publication of this work sparked an online conversation surrounding professionalism in peer review. We collected and analyzed these social media comments as they offered real-time responses to our work and provided insight into the views held by commenters and potential peer-reviewers that would be difficult to quantify using existing empirical tools. Overall, 75% (...) of comments were positive, of which 59% were supportive and 16% shared similar personal experiences. However, a subset of negative comments emerged, that provided potential insight into the reasons underlying unprofessional comments were made during the peer-review process. These comments were classified into three main themes: forced niceness will adversely impact the peer-review process and allow for publication of poor-quality science ; dismissing comments as not offensive to another person because they were not deemed personally offensive to the reader ; and authors brought unprofessional comments upon themselves as they submitted substandard work. Here, we argue against these themes as justifications for directing unprofessional comments towards authors during the peer review process. We argue that it is possible to be both critical and professional, and that no author deserves to be the recipient of demeaning ad hominem attacks regardless of supposed provocation. Suggesting otherwise only serves to propagate a toxic culture within peer review. While we previously postulated that establishing a peer-reviewer code of conduct could help improve the peer-review system, we now posit that priority should be given to repairing the negative cultural zeitgeist that exists in peer-review. (shrink)

This field survey in a fast food restaurant setting tested the hypothesized influences of two social context variables (role responsibility and interests of group members) and justice evaluations (distributive, procedural, and retributive) on respondents' inclination to report theft and their theft reporting behavior. The results provided mixed support for the hypotheses. Inclination to report a peer for theft was associated with role responsibility, the interests of group members, and procedural justice perceptions. Actual reporting behavior was associated with the inclination (...) to report and with retributive justice evaluations. Implications for future research and for management are discussed. (shrink)

Realising the societal gains from publicly funded health and medical research requires a model for a reflexive evaluation precedent for the societal impact of research. This research explores UK Research Excellence Framework evaluators’ values and opinions and assessing societal impact, prior to the assessment taking place. Specifically, we discuss the characteristics of two different impact assessment extremes – the “quality-focused” evaluation and “societal impact-focused” evaluation. We show the wide range of evaluator views about impact, and that these (...) views could be conceptually reflected in a range of different positions along a conceptual evaluation scale. We describe the characteristics of these extremes in detail, and discuss the different beliefs evaluators had which could influence where they positioned themselves along the scale. These decisions, we argue, when considered together, form a dominant definition of societal impact that influences the direction of its evaluation by the panel. (shrink)

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 (...) already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated. (shrink)

In the academic world, peer review is one of the major processes in evaluating a scholars contribution. In this study, we are interested in quantifying the merits of different policies in a peer review process, such as single-blind review, double-blind review, and obtaining authors feedback. Currently, insufficient work has been undertaken to evaluate the benefits of different peer review policies. One of the major reasons for this situation is the inability to conduct any empirical study because data (...) are presently unavailable. In this case, a computer simulation is one of the best ways to conduct a study. We perform a series of simulations to study the effects of different policies on a peer review process. In this study, we focus on the peer review process of a typical computer science conference. Our results point to the crucial role of program chairs in determining the quality and diversity of the articles to be accepted for publication. We demonstrate the importance of discussion among reviewers, suggest circumstances in which the double-blind review policy should be adopted, and question the credibility of the authors feedback mechanism. Finally, we stress that randomness plays an important role in the peer review process, and this role cannot be eliminated. Although our model may not capture every component of a peer review process, it covers some of the most essential elements. Thus, even the simulation results clearly cannot be taken as literal descriptions of an actual peer review process. However, we can at least still use them to identify alternative directions for future study. (shrink)

Peer review of journal submissions has become one of the most important pillars of quality management in academic publishing. Because of growing concerns with the quality and effectiveness of the system, a host of enthusiastic innovators has proposed and experimented with new procedures and technologies. However, little is known about whether these innovations manage to convince other journal editors. This paper will address open questions regarding the implementation of new review procedures, the occurrence rate of various peer review (...) procedures and their distribution over scientific disciplines or academic publishers, as well as the motivations for editors or publishers to engage in novel review procedures. It shows that in spite of enthusiastic innovation, the adoption of new peer review procedures is in fact very slow, with the exception of text similarity scanners. For now, peer review innovations appear to be restricted to specific niches in academic publishing. Analysing these niches, the article concludes with a reflection on the circumstances in which innovations might be more widely implemented. (shrink)

Born in the 17th century, journal peer review is an extremely diverse technology, constantly torn between two often incompatible goals: the validation of manuscripts conceived as a collective industrial-like reproducible process performed to assert scientific statements, and the dissemination of articles considered as a means to spur scientific discussion, raising controversies, and civically challenging a state of knowledge. Such a situation is particularly conducive to clarifying the processes of valuation and evaluation in journal peer review. In this (...) article, such processes are considered as specific tests in order to emphasize the uncertain properties of pre-tests manuscripts. On the one hand, evaluation tests are examined at the core of the validation of manuscripts, such as defining the coordination of judging instances or controlling the modalities of inter-knowledge between reviewers and authors. They are also studied regarding the dissemination of articles, notably through the contemporary conception of a continuing evaluation test termed “post publication peer review”. On the other hand, valuation tests are both part of the validation of manuscripts, such as the weighting of different judgments of the same manuscript and the tensions that these hierarchies cause, and of the dissemination of articles, such as attention metrics recording the uses of articles. The conclusion sketches out how the articulation of these different tests has recently empowered readers as a new key judging instance for dissemination and for validation, potentially transforming the definition of peers, and thus the whole process of journal peer review. (shrink)

An advertising firm''s ethical culture (as defined by the firm''s managerial and peer ethical behaviors) may affect the employees'' comfort levels and ethical behaviors. In this research, scenarios were used to describe advertising firms with various ethical cultures. Respondents'' perceived comfort levels in working for the firms described in the scenarios and the respondents'' behavioral intentions when faced with various advertising situations were assessed. Results of the study indicate that peer ethical behavior exerts a strong influence on the (...) comfort or discomfort level and the ethical behavioral intentions of potential advertising employees. Further, the strong influence exerted by peers seems to transcend the ethical behavior of the manager and carry over to the attitude toward the entire corporate advertising environment. This study provides insights for firms and researchers interested in assessing the impact of an advertising firm''s ethical culture on potential employees. (shrink)

This study aims to evaluate the impact of an educational intervention on social competence and social acceptance among adolescents. The participants were 106 adolescents aged 12-15 years (M = 13.41 years; SD = 0.81 years). Participants were randomly assigned to the control group (n = 44) and an experimental group (n = 69). In the experimental group, an intervention based on the Sport Education Model (SEM) was applied. While in the control group, an intervention based on the Traditional Model of (...) Direct Instruction (TM-DI) was carried out. An experimental design with repeated pretest and posttest measurements was developed. The Adolescent Multidimensional Social Competence Questionnaire (AMSC-Q) was used to assess social competence. The Guess Who (GW4) questionnaire was used to assess social acceptance (SA) among peers. The preliminary results obtained in this investigation showed that the intervention based on the SEM (experimental group) promoted greater improvements in some indicators of social competence and social acceptance among peers than those obtained with the TM-DI (control group). Furthermore, the results confirm a similar impact of the intervention between boys and girls. It is also relevant to note that these findings highlight the efficacy, feasibility, and suitability of this educational intervention with adolescents. (shrink)

Punctuated by joy, disappointments, and conflicts, research evaluation constitutes an intense, emotional moment in scientific life. Yet reviewers and research institutions often expect evaluations to be conducted objectively and dispassionately. Inspired by the scholarship describing the role of emotions in scientific practices, we argue instead, that reviewers actively define, display and manage their emotions in response to the structural organization of research evaluation. Our article examines reviewing practices used in the European Research Council’s (ERC) Starting and Consolidator grants (...) and in the Marie Skłodowska-Curie Action’s (MSCA) Individual Fellowships. These two European funding mechanisms offer different perspectives on the organization of grant evaluation. We conducted interviews with review panel members and analyzed various institutional documents. By drawing on the sociological concepts of feeling rules and emotional work, we demonstrate that reviewers define rules concerning how emotions should be experienced and expressed to ensure the proper functioning of evaluation, and that reviewers experience the need to actively regulate their emotions to comply with these rules. We present four feeling rules concerning the experience and expression of: (1) excitement for novelty during individual evaluation; (2) respect for others’ opinions and the absence of anger in review panels; (3) attentiveness and interest, which are seen as missing in online evaluations. Reviewers also expect ERC candidates to (4) avoid pride and manifest modesty during interviews. These rules demonstrate that proposal peer review is governed by emotional norms, and show the influence of organizational settings and moral requirements on research evaluation. (shrink)

Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available evidence, they are either positive (...) or neutral. We conclude that on present evidence abolishing peer review weakly dominates the status quo. (shrink)

Technology development and innovation are fundamentally different from scientific research. However, in many circumstances, they are evaluated jointly and by the same processes. In these cases, peer review—the most usual procedure for evaluating research—is also applied to the evaluation of technological products and innovation activities. This can lead to unfair results and end up discouraging the involvement of researchers in these fields. This paper analyzes the evaluation processes in Uruguay's National System of Researchers. In this system, all (...) members' activities, both scientific and technological, are evaluated by peer committees. Based on documentary analysis and semi-structured interviews, the difficulties faced by evaluators in assessing technology products are explored. The article highlights the persistence of a linear conception of the link between science and technology and describes the obstacles to assimilate the particularities of technological activities. Refereed publications are presented as the only uncontested product. Other types of output are reviewed with suspicion. This study emphasizes the need for specific mechanisms to evaluate technological production within academic careers. (shrink)

Rooted in the heart of Vygotsky’s Sociocultural Theory, mediation has recently received considerable attention in the field of TEFL. The existing literature suggests that mediation can play an essential role in language learners’ performance development. In addition, learners need to know about their thinking process which is interpreted as metacognition. This study aimed to investigate the effect of teacher- and peer-assisted evaluative mediation on learners’ metacognitive awareness development. To this end, 40 homogenized intermediate EFL learners were selected using a (...) test of English language proficiency. The participants were then randomly divided into teacher-assisted (n=20) and peer-assisted (n=20) groups. Before the instructional phase, a metacognitive awareness questionnaire was given to the participants. At the instructional phase, the learners in the teacher-assisted group received teacher-assisted mediation. The learners in the peer-assisted group, however, were exposed to mediation provided by their peers. After the instructional phase, the metacognitive awareness questionnaire was given to the participants as the posttest of the study. SPSS was used to analyze the data. The descriptive statistics, the Shapiro-Wilk test of normality, and the Paired Sample T-test for inferential statistics were used to analyze the data. The findings showed that peer-assisted evaluative mediation had positive effects on developing students’ level of metacognitive awareness, whereas teacher-assisted evaluative mediation did not reveal such effects. It could be concluded that peer-assisted evaluative mediation is an effective tool for improving students’ metacognitive awareness. (shrink)

This study concerns the problem of late complications of antineo-plastic therapy. Reduced parameters of the cardiorespiratory system in childhood may have a tremendous impact on health. In order to assess the selected parameters, to evaluate physical endurance, and compare the results with those obtained for healthy children, a test was carried out on a treadmill, until 80% of maximum pulse rate was reached. To compare the differences between the treatment group and the control group, three approaches were used. The first (...) one was the classical statistical inference, the second consisted in forming a multidimensional normal model and also involved modelling of the correlation between variables. The unstructured type of the working correlation matrix was chosen to obtain the results and correct standard errors. In the last approach, logistic regression was used to model the relationship between binary outcome and covariates, and to differentiate between the groups of patients on the basis of their cardiovascular parameters. (shrink)

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.

Peer review process helps in evaluating and validating of research that is published in the journals. U.S. Office of Research Integrity reported that data fraudulence was found to be involved in 94% cases of misconduct from 228 identified articles between 1994–2012. If fraud in published article are significantly as high as reported, the question arise in mind, were these articles peer reviewed? Another report said that the reviewers failed to detect 16 cases of fabricated article of Jan Hendrick (...) Schon. Superficial peer reviewing process does not reveals suspicion of misconduct. Lack of knowledge of systemic review process not only demolish the academic integrity in publication but also loss the trust of the people of the institution, the nation, and the world. The aim of this review article is to aware stakeholders specially novice reviewers about the peer review system. Beginners will understand how to review an article and they can justify better action choices in dealing with reviewing an article. (shrink)

Problems in peer review, the backbone of maintaining high standards in scientific publishing, have led to wide spread discontent within the scientific community. Training in the peer review process and a simpler format to assist in decision making are possible courses to improve and expedite the process of peer review and scientific publishing. The authors discuss problems in the peer review process focusing on challenges related to major revisions and reviewer's wish list of experiments; this leads (...) to the loss of time and money. Major revisions rarely lead to significant improvements, while simultaneously delay the dissemination of knowledge. The authors propose simple solutions including training peer reviewers to improve the peer review system. (shrink)

Considerable attention has focused on studying reviewer agreement via inter-rater reliability (IRR) as a way to assess the quality of the peer review process. Inspired by a recent study that reported an IRR of zero in the mock peer review of top-quality grant proposals, we use real data from a complete range of submissions to the National Institutes of Health and to the American Institute of Biological Sciences to bring awareness to two important issues with using IRR for (...) assessing peer review quality. First, we demonstrate that estimating local IRR from subsets of restricted-quality proposals will likely result in zero estimates under many scenarios. In both data sets, we find that zero local IRR estimates are more likely when subsets of top-quality proposals rather than bottom-quality proposals are considered. However, zero estimates from range-restricted data should not be interpreted as indicating arbitrariness in peer review. On the contrary, despite different scoring scales used by the two agencies, when complete ranges of proposals are considered, IRR estimates are above 0.6 which indicates good reviewer agreement. Furthermore, we demonstrate that, with a small number of reviewers per proposal, zero estimates of IRR are possible even when the true value is not zero. (shrink)

An important function in all scholarly and academic activities is the participation in the peer review system. One aspect of this peer review evaluation is service on committees judging candidates for important awards, prizes and fellowships. Some reflective observations on this process are made in which a number of factors determining the final choice are identified. It is pointed out that the decisions of such committees are based not only on relevant and objective criteria but are also (...) influenced by a number of irrelevant criteria; caution must be exercised to minimize the effect of the latter factors in order to maintain the highest ethical standards in the selection process. The ranking practices of awards committees, national academies and learned societies are briefly reviewed by pointing out some ethical pitfalls, anecdotal incidents, and the ways to avoid the tarnishing of the selection process. (shrink)

At present, peer review is the most common method used by funding agencies to make decisions about resource allocation. But how reliable, efficient, and fair is it in practice? The ex ante evaluation of scientific novelty is a fundamentally uncertain endeavor; bias and chance are embedded in the final outcome. In the current study, I will examine some of the most central problems of peer review and highlight the possible benefits of using a lottery as an alternative (...) decision-making mechanism. Lotteries are driven by chance, not reason. The argument made in the study is that the epistemic landscape could benefit in several respects by using a lottery, thus avoiding all types of bias, disagreement, and other limitations associated with the peer review process. Funding agencies could form a pool of funding applicants who have minimal qualification levels and then select randomly within that pool. The benefits of a lottery would not only be that it saves time and resources, but also that it contributes to a more dynamic selection process and increases the epistemic diversity, fairness, and impartiality within academia. (shrink)

Peer review is a widely accepted instrument for raising the quality of science. Peer review limits the enormous unstructured influx of information and the sheer amount of dubious data, which in its absence would plunge science into chaos. In particular, peer review offers the benefit of eliminating papers that suffer from poor craftsmanship or methodological shortcomings, especially in the experimental sciences. However, we believe that peer review is not always appropriate for the evaluation of controversial (...) hypothetical science. We argue that the process of peer review can be prone to bias towards ideas that affirm the prior convictions of reviewers and against innovation and radical new ideas. Innovative hypotheses are thus highly vulnerable to being “filtered out” or made to accord with conventional wisdom by the peer review process. Consequently, having introduced peer review, the Elsevier journal Medical Hypotheses may be unable to continue its tradition as a radical journal allowing discussion of improbable or unconventional ideas. Hence we conclude by asking the publisher to consider re-introducing the system of editorial review to Medical Hypotheses. (shrink)

Previous developmental research shows that young children display a preference for ingroup members when it comes to who they accept information from – even when that information is false. However, it is not clear how this ingroup bias develops into adolescence, and how it affects responses about peers who misinform in intergroup contexts, which is important to explore with growing numbers of young people on online platforms. Given that the developmental span from childhood to adolescence is when social groups and (...) group norms are particularly important, the present study took a Social Reasoning Developmental Approach. This study explored whether children and adolescents respond differently to a misinformer spreading false claims about a peer breaking COVID-19 rules, depending on the group membership of the misinformer and their target and whether the ingroup had a “critical” norm that values questioning information before believing it. 354 United Kingdom-based children and adolescents read about an intergroup scenario in which a peer spreads misinformation on WhatsApp about a competitor. Participants first made moral evaluations, which asked them to judge and decide whether or not to include the misinformer, with follow-up “Why?” questions to capture their reasoning. This was followed by asking them to attribute intentions to the misinformer. Results showed that ingroup preferences emerged both when participants morally evaluated the misinformer, and when they justified those responses. Participants were more likely to evaluate an ingroup compared to an outgroup misinformer positively, and more likely to accuse an outgroup misinformer of dishonesty. Adolescents attributed more positive intentions to the misinformer compared with children, with children more likely to believe an outgroup misinformer was deliberately misinforming. The critical norm condition resulted in children making more positive intentionality attributions toward an ingroup misinformer, but not an outgroup misinformer. This study’s findings highlight the importance of shared group identity with a misinformer when morally evaluating and reasoning about their actions, and the key role age plays in intentionality attributions surrounding a misinformer when their intentions are ambiguous. (shrink)

To arrive at their final evaluation of a manuscript or grant proposal, reviewers must convert a submission’s strengths and weaknesses for heterogeneous peer review criteria into a single metric of quality or merit. I identify this process of commensuration as the locus for a new kind of peer review bias. Commensuration bias illuminates how the systematic prioritization of some peer review criteria over others permits and facilitates problematic patterns of publication and funding in science. Commensuration bias (...) also foregrounds a range of structural strategies for realigning peer review practices and institutions with the aims of science. (shrink)

Peer review is a practice used in the evaluation of scientific and scholarly papers in order to select papers for publication in scholarly journals. The practice has also been extended to other domains, such as the evaluation of grant proposals, medical practice, book publication, and even to such areas as teaching evaluation. The primary area that has been of interest to sociologists, however, has been publication in scientific journals. The practice is usually understood to have begun (...) in the seventeenth century in the Royal Society in London, but it has also been claimed that there have been precursors to this practice. In the popular mind, peer review in science is a means of “bullet‐proofing” research, as a Wall Street Journal article once put it, that is to say as a guarantee of quality. But within science it has a very different meaning. (shrink)

The demand for greater public accountability is changing the nature of ex ante peer review at public science agencies worldwide. Based on a four year research project, this essay examines these changes through an analysis of the process of grant proposal review at two US public science agencies, the National Institutes of Health (NIH) and the National Science Foundation (NSF). Weaving historical and conceptual narratives with analytical accounts, we describe the ways in which these two agencies struggle with the (...) question of incorporating considerations of societal impact into the process of peer review. We use this comparative analysis to draw two main conclusions. First, evaluation of broader societal impacts is not different in kind from evaluation of intellectual merit. Second, the scientific community may actually bolster its autonomy by taking a broader range of considerations into its peer review processes. (shrink)

Anyone who labors at academic scholarship knows vividly—perhaps even painfully—how dependent that enterprise is on a system of peer review. A scholar submits a work to a journal, press, or conference committee, or sends a proposal to a foundation; the submission is then evaluated by other professionals. The judgment of these referees determines whether the work is published by the target journal or press, appears on the conference program, or is funded by the desired institution. In many fields the (...) overwhelming majority of submissions are rejected. Careers are thus often made or destroyed by the process. (shrink)

This article theorizes and analyzes how two aspects of the increasing accountingization of academia in the form of evaluative pressures and group identification, independently and interactively, work to cultivate academics’ self-interest for their social interactions with the scientific community, forming them to adopt more competitive orientations and cynical attitudes. Using data of a large number of faculty members from the 17 universities in Sweden, it is shown that evaluative pressures and group identification perceived by academics jointly reinforce each other in (...) affecting their competitive orientation, and that group identification strengthens the positive relation between evaluative pressures and academics’ rivalry notions and cynical attitudes. It is shown, contributing further to research on performance evaluation and the cultivation of self-interest and an egoistic ethical climate in academia, that it is evaluative pressures from peers rather than from performance measurements that are the major driver of an individual’s competitive orientation and cynical attitudes. It is also concluded that while evaluative pressures are related to an increase in academics' competitive orientations, which may be viewed as an intended effect from control designers in universities, such an orientation is inversely related to cooperativeness and openness toward others and goes hand in hand with an increase in having cynical attitudes about peers and the work environment. Control designers in universities may thus not be able to have the one without the other, something that raises ethical concerns for academic leaders to reflect upon when aiming at cultivating self-interest orientations of academics. (shrink)

A peer instruction model was used whereby 78 residence dons (36 males, 42 females) provided instruction regarding academic integrity for 324 students (125 males, 196 females) under their supervision. Quantitative and qualitative analyses were conducted to assess survey responses from both the dons and students regarding presentation content, quality, and learning. Overall, dons consistently identified information-based slides about academic integrity as the most important material for the presentations, indicating that fundamental information was needed. Although student ratings of the usefulness (...) of the presentations were middling, students did indicate knowledge gains. Both interest and personal value for academic integrity were highly predictive of positive evaluations of the presentations. Dons and students provided suggestions for improvement and identified more global concerns. (shrink)

Introduction: Schools have become a primary setting for providing mental health care to youths in the U.S. School-based interventions have proliferated, but their effects on mental health and academic outcomes remain understudied. In this study we will implement and evaluate the effects of a flexible multidiagnostic treatment called Modular Approach to Therapy for Children with Anxiety, Depression, Trauma, or Conduct Problems on students' mental health and academic outcomes.Methods and Analysis: This is an assessor-blind randomized controlled effectiveness trial conducted across five (...) school districts. School clinicians are randomized to either MATCH or usual care treatment conditions. The target sample includes 168 youths referred for mental health services and presenting with elevated symptoms of anxiety, depression, trauma, and/or conduct problems. Clinicians randomly assigned to MATCH or UC treat the youths who are assigned to them through normal school referral procedures. The project will evaluate the effectiveness of MATCH compared to UC on youths' mental health and school related outcomes and assess whether changes in school outcomes are mediated by changes in youth mental health.Ethics and Dissemination: This study was approved by the Harvard University Institutional Review Board. We plan to publish the findings in peer-reviewed journals and present them at academic conferences.Clinical Trial Registration:ClinicalTrials.gov ID: NCT02877875. Registered on August 24, 2016. (shrink)

Recent science funding policy scholars and practitioners have advocated for the use of lotteries, or elements of random chance, as supplementations of traditional peer review for evaluating grant applications. One of the primary motivations for lotteries is their purported openness to innovative research. The purpose of this paper is to argue that current proponents of funding science by lottery overestimate the viability of peer review and thus unduly restrict the scope of lotteries in science funding practice. I further (...) show how this analysis suggests a different way of introducing lotteries into science funding policy. (shrink)

Mission-oriented climate change research is often unverifiable. Therefore, many stakeholders look to peer-reviewed climate change research for trustworthy information about deeply uncertain and impactful phenomena. This is because peer-review signals that research has been vetted for scientific standards like reproducibility and replicability. Here we evaluate the transparency of research methodologies in mission-oriented computational climate research. We find that only five percent of our sample meets the minimal standard of fully open data and code required for reproducibility and replicability. (...) The widespread lack of minimal openness standards in peer-reviewed climate change research jeopardizes scientific credibility and risks poor societal outcomes as we manage growing climate risks. The scientific community can and must do better. We, the authors, outline necessary openness standards that we commit to as researchers, reviewers, and readers. We invite you to join us. (shrink)

Warning behaviour prior to an act of severe targeted school violence was often not recognized by peers and school staff. With regard to preventive efforts, we attempted to identify barriers to information exchange in German schools, and understand mechanisms that influenced the recognition, evaluation, and reporting of warning behaviour through a teacher or peer. Our analysis is based on inquiry files from eleven cases of German school shootings that were obtained during the three-year research project “Incident and case (...) analysis of highly expressive targeted violence (TARGET)". We conducted a qualitative retrospective case study to analyze witness reports from school staff and peers. Our results point to subjective explanations used by teachers and peers towards conspicuous behaviour (e.g., situational framing, typical adolescent behaviour), as well as reassuring factors that indicated harmlessness (e.g., no access to a weapon). Additionally, we found organizational barriers similar to those described in US-American case studies (e.g., organizational deviance). (shrink)

Psychometrically oriented researchers construe low inter-rater reliability measures for expert peer reviewers as damning for the practice of peer review. I argue that this perspective overlooks different forms of normatively appropriate disagreement among reviewers. Of special interest are Kuhnian questions about the extent to which variance in reviewer ratings can be accounted for by normatively appropriate disagreements about how to interpret and apply evaluative criteria within disciplines during times of normal science. Until these empirical-cum-philosophical analyses are done, it (...) will remain unclear the extent to which low inter-rater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers. (shrink)

Previous research has found that funding disparities are driven by applications’ final impact scores and that only a portion of the black/white funding gap can be explained by bibliometrics and topic choice. Using National Institutes of Health R01 applications for council years 2014–2016, we examine assigned reviewers’ preliminary overall impact and criterion scores to evaluate whether racial disparities in impact scores can be explained by application and applicant characteristics. We hypothesize that differences in commensuration—the process of combining criterion scores into (...) overall impact scores—disadvantage black applicants. Using multilevel models and matching on key variables including career stage, gender, and area of science, we find little evidence for racial disparities emerging in the process of combining preliminary criterion scores into preliminary overall impact scores. Instead, preliminary criterion scores fully account for racial disparities—yet do not explain all of the variability—in preliminary overall impact scores. (shrink)

Reading difficulties that are not addressed at the primary level continue to exist at the secondary level with serious consequences. Thus, it is important to provide struggling students with specific reading support. In particular, many students with learning disabilities and emotional behavioral disorders demonstrate reading obstacles and are at risk for motivation loss. A multiple baseline design was used to evaluate the effects of a motivational reading racetrack as peer-tutoring on the word reading skills of secondary students with LD (...) with and without EBD. The intervention was conducted through 4–5 baseline and 16–18 reading units three times a week for 15 min over 8 weeks. The results showed positive effects indicating a highly effective treatment. In addition, follow-up results were also promising. Our findings indicate that this multicomponent intervention has a positive effect on the word fluidity of low-achieving students in secondary education with LD and/or EBD. (shrink)

Epistemological differences fuel continuous and frequently divisive debates in the social sciences and the humanities. Sociologists have yet to consider how such differences affect peer evaluation. The empirical literature has studied distributive fairness, but neglected how epistemological differences affect perception of fairness in decision making. The normative literature suggests that evaluators should overcome their epistemological differences by ‘‘translating’’ their preferred standards into general criteria of evaluation. However, little is known about how procedural fairness actually operates. Drawing on (...) eighty-one interviews with panelists serving on five multidisciplinary fellowship competitions in the social sciences and the humanities, we show that Evaluators generally draw on four epistemological styles to make arguments in favor of and against proposals. These are the constructivist, comprehensive, positivist, and utilitarian styles; and Peer reviewers define a fair decision-making process as one in which panelists engage in ‘‘cognitive contextualization,’’ that is, use epistemological styles most appropriate to the field or discipline of the proposal under review. (shrink)

Increasingly, the high throughput technologies used by biomedical researchers are bringing about a situation in which large bodies of data are being described using controlled structured vocabularies—also known as ontologies—in order to support the integration and analysis of this data. Annotation of data by means of ontologies is already contributing in significant ways to the cumulation of scientific knowledge and, prospectively, to the applicability of cross-domain algorithmic reasoning in support of scientific advance. This very success, however, has led to a (...) proliferation of ontologies of varying scope and quality. We define one strategy for achieving quality assurance of ontologies—a plan of action already adopted by a large community of collaborating ontologists—which consists in subjecting ontologies to a process of peer review analogous to that which is applied to scientific journal articles. (shrink)

The principles and practices of research peer review are described. While the principles are fundamentally generic and apply to peer review across the full spectrum of performing institutions as well as to manuscript/proposal/program peer review, the focus of this paper is peer review of proposed and ongoing programs in federal agencies. The paper describes desireable characteristics and important intangible factors in successful peer review. Also presented is a heuristic protocol for the conduct of successful (...) class='Hi'>peer review research evaluations and impact assessments. Problems with peer review are then outlined, followed by examples of peer review of proposed and existing programs in selected federal agencies. Some peer review variants, such as the Science Court, are described, and then research requirements to improve peer review are discussed. (shrink)

While some form of evaluation has always been employed in science (e.g. peer review, hiring), formal systems of evaluation of research and researchers have recently come to play a more prominent role in many countries because of the adoption of new models of governance. According to such models, the quality of the output of both researchers and their institutions is measured, and issues such as eligibility for tenure or the allocation of public funding to research institutions crucially (...) depends on the outcomes of such measures. However, concerns have been raised over the risk that such evaluation may be threatening epistemic pluralism by penalizing the existent heterodox schools of thought and discouraging the pursuit of new ones. It has been proposed that this may happen because of epistemic bias favouring mainstream research programmes. In this paper, I claim that (1) epistemic pluralism is desirable and should be preserved; (2) formal evaluation exercises may threaten epistemic pluralism because they may be affected by some form of epistemic bias; therefore, (3) to preserve epistemic pluralism, we need some strategy to actively dampen epistemic bias. (shrink)

BackgroundThere is increasing need for peer reviewers as the scientific literature grows. Formal education in biostatistics and research methodology during residency training is lacking. In this pilot study, we addressed these issues by evaluating a novel method of teaching residents about biostatistics and research methodology using peer review of standardized manuscripts. We hypothesized that mentored peer review would improve resident knowledge and perception of these concepts more than non-mentored peer review, while improving review quality.MethodsA partially blinded, (...) randomized, controlled multi-center study was performed. Seventy-eight neurology residents from nine US neurology programs were randomized to receive mentoring from a local faculty member or not. Within a year, residents reviewed a baseline manuscript and four subsequent manuscripts, all with introduced errors designed to teach fundamental review concepts. In the mentored group, mentors discussed completed reviews with residents. Primary outcome measure was change in knowledge score between pre- and post-tests, measuring epidemiology and biostatistics knowledge. Secondary outcome measures included level of confidence in the use and interpretation of statistical concepts before and after intervention, and RQI score for baseline and final manuscripts.ResultsSixty-four residents (82%) completed initial review with gradual decline in completion on subsequent reviews. Change in primary outcome, the difference between pre- and post-test knowledge scores, did not differ between mentored (−8.5%) and non-mentored (−13.9%) residents (p = 0.48). Significant differences in secondary outcomes (using 5-point Likert scale, 5 = strongly agree) included mentored residents reporting enhanced understanding of research methodology (3.69 vs 2.61; p = 0.001), understanding of manuscripts (3.73 vs 2.87; p = 0.006), and application of study results to clinical practice (3.65 vs 2.78; p = 0.005) compared to non-mentored residents. There was no difference between groups in level of interest in peer review (3.00 vs 3.09; p = 0.72) or the quality of manuscript review assessed by the Review Quality Instrument (RQI) (3.25 vs 3.06; p = 0.50).ConclusionsWe used mentored peer review of standardized manuscripts to teach biostatistics and research methodology and introduce the peer review process to residents. Though knowledge level did not change, mentored residents had enhanced perception in their abilities to understand research methodology and manuscripts and apply study results to clinical practice. (shrink)