This paper explores the issue of epistemic injustice in research evaluation. Through an analysis of the disciplinary cultures of physics and humanities, we attempt to identify some aims and values specific to the disciplinary areas. We suggest that credibility is at stake when the cultural values and goals of a discipline contradict those presupposed by official evaluation standards. Disciplines that are better aligned with the epistemic assumptions of evaluation standards appear to produce more "scientific" findings. To (...) restore epistemic justice in research evaluation, we argue that the specificity of a discipline's epistemic aims, values, and cultural identities must be taken into account. (shrink)

In the 1990s, China created a research evaluation system based on publications indexed in the Science Citation Index (SCI) and on the Journal Impact Factor. Such system helped the country become the largest contributor to the scientific literature and increased the position of Chinese universities in international rankings. Although the system had been criticized by many because of its adverse effects, the policy reform for research evaluation crawled until the breakout of the COVID-19 pandemic, which accidently (...) accelerates the process of policy reform. This paper highlights the background and principles of this reform, provides evidence of its effects, and discusses the implications for global science. (shrink)

This article explores issues in the evaluation of research through an examination of the situation in the field of international development. It is increasingly recognized that traditional evaluation, which served largely a policing function, is not useful in assessing the impact of the development research process. It is argued that the role and perception of evaluation must change if it is to provide a reflection of the learning which takes place in research. The field (...) of international development research provides a rich ground in which to explore the argument. The field is subject to the difficulties of evaluating research and to the difficulties of evaluating development—in both cases, we are dealing with long time frames and the influence of many factors other than the activity itself. If evaluation is to serve a learning function, then the shift must be to a purpose ofempowerment. This article explores the changes which are implicit in such a shift through an examination of three key areas: changing perspectives on evaluation; changing patterns in development; and changing patterns in research for development. (shrink)

This paper addresses some critical issues in the applicability of quantitative performance measures (including bibliometric, economic, and co-occurrence measures) to the assessment of basic research. The strengths and weaknesses of metrics applied as research performance measures are examined. It is concluded that metrics have a useful role to play in the evaluation of research. Each metric employed, whether bibliometric, economic, co-occurrence, or others, brings a new dimension of potential insight to the complex problem of research (...) assessment. However, when used in a stand-alone mode, metrics can be easily misinterpreted and result in misleading conclusions. Metrics should be an integral part of a more comprehensive approach to research evaluation, in which the leading role is assumed by expert peer review. Under this structure, the reviewers must be broadly constituted, so that the technical correctness of the research approaches being taken (doing the job right) can be evaluated by technical experts and the correctness of the research target (doing the right job) can be evaluated as well. This comprehensive approach insures quality in the implementation of existing research paradigms and allows the introduction of revolutionary new research paradigms as well. (shrink)

The quality of doctoral-level chemistry (N=145), computer science (N=58), geoscience (N=91), mathematics (N=115), physics (N=123), and statistics/biostatistics (N=64) programs at United States universities was assessed, using 16 measures. These measures focused on variables related to: program size; characteristics of graduates; reputational factors (scholarly quality of faculty, effectiveness of programs in educating research scholars/scientists, improvement in program quality during the last 5 years); university library size; research support; and publication records. Chapter I discusses prior attempts to assess quality in (...) graduate education, development of the study plans, and the selection of disciplines and programs to be evaluated. Chapter II discusses the methodology used, focusing on each of the assessment measures. Chapters III to VIII present, respectively, findings from the analyses of the chemistry, computer science, geoscience, mathematics, physics, and statistics/biostatistics programs. Chapter IX includes a summary of results, correlations among measures, several additional analyses, and suggestions for future studies. Among the findings reported are those indicating that mathematics programs had, on the average, the largest number of faculty (N=33) in December 1980 followed closely by physics (N=28) and chemistry (N=23), and that 80 percent of computer science students had job commitments by graduation. (Survey instruments and supporting documentation are included in appendices.) (JN). (shrink)

The quality of doctoral-level biochemistry (N=139), botany (N=83), cellular/molecular biology (N=89), microbiology (N=134), physiology (N=101), and zoology (N=70) programs at United States universities was assessed, using 16 measures. These measures focused on variables related to: (1) program size; (2) characteristics of graduates; (3) reputational factors (scholarly quality of faculty, effectiveness of programs in educating research scholars/scientists, improvement in program quality during the last 5 years); (4) university library size; (5) research support; and (6) publication records. Chapter I discusses (...) prior attempts to assess quality in graduate education, development of the study plans, and the selection of disciplines and programs to be evaluated. Chapter II discusses the methodology used, focusing on each of the assessment measures. Chapters III to VIII present, respectively, findings from the analyses of the biochemistry, botany, cellular/molecular biology, microbiology, physiology, and zoology programs. Chapter IX includes a summary of results, correlations among measures, several additional analyses, and suggestions for future studies. Among the findings reported are those indicating that cellular/molecular biology programs had, on the average, the largest number of faculty and that students in cellular/molecular biology, biology, biochemistry, microbiology, and physiology received a relatively high fraction of financial support. (Survey instruments and supporting documentation are included in appendices.) (JN). (shrink)

Biomedical research on humans is an important part of medical progress. But, when lives are at risk, safety and ethical practices need to be the top priority. The need for the committees that regulate and oversee such research -- institutional review boards, or IRBs -- is growing. IRB members face difficult decisions every day. Evaluating the Science and Ethics of Research on Humans is a guide for new and veteran members of IRBs that will help them better (...) understand the issues involved and the tasks they will be required to perform. The most important purpose of an IRB is to protect the lives of human participants. For three major research areas -- drugs, medical devices, and genetic information -- Dennis J. Mazur shares the methods he has found useful in protecting human participants through the systematic review of scientific protocols and informed consent forms and through adherence to the federal regulations that apply. New members will gain understanding of how proposed research projects are to be reviewed from both scientific and ethical dimensions, how and when to ask key questions of principal investigators, how to work with principal investigators and research teams to ensure the best protection of human participants, and why to schedule regularly spaced reviews of a project that may have adverse outcomes. Based on Mazur's thirty years of research experience, this accessible and informative guide will give all IRB members the tools they need to protect human lives and facilitate the research process. (shrink)

This article advocates for the validity of Realist Evaluation (RE) as a manifestation of Critical Realism in evaluation research despite criticisms suggesting that the former disregards principles from Bhaskarian ontology. Specifically, I argue that critics overstate RE's philosophical actualism in their argument that its inclination towards technocratic knowledge impedes its scrutiny of stratified social systems. Notwithstanding its limitations in fully elucidating causal structural mechanisms in social inquiry, I argue that RE's research rationale can contribute to the (...) stages of explanatory research based on CR by bridging the comprehension of mechanisms linked to micro-interventions with exploring potential structural forces operating across levels of reality. To illustrate this point, I present a case of a policy evaluation study I led that demonstrates how a RE-oriented research design facilitated a transition from investigating specific programme mechanisms within the interventions' context to proposing transcendental questions about the governance of the education system in Colombia. (shrink)

Original research is of course what scientists are expected to do. Therefore the research project is in many ways the unit of science in the making: it is the center of the professional life of the individual scientist and his coworkers. It is also the means towards the culmination of their specific activities: the original publication they hope to contribute to the scientific literature. The scientific project should therefore be of central interest to all the students of science, (...) particularly the philosophers and sociologists of science. We shall focus on the preliminary evaluation of research projects—the specific task of referees—and will emphasize the problem of their scientificity—the chief concern of scientific gatekeepers. In the past such an examination aimed only at protecting the taxpayer from swindlers and incompetent amateurs, such as the inventors of continuous motion artifacts. In recent times a similar issue has resurfaced with regard to some of the most prestigious and most handsomely funded projects, namely work on string theory and many-worlds cosmology. Indeed, some of their faithful have claimed that these theories are so elegant, and so full of high-grade mathematics, that they should be exempted from empirical tests. This claim provoked the spirited rebuttal of the well-known cosmologists Ellis and Silk, which the present paper is intended to reinforce. Indeed, we shall try to show why empirical testability is necessary though insufficient for a piece of work to qualify as scientific. Finally, the present paper may also be regarded as an indirect contribution to the current debate over the reliability of quantitative indicators of scientific worth, such as the h-index of scientific productivity. But we shall touch only tangentially on the sociological, political, and economics of research teams: our focus will be the acquisition of new scientific knowledge. (shrink)

BackgroundThis study aims to quantitatively and qualitatively evaluate the activities of a Bioethics Unit (BU) 5 years since its implementation (2016–2020). The BU is a research unit providing empirical research on ethical issues related to clinical practice, clinical ethics consultation, and ethical education for health care professionals (HPS).MethodsWe performed an explanatory, sequential, mixed-method, observational study, using the subsequent qualitative data to explain the initial quantitative findings. Quantitative data were collected from an internal database and analyzed by descriptive analysis. (...) Qualitative evaluation was performed by semi-structured interviews with 18 HPs who were differently involved in the BU’s activities and analyzed by framework analysis.ResultsQuantitative results showed an extensive increment of the number of BU research projects over the years and the number of work collaborations with other units and wards. Qualitative findings revealed four main themes, concerning: 1. the reasons for contacting the BU and the type of collaboration; 2. the role of the bioethicist; 3. the impact of BU activities on HPs, in terms of developing deeper and more mature thinking; 4. the need to extend ethics support to other settings. Overall, our results showed that performing both empirical bioethics research and more traditional clinical ethics activities at the same unit would produce an impetus to increase collaboration and spread an 'ethical culture' among local HPs.ConclusionsOur findings contribute to a growing body of literature on the models of clinical ethics support services and the role of empirical research in bioethics internationally. They also prepare the ground for the implementation of a multidisciplinary Clinical Ethics Committee (CEC) that aims to support the BU’s ethics consultation service within the local context. (shrink)

While some form of evaluation has always been employed in science (e.g. peer review, hiring), formal systems of evaluation of research and researchers have recently come to play a more prominent role in many countries because of the adoption of new models of governance. According to such models, the quality of the output of both researchers and their institutions is measured, and issues such as eligibility for tenure or the allocation of public funding to research institutions (...) crucially depends on the outcomes of such measures. However, concerns have been raised over the risk that such evaluation may be threatening epistemic pluralism by penalizing the existent heterodox schools of thought and discouraging the pursuit of new ones. It has been proposed that this may happen because of epistemic bias favouring mainstream research programmes. In this paper, I claim that (1) epistemic pluralism is desirable and should be preserved; (2) formal evaluation exercises may threaten epistemic pluralism because they may be affected by some form of epistemic bias; therefore, (3) to preserve epistemic pluralism, we need some strategy to actively dampen epistemic bias. (shrink)

Richard Whitley, Jochen Gläser, The Changing Governance of the Sciences. The Advent of Research Evaluation Systems. Sociology of the Sciences Yearbook Content Type Journal Article Pages 465-468 DOI 10.1007/s11024-009-9132-4 Authors Jürgen Enders, University of Twente Enschede The Netherlands Journal Minerva Online ISSN 1573-1871 Print ISSN 0026-4695 Journal Volume Volume 47 Journal Issue Volume 47, Number 4.

This article addresses the question how philosophy should be evaluated in a research-grant funding environment. It offers a new conception of philosophy that is inclusive and builds on familiar elements of professional, philosophical practice. Philosophy systematically questions the questions we ask, the concepts we use, and the values we hold. Its product is therefore rarely conclusive but can be embodied in everything we do. This is typical of explorative research and differentiates it from exploitative research, which constitutes (...) the bulk of funded research activity. This article argues that exploratory research is crucial for long-term progress and requires a distinct evaluative regime. (shrink)

This article investigates whether or not theories of justice from political philosophy, first, support the position that health research should contribute to justice in global health, and second, provide guidance about what is owed by international clinical research (ICR) actors to parties in low- and middle-income countries. Four theories—John Rawls's theory of justice, the rights-based cosmopolitan theories of Thomas Pogge and Henry Shue, and Jennifer Ruger's health capability paradigm—are evaluated. The article shows that three of the four theories (...) require the conduct of health research for justice in global health. The theories help identify the ends of justice to which ICR is to contribute, but they cannot tell us how to organize ICR to promote these ends. Aside from Ruger's health capability paradigm, the theories also lack an allocative principle for assigning specific duties to specific actors. This creates difficulties for establishing obligations for certain types of ICR actors. (shrink)

The subject of journal quality has received little attention in the business ethics literature. While there are reasons for this past neglect, there are important new considerations which make it vital that researchers now address this topic. First, virtually all business school departments use evaluations of journal quality as an important indicator of scholarly achievement, yet business ethics has no such studies. Second, as many schools are beginning to ask ethicists to publish in the wider management literature, it is important (...) to find a way to compare journal quality across these domains. In the absence of such studies, performance review committees (including tenure decisions) may well make incorrect inferences about journal quality within business ethics or assume that ethics journals are not prestigious unless a candidate can provide evidence to the contrary (i.e. as in a study like this one). This study addresses these two issues. It relies on an opinion-based survey to establish journal quality by incorporating a list of journals within business ethics and the wider management literature. The target population is active business ethics researchers within the Society for Business Ethics (SBE). We present the results of the study and discuss their implications. (shrink)

While Specific Informed Consent has been the established standard for obtaining consent for medical research for many years, it does not appear suitable for large-scale biobank and health data research. Thus, alternative forms of consent have been suggested, based on a variety of ethical background assumptions. This article identifies five main ethical perspectives at stake. Even though Tiered Consent, Dynamic Consent and Meta Consent are designed to the demands of the self-determination perspective as well as the perspective of (...) research as a public good, they are still also criticized from both perspectives. In addition, criticisms based on concerns of justice, participation and democratic deliberation, and relational concerns have been levelled at each of the models. As all of these perspectives have valid points to make, the task at hand lies in balancing these ethical perspectives. What constitutes an adequate balancing depends on contextual factors. These factors include digital infrastructure and digital literacy, data safety regulation, good scientific and clinical practice, transparent debates on ethically relevant features of research, social inequalities, anti-discrimination laws and practices, trust in health care institutions and recognition of patient preferences, and consensus on unethical research. We argue that the role of context in determining acceptable models of consent puts the ethical importance of models of consent into perspective. Since altering contextual factors can help to live up to the ethical concerns at stake in debates about models of consent, opting for such a shift of focus comes without ethical loss. (shrink)

We are pleased to publish this WSIA edition of Trudy’s Govier’s seminal volume, Problems in Argument Analysis and Evaluation. Originally published in 1987 by Foris Publications, this was a pioneering work that played a major role in establishing argumentation theory as a discipline. Today, it is as relevant to the field as when it first appeared, with discussions of questions and issues that remain central to the study of argument. It has defined the main approaches to many of those (...) issues and guided the ways in which we might respond to them. From this foundation, it sets the stage for further investigations and emerging research. This is a second edition of the book that is corrected and updated by the author, with new prefaces to each chapter. (shrink)

In Australia research projects proposing the use of linked data require approval by a Human Research Ethics Committee . A sound evaluation of the ethical issues involved requires understanding of the basic mechanics of data linkage, the associated benefits and risks, and the legal context in which it occurs. The rapidly increasing number of research projects utilising linked data in Australia has led to an urgent need for enhanced capacity of HRECs to review research applications (...) involving this emerging research methodology. The training described in this article was designed to respond to an identified need among the data linkage units in the Australian Population Health Research Network and HREC members in Australia. (shrink)

BackgroundCommunity engagement in research has gained momentum as an approach to improving research, to helping ensure that community concerns are taken into account, and to informing ethical decision-making when research is conducted in contexts of vulnerability. However, guidelines and scholarship regarding community engagement are arguably unsettled, making it difficult to implement and evaluate.DiscussionWe describe normative guidelines on community engagement that have been offered by national and international bodies in the context of HIV-related research, which set the (...) stage for similar work in other health related research. Next, we review the scholarly literature regarding community engagement, outlining the diverse ethical goals ascribed to it. We then discuss practical guidelines that have been issued regarding community engagement. There is a lack of consensus regarding the ethical goals and approaches for community engagement, and an associated lack of indicators and metrics for evaluating success in achieving stated goals. To address these gaps we outline a framework for developing indicators for evaluating the contribution of community engagement to ethical goals in health research.SummaryThere is a critical need to enhance efforts in evaluating community engagement to ensure that the work on the ground reflects the intentions expressed in the guidelines, and to investigate the contribution of specific community engagement practices for making research responsive to community needs and concerns. Evaluation mechanisms should be built into community engagement practices to guide best practices in community engagement and their replication across diverse health research settings. (shrink)

In calling for the transformation of military medical education and training, the 2005 Base Realignment and Closure Commission recommended relocating basic and specialty enlisted medical training to a single site to take advantage of economies of scale and the opportunity for joint training. As a result, a joint medical education and training campus (METC) has been established at Fort Sam Houston, Texas. Two of METC's primary long-term goals are to become a high-performing learning organization and to seek accreditation as a (...) community college. Such goals require a clear model of organizational improvement with well-defined metrics for measuring its performance and using research and evaluation to assess and improve that performance. Lessons learned from a review of practices at institutions with similar missions -- such as community colleges, corporate universities, the UK's Defence Medical Education and Training Agency, and other federal agencies, such as the Veterans Health Administration -- establish a clear need for an office of institutional research to help METC attain its organizational goals. They also provide useful recommendations regarding the METC office's structure, scope, and governance. (shrink)

A Higher education system is closely related to a country’s development. Governments need to make appropriate adjustments to ensure the healthy and sustainable development of higher education. To measure and evaluate the development status of the national higher education system and propose a set of policies, the following work has been done in this paper: First, this paper establishes a comprehensive evaluation index system based on the AHP-FCE model. The healthy development of education in significant countries was compared horizontally (...) and vertically. The results of weighted scores were used to determine the Japanese higher education system as the research object. Secondly, this paper uses factor analysis and principal component analysis to evaluate the level of sustainable development of higher education, detecting development barriers that migrate from the current to the future. This paper proposes a healthy and sustainable development policy blueprint from four aspects. Finally, this article uses the ARMA time series model to predict the development trend of Japanese higher education in the next 30 years and formulates a timetable for corresponding implementation arrangements. In addition, this paper quantitatively evaluates the effectiveness of the policies we propose and their impact on reality and provides references for other countries in the world. (shrink)

This book examines very important issues in research evaluation in the Social Sciences and Humanities. It is based on recent experiences carried out in Italy in the fields of research assessment, peer review, journal classification, and construction of indicators, and presents a systematic review of theoretical issues influencing the evaluation of Social Sciences and Humanities. Several chapters analyse original data made available through research assessment exercises. Other chapters are the result of dedicated and independent (...) class='Hi'>research carried out in 2014-2015 aimed at addressing some of the debated and open issues, for example in the evaluation of books, the use of Library Catalog Analysis or Google Scholar, the definition of research quality criteria on internationalization, as well as opening the way to innovative indicators. The book is therefore a timely and important contribution to the international debate. (shrink)

Grand challenges stress the importance of multi-disciplinary research, a multi-actor approach in examining the current state of affairs and exploring possible solutions, multi-level governance and policy coordination across geographical boundaries and policy areas, and a policy environment for enabling change both in science and technology and in society. The special nature of grand challenges poses certain needs in evaluation practice: the need for learning at the operational, policy and, especially, system level; and the importance of a wider set (...) of impacts and behavioural change. The examination of the usefulness of evaluations as learning tools thus becomes relevant as does the way current evaluation practices address broader impacts and issues such as behavioural additionality. The suitability of existing evaluation contexts in meeting the specific issues posed by the ‘grand challenges’ orientation is also worth examining. The paper argues that learning at the policy and system levels is largely unaddressed while concepts such as behavioural additionality are still underexploited. (shrink)

The concept of public participation is one of growing interest in the UK and elsewhere, with a commensurate growth in mechanisms to enable this. The merits of participation, however, are difficult to ascertain, as there are relatively few cases in which the effectiveness of participation exercises have been studied in a structured manner. This seems to stem largely from uncertainty in the research community as to how to conduct evaluations. In this article, one agenda for conducting evaluation (...) class='Hi'>research that might lead to the systematic acquisition of knowledge is presented. This agenda identifies the importance of defining effectiveness and of operationalizing one’s definition. The article includes analysis of the nature of past evaluations, discussion of potential difficulties in the enactment of the proposed agenda, and discussion of some potential solutions. (shrink)

A team of philosophers and scientists at Dartmouth College worked for three years to create, train faculty and pilot test an adequate and exportable class in research methods for graduate students of science and engineering. Developing and testing methods for evaluating students’ progress in learning research ethics were part of the project goals. Failure of methods tried in the first year led to the refinement of methods for the second year. These were used successfully in the pilot course (...) and in one university setting external to Dartmouth. The process of development and justification for the final methods are discussed here. (shrink)

The new field of meta-research investigates industry bias, publication bias, contradictions between studies, and other trends in medical research. I argue that its findings should be used as meta-evidence for evaluating therapies. ‘Meta-evidence’ is evidence about the support that direct ‘first-order evidence’ provides the hypothesis. I consider three objections to my proposal: the irrelevance objection, the screening-off objection, and the underdetermination objection. I argue that meta-research evidence works by rationally revising our confidence in first-order evidence and, consequently, (...) in the hypothesis—typically, downward. (shrink)

Investigators and institutional review boards should integrate plans about the appropriate disclosure of individual genetic results when designing research studies. The ethical principles of beneficence, respect, reciprocity, and justice provide justification for routinely offering certain results to research participants. We propose a result-evaluation approach that assesses the expected information and the context of the study in order to decide whether results should be offered. According to this approach, the analytic validity and the clinical utility of a specific (...) result determine whether it should be offered routinely. Different results may therefore require different decisions even within the same study. We argue that the threshold of clinical utility for disclosing a result in a research study should be lower than the threshold used for clinical use of the same result. The personal meaning of a result provides additional criteria for evaluation. Finally, the context of the study allows for a more nuanced analysis by addressing the investigators' capabilities for appropriate disclosure, participants' alternative access to the result, and their relationship with the investigators. This analysis shows that the same result may require different decisions in different contexts. (shrink)

In order to fully implement the new development concept, bring into full play the potential of sports development, and maintain the resilience of China’s sports development. This paper studies the resilience evaluation and spatial correlation of Chinese sports development under the new development concept. First, we constructed Resilience Evaluation Indexes System for Sports Development in China based on the analysis of the resilience features of sports development and the DPSIR model, which is from the five aspects of “driving (...) force – pressure – state – influence – response.” Second, used Coefficient of Variation and Technique for Order Preference by Similarity to an Ideal Solution Method to measure the resilience level of sports development in 31 provinces in China from 2013 to 2017. Then, we introduced the obstacle degree model to identify the obstacle factors that hinder the resilience of Chinese sports development in different periods. Finally, we used the global and local Moran indexes to analyze the spatial correlation of China sports regional development. The results showed that: overall, the development level of sports resilience in 31 provinces in China showed an upward trend from 2013 to 2017, while some provinces showed obvious fluctuations. The obstacles to the development of sports resilience in China mainly include sports scientific research equipment, the number of national fitness monitoring stations, the number of national fitness centers, the full-time equivalent of personnel, and the number of sports scientific research projects. The response subsystem is the main obstacle factor that affects the improvement of the resilience level of sports development in China. There is a positive spatial autocorrelation between the resilience level of sports development and regional spatial distribution, and the correlation shows a weakening trend, and the internal difference is significant. Finally, we concluded that we must take the new development philosophy as the guiding principle. First, we should stick to innovation-driven development to fully upgrade the resilience of China’s sports development. Second, we should adhere to the principle of coordinated development to promote the overall and balanced development of sports. Lastly, we should promote shared development so as to deliver benefits for all in an equal way. (shrink)

Ethics designates a structured process by which important human values and meanings of life are understood and tackled. Therein, the ability to discuss openly and reflect on (aka deliberation) understandings of moral problems, on solutions to these problems, and to explore what a meaningful resolution could amount to is highly valued. However, the indicators of what constitutes a high-quality ethical deliberation remain vague and unclear. This article proposes and develops a pragmatist approach to evaluate the quality of deliberation. Deliberation features (...) three important moments: (1) broadening and deepening the understanding of the situation, (2) envisioning action scenarios, (3) coming to a judgment based on the comparative evaluation of scenarios. In this paper, we propose seven criteria to evaluate ethical deliberations: (1) collaborative diversity, (2) experiential literacy, (3) organization of experiences, (4) reflective capacity to instrumentalize the experiences of others, (5) interactional creativity, (6) openness of agents, (7) quality of the reformulation of scenarios. These criteria are explained and applied to the three moments of deliberation. Based on these criteria, three kinds of outcomes for deliberations are identified and discussed: good ethical deliberations, partial ethical deliberations, bad ethical deliberations. Our proposal will guide researchers and practitioners interested in the evaluation of the quality of ethical deliberations. It provides a reference tool that allows them to identify the possible limitations of a deliberation and to implement actions aimed at correcting these limitations in order to achieve the desired qualitative objectives. (shrink)

Investing in research is a long-term, risky proposition. In agriculture, it could take fifteen years or more for a research finding to show an improvement in a farmer’s field. Yet, research institutions, like other organizations it needs to be evaluated. For more than twenty years, independent panels of outside experts have evaluated each of the international research centers that the Consultative Group of International Agricultural Research (CGIAR) supports. This paper examines the evolution of this review (...) system, outlines the key methodological challenges faced, and draws lessons for others engaged in evaluating research institutions. It notes that the scope of the CGIAR reviews have been broadened over time in response to users’ concerns. Reviews now cover four dimensions of performance: research results, quality and relevance of research, vision and strategic directions, and management efficiency. The methodological challenges faced in measurement, valuation, and attribution are reviewed, along with practices found to be helpful in addressing these concerns. The paper concludes that the panel approach to institutional evaluation has served CGIAR’s needs well, and recommends it as an evaluation technique for integrating quantitative and qualitative dimensions of institutional performance. (shrink)

In current philosophical discussions on evidence in the medical sciences, epidemiology has been used to exemplify a specific version of evidential pluralism. According to this view, known as the Russo–Williamson Thesis, evidence of both difference-making and mechanisms is produced to make causal claims in the health sciences. In this paper, I present an analysis of data and evidence in epidemiological practice, with a special focus on research on the exposome, and I cast doubt on the extent to which evidential (...) pluralism holds in this case. I start by focusing on the claim that molecular data allows for the production of mechanistic evidence. On the basis of a close look at the ways in which molecular data is used in exposome research, I caution against interpretations in terms of mechanistic evidence. Secondly, I expand my critical remarks on the thesis by addressing the conditions under which data is categorised as evidence in exposome research. I argue that these show that the classification of a dataset as a type of evidence is dependent on the ways in which the data is used. This is in contrast with the approach of evidential pluralism, where evidence is classified in different types on the basis of its intrinsic properties. Finally, I come back to what I consider the core of the thesis and suggest that the epidemiological research analysed in the paper indicates different interpretations of evidential pluralism and its applicability in the health sciences. (shrink)

Many philosophers claim that interesting forms of epistemic evaluation are insensitive to truth in a very specific way. Suppose that two possible agents believe the same proposition based on the same evidence. Either both are justified or neither is; either both have good evidence for holding the belief or neither does. This does not change if, on this particular occasion, it turns out that only one of the two agents has a true belief. Epitomizing this line of thought are (...) thought experiments about radically deceived “brains in vats.” It is widely and uncritically assumed that such a brain is equally justified as its normally embodied human “twin.” This “parity” intuition is the heart of truth-insensitive theories of core epistemological properties such as justification and rationality. Rejecting the parity intuition is considered radical and revisionist. In this paper, I show that exactly the opposite is true. The parity intuition is idiosyncratic and widely rejected. A brain in a vat is not justified and has worse evidence than its normally embodied counterpart. On nearly every ordinary way of evaluating beliefs, a false belief is significantly inferior to a true belief. Of all the evaluations studied here, only blamelessness is truth-insensitive. (shrink)

This paper critically assesses the model of evaluation of scientific research for democratic societies defended by Philip Kitcher. The “significant truth” approach proposes a viable alternative to two classic images of science: that of the “critics”, who believe that science always serves the interests of the powerful and that of the “faithful”, who argue that the pursuit of scientific knowledge is always valuable and necessary. However, the democratic justification of Kitcher’s proposal is not compatible with the ethical problems (...) generated by the international biomedical research in developing countries. To overcome this problem, I revise the national democratic framing of the significant truth approach in light of the theory of justice of John Rawls. (shrink)

We have national guidelines for the responsible conduct of research (RCR) and procedures for handling allegations of misconduct in Finland. The guidelines have been formulated and updated by the Finnish Advisory Board on Research Integrity (TENK). In this article, we introduce and evaluate the national RCR guidelines. We also present statistics of alleged and proven RCR violation cases and frequency of appeals to TENK on the decisions or procedures of the primary institutions. In addition, we analyze the available (...) data on seven investigated cases in more detail. Positive aspects in the Finnish system are a fairly good infrastructure to investigate suspected RCR violations and a wide concept of RCR violations, which consists of fabrication, falsification, plagiarism, misappropriation, and other misbehaviors. However, the guidelines contain poorly elaborated definitions, do not treat the complainant and the suspect in an equal way, and need to be revised. Confusion about the concepts and criteria of the RCR violations seems to be common in primary institutions and among the complainants. Even if research institutions and universities have officially adhered to the national RCR guidelines, slipping from the guidelines occurs quite commonly. All these factors lead to frequent dissatisfaction with the decisions or procedures applied, high rate of appeals to TENK, and far from optimal functionality of the system. (shrink)

Background: The United States symbolizes democracy in the new world and contributes to global prosperity. Nevertheless, incrementalism is a historically dominant national approach to public policy implementation that delays democracy and undermines human dignity. Human flourishing and national development are endangered by slow-moving democratic changes. This necessitates a social justice framework that traces the exploitation of incrementalism and the consequences of opportunity gaps. Objectives: This study aims to construct a grounded theory to address and answer the following research question: (...) Are anti-democratic concepts part of a process or strategy that hinders democratic change, and what reinforces it, if anything? Methods: Two categories emerged from a grounded theory of democratic social change. This combination of anti-democratic tenets and democratic principles led to abductive reasoning to fill the research gap. Results: The key finding was the existence of a systematic approach that led to democracy while simultaneously disempowering the same group through legal and extralegal means. Two processes hinder democracy: incrementalism and an anti-democratic demoralizing process. A constructivist social justice grounded theory was developed to analyze and evaluate public policy for barriers and facilitators of democracy. A thorough analysis and evaluation of the impact and consequences of public policy is recommended to better understand how such policies impede democracy and influence American perceptions of justice. (shrink)

There is a growing call for greater public involvement in establishing science and technology policy, in line with democratic ideals. A variety of public participation procedures exist that aim to consult and involve the public, ranging from the public hearing to the consensus conference. Unfortunately, a general lack of empirical consideration of the quality of these methods arises from confusion as to the appropriate benchmarks for evaluation. Given that the quality of the output of any participation exercise is difficult (...) to determine, the authors suggest the need to consider which aspects of the process are desirable and then to measure the presence or quality of these process aspects. To this end, a number of theoretical evaluation criteria that are essential for effective public participation are specified. These comprise two types: acceptance criteria, which concern features of a method that make it acceptable to the wider public, and process criteria, which concern features of the process that are liable to ensure that it takes place in an effective manner. Future research needs to develop instruments to measure these criteria more precisely and identify the contextual and environmental factors that will mediate the effectiveness of the different participation methods. (shrink)

In a recent issue of Journal of Medical Ethics, we discussed the ethical review of evaluations of interventions that would occur whether or not the evaluation was taking place. We concluded that standard research ethics frameworks including the Ottawa Statement, which requires justification for all aspects of an intervention and its roll-out, were a poor guide in this area. We proposed that a consideration of researcher responsibility, based on the consequences of the research taking place, would be (...) a more appropriate way delineate the scope of research ethics review. Weijer and Taljaard present a counterargument to our proposal, which we address in this reply. They claim that a focus on researcher responsibility will weaken the protection of research participants and link it to ‘unethical research’ and a ‘government experimenting on its own people’. However, the moral responsibility of researchers is defined in terms of the consequences of the research on human welfare and harm, not in opposition to it. Weijer and Taljaard argue that researchers must justify what they are studying whether or not they have any control over it and that governments must justify their programmes, including by demonstrating equipoise, to a research ethics committee if they implement them in a randomised way. We strongly disagree that this is a defensible way to define the scope of research ethics review and argue that this provides no further protections to research participants beyond what we propose, but places a potential barrier to learning from government programmes. (shrink)

Appraisal theory has become one of the most active aproaches in the domain of emotion psychology. The appraisal process consists of the subjective evaluation that occurs during the individual's encounter with significant events in the environment, determining the nature of the emotional reaction and experience. The organism's interpretation of events and situations elicits and differentiates its emotional responses, although the exact processes involved and the limits of the theory are still a matter of debate and are currently the object (...) of active research. This volume is intended to become the primary source of information on appraisal for all those interested in emotion, from beginning graduate students to accomplished researchers in emotion psychology. (shrink)

Previous research in Explainable Artificial Intelligence (XAI) suggests that a main aim of explainability approaches is to satisfy specific interests, goals, expectations, needs, and demands regarding artificial systems (we call these “stakeholders' desiderata”) in a variety of contexts. However, the literature on XAI is vast, spreads out across multiple largely disconnected disciplines, and it often remains unclear how explainability approaches are supposed to achieve the goal of satisfying stakeholders' desiderata. This paper discusses the main classes of stakeholders calling for (...) explainability of artificial systems and reviews their desiderata. We provide a model that explicitly spells out the main concepts and relations necessary to consider and investigate when evaluating, adjusting, choosing, and developing explainability approaches that aim to satisfy stakeholders' desiderata. This model can serve researchers from the variety of different disciplines involved in XAI as a common ground. It emphasizes where there is interdisciplinary potential in the evaluation and the development of explainability approaches. (shrink)

An advertising firm''s ethical culture (as defined by the firm''s managerial and peer ethical behaviors) may affect the employees'' comfort levels and ethical behaviors. In this research, scenarios were used to describe advertising firms with various ethical cultures. Respondents'' perceived comfort levels in working for the firms described in the scenarios and the respondents'' behavioral intentions when faced with various advertising situations were assessed. Results of the study indicate that peer ethical behavior exerts a strong influence on the comfort (...) or discomfort level and the ethical behavioral intentions of potential advertising employees. Further, the strong influence exerted by peers seems to transcend the ethical behavior of the manager and carry over to the attitude toward the entire corporate advertising environment. This study provides insights for firms and researchers interested in assessing the impact of an advertising firm''s ethical culture on potential employees. (shrink)

Evaluation of clinical ethical case consultations has been discussed as an important research task in recent decades. A rigid framework of evaluation is essential to improve quality of consultations and, thus, quality of patient care. Different approaches to evaluate those services appropriately and to determine adequate empirical endpoints have been proposed. A key challenge is to provide an answer to the question as to which empirical endpoints—and for what reasons—should be considered when evaluating the quality of a (...) service. In this paper, we argue for an approach that adopts the role of ethics consultants as its point of departure. In a first step, we describe empirical and ethical characteristics of evaluating clinical ethical case. We show that the mode of action and the explicit normative character of the interventions constitute two characteristics which pose challenges to the selection of appropriate quality criteria and require special attention. In a second step, we outline the way in which an analysis of the role of ethics consultants in the context of a clinical ethical case consultation services can account for the existing challenges by linking empirically measurable endpoints with normative theory. Finally, we discuss practical implications of our model for evaluation research. (shrink)