Here we present the framework of a new approach to assessing the capacity of research programs to achieve social goals. Research evaluation has made great strides in addressing questions of scientific and economic impacts. It has largely avoided, however, a more important challenge: assessing (prospectively or retrospectively) the impacts of a given research endeavor on the non-scientific, non-economic goals—what we here term public values —that often are the core public rationale for the endeavor. Research programs are typically justified in terms (...) of their capacity to achieve public values, and that articulation of public values is pervasive in science policy-making. We outline the elements of a case-based approach to public value mapping of science policy, with a particular focus on developing useful criteria and methods for assessing public value failure, with an intent to provide an alternative to market failure thinking that has been so powerful in science policy-making. So long as research evaluation avoids the problem of public values, science policy decision makers will have little help from social science in making choices among competing paths to desired social outcomes. (shrink)

Scholars and policy-makers have expressed concerns about the crediting of coauthors in research publications. Most such problems fall into one of two categories, excluding deserving contributors or including undeserving ones. But our research shows that there is no consensus on “deserving” or on what type of contribution suffices for co-authorship award. Our study uses qualitative data, including interviews with 60 US academic science or engineering researchers in 14 disciplines in a set of geographically distributed research-intensive universities. We also employ data (...) from 161 website posts provided by 93 study participants, again US academic scientists. We examine a variety of factors related to perceived unwarranted exclusion from co-author credit and unwarranted inclusion, providing an empirically-informed conceptual model to explain co-author crediting outcomes. Determinants of outcomes include characteristics of disciplines and fields, institutional work culture, power dynamics and team-specific norms and decision processes. (shrink)

The question of what the relevant entities or units of analysis for studying the dynamics of R&D are is central not only for adequate characterizations of the system of scientific and technological knowledge production but also for determining the correct focus for evaluation of R&D activities. Typically, R&D performance evaluations have focused not only on the wrong thing but have looked in the wrong place. Most evaluations have been project or program based. Often this focus is misleading. This article presents (...) a “knowledge value” framework as an alternative focus for understanding and evaluating scientific and technical work. This framework consists of two core concepts: the Knowledge Value Collective and the Knowledge Value Alliance. On the basis of the analysis of twenty-eight case studies of research activities, the authors present a typology of KVAs and conclude that they are a better object of evaluation than discipline-based projects. (shrink)

We use the term “Zoomification” to refer to the primary mode of research collaboration used by academic researchers during much of the COVID-19 pandemic. While neither video-enabled technology or remote collaboration is new, the technology developments and needs that occurred during the pandemic proved exceptional, indeed a step-change in approaches to research collaboration. This study, based on in-depth interviews with 65 tenured and tenure track professors in dozens of United States universities in a wide variety of STEM disciplines, focuses on (...) collegial effects of Zoomification on research collaboration, including research with graduate students. We find diverse impacts according to career status, with younger faculty and doctoral students faring least well with an absence of face-to-face communication. As expected, impacts vary according to the nature of work, including the need to work in the field, reliance on laboratory equipment, laboratory animals, samples and high cost, centralized equipment. The effects of remote collaboration are to some extent predictable but there are results that point toward realignment in some of the ways research collaboration is established and maintained. We conclude with speculations about the long-range implications of Zoomification of research collaboration. (shrink)

We provide a critical assessment of the National Science Foundation's (NSF) “broader impacts criterion” for peer review, which has met with resistance from the scientific community and been characterized as unlikely to have much positive effect due to poor implementation and adherence to the linear model heuristic for innovation. In our view, the weakness of NSF's approach owes less to these issues than to the misguided assumption that the peer review process can be used to leverage more societal value from (...) research. This idea, although undoubtedly well-meaning, is fundamentally flawed. Retooling or refining the Broader Impacts Criterion does not alter the fact that conventional peer review, based on specialized scientific and technical expertise, is not up to the task of ensuring adequate judgements about social impact. We consider some possible alternative approaches to providing greater social impact in science and include in our assessment past and current efforts at NSF and throughout the federal research establishment that address, in some cases having addressed for decades, the intentions and goals of the Broader Impacts Criterion, albeit using alternate mechanisms. We conclude that institution-building and explicit and targeted policy-making are more useful and democratically legitimate approaches to ensuring broad social impacts. (shrink)

We provide a critical assessment of the National Science Foundation's “broader impacts criterion” for peer review, which has met with resistance from the scientific community and been characterized as unlikely to have much positive effect due to poor implementation and adherence to the linear model heuristic for innovation. In our view, the weakness of NSF's approach owes less to these issues than to the misguided assumption that the peer review process can be used to leverage more societal value from research. (...) This idea, although undoubtedly well-meaning, is fundamentally flawed. Retooling or refining the Broader Impacts Criterion does not alter the fact that conventional peer review, based on specialized scientific and technical expertise, is not up to the task of ensuring adequate judgements about social impact. We consider some possible alternative approaches to providing greater social impact in science and include in our assessment past and current efforts at NSF and throughout the federal research establishment that address, in some cases having addressed for decades, the intentions and goals of the Broader Impacts Criterion, albeit using alternate mechanisms. We conclude that institution-building and explicit and targeted policy-making are more useful and democratically legitimate approaches to ensuring broad social impacts. (shrink)

Based on results of an experiment, hypotheses are tested concerning the effects of computer use on decision commitment. The experiment required subjects to make an adoption decision regarding a hypothetical government agency's innovation. Subjects could choose from a variety of information sets, some computer based, some not, before making the decision. After their decision the subjects were given “new evidence” that contradicted their initial position. Two experimental treatments included more difficult access to the computer-based information and higher cost for the (...) computer-based information. Results indicate that access difficulty diminishes confidence in decisions and leads to lesser commitment. However, the cost of the computer information seems to have little bearing on decision commitment. (shrink)