There have recently been novel applications of medical systematic review guidelines to economic policy interventions which contain controversial methodological assumptions that require further scrutiny. A landmark 2017 Cochrane review of unconditional cash transfer (UCT) studies, based on the Grading of Recommendations Assessment, Development and Evaluation (GRADE), exemplifies both the possibilities and limitations of applying medical systematic review guidelines to UCT and universal basic income (UBI) studies. Recognizing the need to upgrade GRADE to incorporate the differences between medical and policy interventions, (...) the GRADE Public Health Project Group (PHPG) was convened to enumerate and address these methodological challenges. However, in light of our analysis of additional methodological challenges that arise for UCT and UBI studies, we argue that the adaptation of medical systematic review guidelines to economic methodology is far from straightforward and is in fact more challenging than claimed by the PHPG. (shrink)

Reactivity in qualitative data collection occurs when a researcher generates data about a situation with reactivity, that is, a situation in which the ongoing research affects the research participants such that they, say, diverge from their routines when the researcher is present, or tell the researcher what they think she wants to hear. In qualitative research, there are two basic approaches to reactivity. The traditional position maintains that data should ideally be collected in situations without any reactivity. In other words, (...) good data are reactivity free. By contrast, the more recent view holds that data from situations with reactivity are fine as long as the researcher is aware of the occurring reactivity so that she can take it into account when interpreting her data. In this fashion, good data are reactivity transparent. In this paper, I first spell out and defend the more recent approach to reactivity. I argue that qualitative data are reactivity transparent when conjoined with true reactivity assumptions and that, thus supplemented, data are informative about social life independently of its being studied. Next, I examine various issues raised by the requirement to put forth true reactivity assumptions. Lastly, I use my discussion of reactivity transparency as a basis for providing a framework for thinking about good qualitative data. (shrink)

Philosophers of science typically focus on the epistemic performance of scientific models when evaluating them. Analysing the effects that models may have on the world has typically been the purview of sociologists of science. We argue that the reactive (or “performative”) effects of models should also figure in model evaluations by philosophers of science. We provide a detailed analysis of how models in financial economics created the impetus for the growing importance of the phenomenon of “passive investing” in financial markets. (...) Considering this case motivates the position that we call contextualism about model evaluation, or _model contextualism_ for short. Model contextualism encompasses standard analyses of the epistemic performance of the model, but also includes their reactive aspects. It entails identifying the _epistemic and contextual import of the model_, the ways in which a model can engender change in the world (which we call the _channels of transmission_), and the _interactions between the epistemic and reactive import of a model_. (shrink)

While the history and methodology of laboratory experiments in economics have been extensively studied by philosophers, those of field experiments have not attracted much attention until recently. What is the historical context in which field experiments have been advocated? And what are the methodological rationales for conducting experiments in the field as opposed to in the lab? This article addresses these questions by combining historical and methodological perspectives. In terms of history, we show that the movement toward field experiments in (...) economics has two distinct roots. One is the general orientation of medical and social sciences to evidence-based policy evaluation, which gave rise to randomized field experiments in economics. The other is an awareness of several methodological limitations of lab experiments in economics, which required practitioners to get out of the lab and into the field. In these senses, the movement is a consequence of influences from both outside and inside economics: the general evidence-based trend in policy science and an internal methodological development of experimental economics. In terms of methodology, we show that these two roots resulted in two somewhat different notions of “external validity” as methodological rationales of field experiment. Finally, we suggest that analysis of experiments as exhibits highlights a methodological strategy in which both strands complement each other. (shrink)

While the term “reactivity” has come to be associated with specific phenomena in the social sciences, having to do with subjects’ awareness of being studied, this paper takes a broader stance on this concept. I argue that reactivity is a ubiquitous feature of the psychological subject matter and that this fact is a precondition of experimental research, while also posing potential problems for the experimenter. The latter are connected to the worry about distorted data and experimental artifacts. But what are (...) experimental artifacts and what is the most productive way of dealing with them? In this paper, I approach these questions by exploring the ways in which experimenters in psychology simultaneously exploit and suppress the reactivity of their subject matter in order to produce experimental data that speak to the question or subject matter at hand. Highlighting the artificiality of experimental data, I raise the question of what distinguishes a genuine experimental result from an experimental artifact. My analysis construes experimental results as the outcomes of inferences from the data that take material background assumptions as auxiliary premises. Artifacts occur when one or more of these background assumptions are false, such that the data do not reliably serve the purposes they were generated for. I conclude by laying out the ways in which my analysis of data quality is relevant to, and informed by, recent debates about the replicability of experimental results. (shrink)