There have recently been novel applications of medical systematic review guidelines to economic policy interventions which contain controversial methodological assumptions that require further scrutiny. A landmark 2017 Cochrane review of unconditional cash transfer (UCT) studies, based on the Grading of Recommendations Assessment, Development and Evaluation (GRADE), exemplifies both the possibilities and limitations of applying medical systematic review guidelines to UCT and universal basic income (UBI) studies. Recognizing the need to upgrade GRADE to incorporate the differences between medical and policy interventions, (...) the GRADE Public Health Project Group (PHPG) was convened to enumerate and address these methodological challenges. However, in light of our analysis of additional methodological challenges that arise for UCT and UBI studies, we argue that the adaptation of medical systematic review guidelines to economic methodology is far from straightforward and is in fact more challenging than claimed by the PHPG. (shrink)

Reactivity in qualitative data collection occurs when a researcher generates data about a situation with reactivity, that is, a situation in which the ongoing research affects the research participants such that they, say, diverge from their routines when the researcher is present, or tell the researcher what they think she wants to hear. In qualitative research, there are two basic approaches to reactivity. The traditional position maintains that data should ideally be collected in situations without any reactivity. In other words, (...) good data are reactivity free. By contrast, the more recent view holds that data from situations with reactivity are fine as long as the researcher is aware of the occurring reactivity so that she can take it into account when interpreting her data. In this fashion, good data are reactivity transparent. In this paper, I first spell out and defend the more recent approach to reactivity. I argue that qualitative data are reactivity transparent when conjoined with true reactivity assumptions and that, thus supplemented, data are informative about social life independently of its being studied. Next, I examine various issues raised by the requirement to put forth true reactivity assumptions. Lastly, I use my discussion of reactivity transparency as a basis for providing a framework for thinking about good qualitative data. (shrink)

Why scientists reach an agreement on new experimental methods when there are conflicts of interest about the evidence they yield? I argue that debiasing methods play a crucial role in this consensus, providing a warrant about the impartiality of the outcome regarding the preferences of different parties involved in the experiment. From a contractarian perspective, I contend that an epistemic pre-requisite for scientists to agree on an experimental method is that this latter is neutral regarding their competing interests. I present (...) two medical experiments (on smallpox inoculation and Mesmerism) in which debiasing procedures such as blinding and data tabulation provided warrants of impartiality that made people agree on the experimental design even if they disagreed on the outcome. (shrink)

While the history and methodology of laboratory experiments in economics have been extensively studied by philosophers, those of field experiments have not attracted much attention until recently. What is the historical context in which field experiments have been advocated? And what are the methodological rationales for conducting experiments in the field as opposed to in the lab? This article addresses these questions by combining historical and methodological perspectives. In terms of history, we show that the movement toward field experiments in (...) economics has two distinct roots. One is the general orientation of medical and social sciences to evidence-based policy evaluation, which gave rise to randomized field experiments in economics. The other is an awareness of several methodological limitations of lab experiments in economics, which required practitioners to get out of the lab and into the field. In these senses, the movement is a consequence of influences from both outside and inside economics: the general evidence-based trend in policy science and an internal methodological development of experimental economics. In terms of methodology, we show that these two roots resulted in two somewhat different notions of “external validity” as methodological rationales of field experiment. Finally, we suggest that analysis of experiments as exhibits highlights a methodological strategy in which both strands complement each other. (shrink)

In the biomedical, behavioural and social sciences, the leading method used to estimate causal effects is commonly randomised controlled trials (RCTs) that are generally viewed as both the source and justification of the most valid evidence. In studying the foundation and theory behind RCTs, the existing literature analyses important single issues and biases in isolation that influence causal outcomes in trials (such as randomisation, statistical probabilities and placebos). The common account of biased causal inference is described in a general way (...) in terms of probabilistic imbalances between trial groups. This paper expands the common account of causal bias by distinguishing between the range of biases arising between trial groups but also within one of the groups or across the entire sample during trial design, implementation and analysis. This is done by providing concrete examples from highly influential RCT studies. In going beyond the existing RCT literature, the paper provides a broader, practice-based account of causal bias that specifies the between-group, within-group and across-group biases that affect the estimated causal results of trials – impacting both the effect size and statistical significance. Within this expanded framework, we can better identify the range of different types of biases we face in practice and address the central question about the overall validity of the RCT method and its causal claims. A study can face several smaller biases (related simultaneously to a smaller sample, smaller estimated effect, greater unblinding etc.) that generally add up to greater aggregate bias. Though difficult to measure precisely, it is important to assess and provide information in studies on how much different sources of bias, combined, can explain the estimated causal effect. The RCT method is thereby often the best we have to inform our policy decisions – and the evidence is strengthened when combined with multiple studies and other methods. Yet there is room for continually improving trials and identifying ways to reduce biases they face and to increase their overall validity. Implications are discussed. (shrink)

Randomized experiments, as developed by Esther Duflo and Abhijit Banerjee at the Abdul Latif Jameel Poverty Action Lab, offer a novel, evidence-based approach to fighting poverty. This approach is original, in that it imports the methodology of clinical trials for application in development economics. This paper examines the analogy between J-PAL’s field experiments in development economics and randomized controlled trials in medicine. RCTs and randomized field experiments are commonly treated as identical, but such treatment neglects some of the major distinguishing (...) features that make each experiment specifically apt for use in its respective field. The central claim of this paper is that the analogy between medicine and development economics is incomplete because the central dimensions of RCTs are not simply different but altogether lacking in J-PAL’s approach. This weakens both the political and the theoretical power of such experiments in development economics. (shrink)

In this paper, we critically and constructively examine the methodology of evidence-based development economics, which deploys randomized field experiments as its main tool. We describe the...