Kuhn claimed that several algorithms can be defended to select the best theory based on epistemic values such as simplicity, accuracy, and fruitfulness. In a recent paper, Okasha :83–115, 2011) argued that no theory choice algorithm exists which satisfies a set of intuitively compelling conditions that Arrow had proposed for a consistent aggregation of individual preference orderings. In this paper, we put forward a solution to avoid this impossibility result. Based on previous work by Gaertner and Xu, we suggest to (...) view the theory choice problem in a cardinal context and to use a general scoring function defined over a set of qualitative verdicts for every epistemic value. This aggregation method yields a complete and transitive ranking and the rule satisfies all Arrovian conditions appropriately reformulated within a cardinal setting. We also propose methods that capture the aggregation across different scientists. (shrink)

In the biomedical context, policy makers face a large amount of potentially discordant evidence from different sources. This prompts the question of how this evidence should be aggregated in the interests of best-informed policy recommendations. The starting point of our discussion is Hunter and Williams’ recent work on an automated aggregation method for medical evidence. Our negative claim is that it is far from clear what the relevant criteria for evaluating an evidence aggregator of this sort are. What is the (...) appropriate balance between explicitly coded algorithms and implicit reasoning involved, for instance, in the packaging of input evidence? In short: What is the optimal degree of ‘automation’? On the positive side: We propose the ability to perform an adequate robustness analysis as the focal criterion, primarily because it directs efforts to what is most important, namely, the structure of the algorithm and the appropriate extent of automation. Moreover, where there are resource constraints on the aggregation process, one must also consider what balance between volume of evidence and accuracy in the treatment of individual evidence best facilitates inference. There is no prerogative to aggregate the total evidence available if this would in fact reduce overall accuracy. (shrink)

We are facing uncertainties regarding climate change and its impacts. To conceptualize and communicate these uncertainties to policy makers, the Intergovernmental Panel on Climate Change (IPCC) has introduced an uncertainty framework. In this paper, I assess the latest, most developed, version of this framework. First, I provide an interpretation of this framework, which draws from supporting documents and the practice of its users. Second, I argue that even a charitable interpretation exhibits three substantial conceptual problems. These problems point towards untenable (...) assumptions regarding evidence aggregation in the context of climate scientific findings. Third, I put forward a tentative roadmap for improving the uncertainty framework. (shrink)

We are facing uncertainties regarding climate change and its impacts. To conceptualize and communicate these uncertainties to policy makers, the Intergovernmental Panel on Climate Change (IPCC) has introduced an uncertainty framework. In this paper, I assess the latest, most developed, version of this framework. First, I provide an interpretation of this framework, which draws from supporting documents and the practice of its users. Second, I argue that even a charitable interpretation exhibits three substantial conceptual problems. These problems point towards untenable (...) assumptions regarding evidence aggregation in the context of climate scientific findings. Third, I put forward a tentative roadmap for improving the uncertainty framework. (shrink)

Policy-makers face an uncertain world. One way of getting a handle on decision-making in such an environment is to rely on evidence. Despite the recent increase in post-fact figures in politics, evidence-based policymaking takes centre stage in policy-setting institutions. Often, however, policy-makers face large volumes of evidence from different sources. Robustness analysis can, prima facie, handle this evidential diversity. Roughly, a hypothesis is supported by robust evidence if the different evidential sources are in agreement. In this thesis, I strengthen the (...) case for the use of robustness analysis in evidence-based policymaking by answering open research questions about this inference technique. First, I argue that existing taxonomies miss a fruitful category of robustness reasoning, that is predictive stability. Second, I claim that derivational robustness analysis – the investigation of whether the results of different models are in agreement – can yield interesting insights even if not the entire relevant model space is covered by available models or if the model results are only partially in agreement. Third, I claim that expert knowledge is necessary to address questions that arise when one applies measurement robustness analysis – the investigation into whether multiple means of measurement yield the same result. Finally, I argue that, in situations where evidence from different measurements is not in agreement, it can be advisable to no longer take all of the evidence into account. This can be done in a rationally defensible way by choosing the most adequate theory or model underlying parts of the evidence set. I discuss examples from climate, medical, and economic policy-making to establish my claims. (shrink)