Group decisions raise a number of substantial philosophical and methodological issues. We focus on the goal of the group decision exercise itself. We ask: What should be counted as a good group decision-making result? The right decision might not be accessible to, or please, any of the group members. Conversely, a popular decision can fail to be the correct decision. In this paper we discuss what it means for a decision to be "right" and what components are required in a (...) decision process to produce happy decision-makers. Importantly, we discuss how "right" decisions can produce happy decision-makers, or rather, the conditions under which happy decision-makers and right decisions coincide. In a large range of contexts, we argue for the adoption of formal consensus models to assist in the group decision-making process. In particular, we advocate the formal consensus convergence model of Lehrer and Wagner (1981), because a strong case can be made as to why the underlying algorithm produces a result that should make each of the experts in a group happy. Arguably, this model facilitates true consensus, where the group choice is effectively each person's individual choice. We analyse Lehrer and Wagner's algorithm for reaching consensus on group probabilities/utilities in the context of complex decision-making for conservation biology. While many conservation decisions are driven by a search for objective utility/probability distributions (regarding extinction risks of species and the like), other components of conservation management primarily concern the interests of stakeholders. We conclude with cautionary notes on mandating consensus in decision scenarios for which no fact of the matter exists. For such decision settings alternative types of social choice methods are more appropriate. (shrink)

In the wake of 9/11 and the assessment of Iraq's WMD, several inquiries placed the blame primarily on the Intelligence Community. Part of the reform that followed was a codification of analytic tradecraft standards into Intelligence Community Directive (ICD) 203 and the appointment of an analytic ombudsman in the newly created Office of the Director of National Intelligence charged with monitoring the quality of analytic products from across the intelligence community. In this paper we identify three assumptions behind ICD203: (1) (...) tradecraft standards can be employed consistently; (2) tradecraft standards sufficiently capture the key elements of good reasoning; (3) good reasoning leads to more accurate judgments. We then report on two controlled experiments that uncover operational constraints in the reliable application of the ICD203 criteria for the assessment of intelligence products. Despite criticisms of the post-9/11 and post-Iraq reform, our results highlight that ICD203, properly applied, holds potential to improve precision and accountability of intelligence processes and products. (shrink)

methods that have shown promise for improving extreme risk analysis, particularly for assessing the risks of invasive pests and pathogens associated with international trade. We describe the legally inspired regulatory regime for banks, where these methods have been brought to bear on extreme ‘operational risks’. We argue that an ‘advocacy model’ similar to that used in the Basel II compliance regime for bank operational risks and to a lesser extent in biosecurity import risk analyses is ideal for permitting the diversity (...) of relevant evidence about invasive species to be presented and soundly evaluated. We recommend that the process be enhanced in ways that enable invasion ecology to make more explicit use of the methods found successful.. (shrink)

Background. The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq's Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the US Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Community Directive (ICD) 203 – and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate (...) the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products. Method. Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments. Results. Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability. Conclusions. Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards. (shrink)