A Global Catastrophic Risk is one that has the potential to inflict serious damage to human well-being on a global scale. This book focuses on such risks arising from natural catastrophes, nuclear war, terrorism, biological weapons, totalitarianism, advanced nanotechnology, artificial intelligence and social collapse.

Catastrophic risk raises questions that are not only of practical importance, but also of great philosophical interest, such as how to define catastrophe and what distinguishes catastrophic outcomes from non-catastrophic ones. Catastrophic risk also raises questions about how to rationally respond to such risks. How to rationally respond arguably partly depends on the severity of the uncertainty, for instance, whether quantitative probabilistic information is available, or whether only comparative likelihood information is available, or neither (...) type of information. Finally, catastrophic risk raises important ethical questions about what to do when catastrophe avoidance conflicts with equity promotion. (shrink)

A classification of the global catastrophic risks of AI is presented, along with a comprehensive list of previously identified risks. This classification allows the identification of several new risks. We show that at each level of AI’s intelligence power, separate types of possible catastrophes dominate. Our classification demonstrates that the field of AI risks is diverse, and includes many scenarios beyond the commonly discussed cases of a paperclip maximizer or robot-caused unemployment. Global catastrophic failure could happen at various (...) levels of AI development, namely, before it starts self-improvement, during its takeoff, when it uses various instruments to escape its initial confinement, or after it successfully takes over the world and starts to implement its goal system, which could be plainly unaligned, or feature-flawed friendliness. AI could also halt at later stages of its development either due to technical glitches or ontological problems. Overall, we identified around several dozen scenarios of AI-driven global catastrophe. The extent of this list illustrates that there is no one simple solution to the problem of AI safety, and that AI safety theory is complex and must be customized for each AI development level. (shrink)

Recent progress in artificial intelligence (AI) has drawn attention to the technology’s transformative potential, including what some see as its prospects for causing large-scale harm. We review two influential arguments purporting to show how AI could pose catastrophic risks. The first argument — the Problem of Power-Seeking — claims that, under certain assumptions, advanced AI systems are likely to engage in dangerous power-seeking behavior in pursuit of their goals. We review reasons for thinking that AI systems might seek power, (...) that they might obtain it, that this could lead to catastrophe, and that we might build and deploy such systems anyway. The second argument claims that the development of human-level AI will unlock rapid further progress, culminating in AI systems far more capable than any human — this is the Singularity Hypothesis. Power-seeking behavior on the part of such systems might be particularly dangerous. We discuss a variety of objections to both arguments and conclude by assessing the state of the debate. (shrink)

An anthropogenic global catastrophic risk is a human-induced risk that threatens sustained and wide-scale loss of life and damage to civilisation across the globe. In order to understand how new research on governance mechanisms for emerging technologies might assuage such risks, it is important to ask how perceptions, beliefs, and attitudes towards the governance of global catastrophic risk within the research community shape the conduct of potentially risky research. The aim of this study is to (...) deepen our understanding of emerging technology research culture as it relates to global catastrophic risks, and to shed new light on how new research governance mechanisms might be developed. We analyse in-depth interviews with leading AI and biotech researchers both from universities and the private sector. We develop new insights in terms of four salient themes. First, ‘engineering mindset’, which highlights the premium placed by many interviewees on pursuing interesting research about the physical world for its own sake. Second, ‘self-government’, which looks at how self-regulation of technological development currently occurs. Third, ‘pure incentives’, focussing on how career and other incentives shapes research. Fourth, ‘norms and persuasion’, which examines the role of moral considerations in guiding the research choices of scientists. We end by considering the implications of these findings for future research on governance of anthropogenic global catastrophic risk. (shrink)

Paper originally prepared for the International Summit on "Deomcracy, Terrorism and Security," Madrid, 8-11 March 2005.

Suppose that a decision-maker's aim, under certainty, is to maximise some continuous value, such as lifetime income or continuous social welfare. Can such a decision-maker rationally satisfy what has been called "continuity for easy cases" while at the same time satisfying what seems to be a widespread intuition against the full-blown continuity axiom of expected utility theory? In this note I argue that the answer is "no": given transitivity and a weak trade-off principle, continuity for easy cases violates the anti-continuity (...) intuition. I end the note by exploring an even weaker continuity condition that is consistent with the aforementioned intuition. (shrink)

This paper examines the role that insurance and mitigation can play in reducing losses from natural disasters using data collected as part of a large-scale study on catastrophic risk jointly undertaken by the Wharton Risk Management Center in conjunction with Georgia State University and the Insurance Information Institute. The paper graphically demonstrates why disaster losses have increased in the past twenty-five years and the magnitude of the problem today. It then shows how mitigation measures can reduce future (...) losses using data on residential homes from four states facing severe risks from hurricanes. Insurance premiums can be used to incentivize homeowners to invest in protective measures if disaster coverage programs adhere to a set of guiding principles. The paper proposes long-term insurance as a way of reducing losses from future disasters and addressing concerns facing insurers and homeowners in hazard-prone areas. (shrink)

This paper examines the challenge of the calculus of risk while arguing for the necessity of new risk architecture. The field of catastrophe risk management today is faced with disasters of a totally new nature and scale as well recent important events that pose significant challenges to the established paradigm. This paper proposes a new risk architecture based around the following six central features: Growing interdependencies/globalization; change in scale from local to global risks; extreme costs, extreme (...) benefits ; confusing distribution of roles and responsibilities of preparedness; celerity ; and uncertainty, if not ignorance. The paper then looks at large-scale natural disasters and terrorism threats as two illustrative cases for understanding this new threat architecture, arguing that uncertainty related to the estimate of future disaster is even more challenging because it is so dynamic. The paper argues that in this new era of global risks and interdependencies, how people make long-term decision under conditions of uncertainty will be an important element to consider in developing the new risk architecture. (shrink)

The aim of this article is to investigate around the life in the contemporary society, characterized by risks and catastrophes. What does mean to live fearing that in any moment a catastrophe could happen (a tsunami, an earthquake, a nuclear explosion)? Despite of the failure of science and public institutions in the prevention of the catastrophes, the question is the following: Can we use the catastrophe as a paradigm of the contemporary uncertain life, trying to mean it as a theme (...) of mediation? Can we use the catastrophe for developing a philosophy of the recognition of the other who has been involved in the same tragedy? (shrink)

Recent advances in natural language processing have given rise to a new kind of AI architecture: the language agent. By repeatedly calling an LLM to perform a variety of cognitive tasks, language agents are able to function autonomously to pursue goals specified in natural language and stored in a human-readable format. Because of their architecture, language agents exhibit behavior that is predictable according to the laws of folk psychology: they function as though they have desires and beliefs, and then make (...) and update plans to pursue their desires given their beliefs. We argue that the rise of language agents significantly reduces the probability of an existential catastrophe due to loss of control over an AGI. This is because the probability of such an existential catastrophe is proportional to the difficulty of aligning AGI systems, and language agents significantly reduce that difficulty. In particular, language agents help to resolve three important issues related to aligning AIs: reward misspecification, goal misgeneralization, and uninterpretability. (shrink)

This paper discusses ethical issues surrounding Integrated Assessment Models (IAMs) of the economic effects of climate change, and how climate economists acting as policy advisors ought to represent the uncertain possibility of catastrophe. Some climate economists, especially Martin Weitzman, have argued for a precautionary approach where avoiding catastrophe should structure climate economists’ welfare analysis. This paper details ethical arguments that justify this approach, showing how Weitzman’s “fat tail” probabilities of climate catastrophe pose ethical problems for widely used IAMs. The main (...) claim is that economists who ignore or downplay catastrophic risks in their representations of uncertainty likely fall afoul of ethical constraints on scientists acting as policy advisors. Such scientists have duties to honestly articulate uncertainties and manage (some) inductive risks, or the risks of being wrong in different ways. (shrink)

In this chapter, we offer an account of fear and risk in anticipation of catastrophe. We draw on the narrative response to the Mount Enta volcano in medieval Sicily to frame an evaluation of how fear can be seen to impact the understanding of risk when the event of that risk is the catastrophic suffering of an entire community. We aim to demonstrate how an exploration of the philosophical questions surrounding the emotion of fear and the (...) understanding of risk can contribute to broader, interdisciplinary dialogue on the experience of the disastrous and deadly. (shrink)

In this chapter, we offer an account of fear and risk in anticipation of catastrophe. We draw on the narrative response to the Mount Enta volcano in medieval Sicily to frame an evaluation of how fear can be seen to impact the understanding of risk when the event of that risk is the catastrophic suffering of an entire community. We aim to demonstrate how an exploration of the philosophical questions surrounding the emotion of fear and the (...) understanding of risk can contribute to broader, interdisciplinary dialogue on the experience of the disastrous and deadly. (shrink)

Longtermists have argued that humanity should significantly increase its efforts to prevent catastrophes like nuclear wars, pandemics, and AI disasters. But one prominent longtermist argument overshoots this conclusion: the argument also implies that humanity should reduce the risk of existential catastrophe even at extreme cost to the present generation. This overshoot means that democratic governments cannot use the longtermist argument to guide their catastrophe policy. In this paper, we show that the case for preventing catastrophe does not depend on (...) longtermism. Standard cost-benefit analysis implies that governments should spend much more on reducing catastrophic risk. We argue that a government catastrophe policy guided by cost-benefit analysis should be the goal of longtermists in the political sphere. This policy would be democratically acceptable, and it would reduce existential risk by almost as much as a strong longtermist policy. (shrink)

Actions to prepare for and prevent pandemics are a common topic for bioethical analysis. However, little attention has been paid to global catastrophic biological risks more broadly, including pandemics with artificial origins, the creation of agents for biological warfare, and harmful outcomes of human genome editing. What’s more, international policy discussions often focus on economic arguments for state action, ignoring a key potential set of reasons for states to mitigate global catastrophic biological risks: moral reasons. In this paper, (...) I frame the mitigation of such risks as a global public good, and I explore three possible categories of moral reasons that might motivate states to provide this global public good: nationalism, cosmopolitanism, and interstate obligations. Whilst there are strong objections to moral nationalism as a reason for states to act, moral cosmopolitanism may provide a broad reason which is further supplemented for individual states through the elaboration of interstate moral obligations. The obligations I consider are moral leadership, fairness, and reciprocity. Moral reasons for individual states action may more effectively or more appropriately motivate states to mitigate global catastrophic biological risks. (shrink)

For quite some time, reinsurance companies have been pricing the ongoing climate change using weather- and catastrophe-related instruments and thus have been able to make money through climate change. Yet, at the same time, for reinsurance companies it is crucial that the likelihood of the events they underwrite is diminished as much as possible. Consequently, while profiting from the situation, these key actors of global capitalism also work to prevent climate change from taking place, and support the kinds of measures, (...) on all political scales, that diminish the likelihood of severe climate change destruction. This article analyzes the materials that the reinsurance company Munich Re has distributed to stakeholders and asks how climate change is objectified by the reinsurance industry. How are weather-related catastrophes made into a financial risk and opportunity? The key conceptual tools for answering these questions are provided by Michel Serres’s work on world-objects. (shrink)

The catastrophic risks posed by new technologies such as killer robots and geoengineering have triggered calls for halting new research. Arguments for restricting research typically have a slippery‐slope structure: Researching A will lead to deployment; we have decisive moral reasons against deployment; therefore, we should not research A. However, scientific uncertainty makes it difficult to prove or disprove the conclusion of slippery‐slope arguments. This article accepts this indeterminacy and asks whether and when it would be permissible to restrict research (...) under empirical and normative uncertainty. The argument starts from the ethical framework for the regulation of scientific research with human subjects and offers modifications to adapt it to the purpose of restricting new technologies. Two main questions arise in the process: whether it is permissible to impose restrictions at the research stage to prevent harms that will arise from the use of a technology and whether it is permissible to restrict research preemptively on the grounds of public fear and anxiety, before there is sufficient evidence establishing the risk of harm. I answer both questions in the affirmative and defend this position against objections. (shrink)

A small but growing number of studies have aimed to understand, assess and reduce existential risks, or risks that threaten the continued existence of mankind. However, most attention has been focused on known and tangible risks. This paper proposes a heuristic for reducing the risk of black swan extinction events. These events are, as the name suggests, stochastic and unforeseen when they happen. Decision theory based on a fixed model of possible outcomes cannot properly deal with this kind of (...) event. Neither can probabilistic risk analysis. This paper will argue that the approach that is referred to as engineering safety could be applied to reducing the risk from black swan extinction events. It will also propose a conceptual sketch of how such a strategy may be implemented: isolated, self-sufficient, and continuously manned underground refuges. Some characteristics of such refuges are also described, in particular the psychosocial aspects. Furthermore, it is argued that this implementation of the engineering safety strategy safety barriers would be effective and plausible and could reduce the risk of an extinction event in a wide range of possible scenarios. Considering the staggering opportunity cost of an existential catastrophe, such strategies ought to be explored more vigorously. (shrink)

This essay proposes to extend the model of apocalyptic argument developedin my recent book Arguing the Apocalypse (O‘Leary, 1994) beyond the study ofreligious discourse, by applying this model to the debate over awell-publicized earthquake prediction that caused a widespread panic in theAmerican midwest in December, 1990. The first section of the essay willsummarize the essential elements of apocalyptic argument as I have earlierdefined them; the second section will apply the model to the case of the NewMadrid, Missouri, earthquake prediction, in (...) order to demonstrate thatcertain patterns of reasoning characteristic of religious apocalyptic arepresent in the discourse over an anticipated local disaster. My ultimatepurpose is to show that predictions of global and local catastrophe mayserve as extreme cases that will illuminate the dynamics of predictiveargument in general. Thus my argument will seek to undercut Daniel Bell‘sdistinction between prophecy and prediction (Bell, 1973) by establishingthat these discourses share identifiable formal and substantivecharacteristics, and depend for their rhetorical effect on anxiety, hope,far, and excitement as modes of temporal anticipation. (shrink)

This paper examines and analyzes five definitions of ‘existential risk.’ It tentatively adopts a pluralistic approach according to which the definition that scholars employ should depend upon the particular context of use. More specifically, the notion that existential risks are ‘risks of human extinction or civilizational collapse’ is best when communicating with the public, whereas equating existential risks with a ‘significant loss of expected value’ may be the most effective definition for establishing existential risk studies as a legitimate (...) field of scientific and philosophical inquiry. In making these arguments, the present paper hopes to provide a modicum of clarity to foundational issues relating to the central concept of arguably the most important discussion of our times. (shrink)

Recently many methods for reducing the risk of human extinction have been suggested, including building refuges underground and in space. Here we will discuss the perspective of using military nuclear submarines or their derivatives to ensure the survival of a small portion of humanity who will be able to rebuild human civilization after a large catastrophe. We will show that it is a very cost-effective way to build refuges, and viable solutions exist for various budgets and timeframes. Nuclear submarines (...) are surface independent, and could provide energy, oxygen, fresh water and perhaps even food for their inhabitants for years. They are able to withstand close nuclear explosions and radiation. They are able to maintain isolation from biological attacks and most known weapons. They already exist and need only small adaptation to be used as refuges. But building refuges is only “Plan B” of existential risk preparation; it is better to eliminate such risks than try to survive them. (shrink)

The notion of existential catastrophe is increasingly appealed to in discussion of risk management around emerging technologies, but it is not completely clear what this notion amounts to. Here, I provide an opinionated survey of the space of plausibly useful definitions of existential catastrophe. Inter alia, I discuss: whether to define existential catastrophe in ex post or ex ante terms, whether an ex ante definition should be in terms of loss of expected value or loss of potential, and what (...) kind of probabilities should be involved in any appeal to expected value. (shrink)

Many scientists have expressed concerns about potential catastrophic risks associated with new technologies. But expressing concern is one thing, identifying serious candidates another. Such risks are likely to be novel, rare, and difficult to study; data will be scarce, making speculation necessary. Scientists who raise such concerns may face disapproval not only as doomsayers, but also for their unconventional views. Yet the costs of false negatives in these cases -- of wrongly dismissing warnings about catastrophic risks -- are (...) by definition very high. For these reasons, aspects of the methodology and culture of science, such as its attitude to epistemic risk and to unconventional views, are relevant to the challenges of managing extreme technological risks. In this piece I discuss these issues with reference to a real-world example that shares many of the same features, that of so-called 'cold fusion'. (shrink)

What kind of moral principle could be sufficiently restrictive to avoid the kind of large-scale risks that have resulted in catastrophe in the past, while at the same time not be so restrictive as to halt desirable progress? Is there such a principle that is not merely a precautionary principle, but one that could be based on firm moral grounds? In this article, I set out to explore a simple idea: might it be the case that reparability could serve as (...) a moral constraint against risky policy decisions? The idea is simple, but comes in two forms. First, that it is morally wrong to impose a risk for harm that is, in principle, irreparable, such that it would bring about a permanent loss of a kind qua kind. Second, that it is morally wrong to impose a risk for a reparable (but not compensable) harm that exceeds what realistically could be repaired. I set out here to do two things. First, I describe the moral problem to be addressed for any principle-guiding decisions about risk. A central claim in this article is that risk decisions are epistemically impaired decisions, and that we must, alongside outcome uncertainty, also take both epistemic uncertainty and moral uncertainty into account. Second, I introduce the idea of reparability as a moral constraint in the form of two versions of the Reparability Principle. Such a principle, I argue, could have some interesting advantages that seem both morally intuitive and that come with some advantages against some of the epistemic challenges posed by risk impositions. (shrink)

An asteroid colliding with the earth could cause the extinction of our species. Is this a risk worth worrying about? More important, is it a risk worth doing something about? Richard A. Posner, a judge on the United States Court of Appeals for the Seventh Circuit, who produces more books in his leisure hours than most authors do working full time, thinks it is.

A substantial literature on risk perception demonstrates the limits of human rationality, especially in the face of catastrophic risks. Human judgment, it seems, is flawed by the tendency to overestimate the magnitude of rare but evocative risks, while underestimating risks associated with commonplace dangers. Such findings are particularly relevant to the problem of crafting responsible public policy in the face of the kinds of threat posed by climate change. If the risk perception of ordinary citizens cannot be (...) trusted, then it would seem logical to base policy decisions on expert judgment. But how rational, how trustworthy, are expert assessments of catastrophic risk? I briefly review the limitations of conventional models of expert risk analysis, especially in dealing with the large uncertainties endemic to the risk of low-probability, high-impact events in the distant future. The challenges such events pose to the underlying assumptions of these analyses are severe enough to question their basic rationality. I argue that a conception of rationality premised on the bounded knowledge of experts and lay citizens alike, based on context-appropriate heuristics, may help us in the search for a more trustworthy basis for decision making. (shrink)

Catastrophe now forms part of our daily lives, as though the apocalypse could hit us every morning. Yet this crazed relation to the world is legitimate, constructed and not imaginary, entirely coherent with the postmodern socius. A biopolitics of catastrophe has come into being, in the attempt to include this new given and thereby conjure away the risks that, for Ulrich Beck, compose the measuring-stick of our post-progressive societies. However, by its very practices, this biopolitics seems to block the advent (...) of an ecopolitics that could act on the causes and not the effects of the injury that we are already suffering. (shrink)

Climate change is often described as an existential risk to the human species, but this terminology has generally been avoided in the climate-justice literature in analytic philosophy. I investigate the source of this disconnect and explore the prospects for incorporating the idea of climate change as an existential risk into debates about climate justice. The concept of existential risk does not feature prominently in these discussions, I suggest, because assumptions that structure ‘ideal’ accounts of climate justice ensure (...) that the prospect of climate change as an extinction-level threat does not arise. Given persistent noncompliance with mitigation duties, however, we have reason to revisit these assumptions. I argue that the most promising way for theories of climate justice to account for the significance of existential risk is to look to the practices of protest and resistance in which a concern about extinction or global catastrophe is frequently given expression. (shrink)

The Author presents the theoretical framework of the special issue on the fear of nature. The main goal of the essays in the special issue is reflecting on catastrophe and on its perception within the areas of contemporary human and social sciences; that is, revising the specific forms of the relationship between man, nature, society, as well as the relationship risk-fear-security, between society and government.

This paper aims to scrutinize how the notion of risk should be understood and applied to possibly catastrophic cases. I begin with clarifying the standard usage of the notion of risk, particularly emphasizing the conceptual relation between risk and probability. Then, I investigate how to make decisions in the case of seemingly catastrophic disasters by contrasting the precautionary principle with the preventive (prevention) principle. Finally, I examine what kind of causal thinking tends to be actually (...) adopted when we make decisions on emergent cases. My arguments are mainly based on Japan's 2011 massive earthquake and nuclear power station accident. Masaki Ichinose. (shrink)

The greatest existential threats to humanity stem from increasingly powerful advanced technologies. Yet the “risk potential” of such tools can only be realized when coupled with a suitable agent who; through error or terror; could use the tool to bring about an existential catastrophe. While the existential risk literature has provided many accounts of how advanced technologies might be misused and abused to cause unprecedented harm; no scholar has yet explored the other half of the agent-tool coupling; namely (...) the agent. This paper aims to correct this failure by offering a comprehensive overview of what we could call “agential riskology.” Only by studying the unique properties of different agential risk types can one acquire an accurate picture of the existential danger before us. (shrink)

The aim of the paper is to propose a preferences representation model under risk where risk perception can be past experience dependent. A first step consists in considering a one period decision problem where individual preferences are no more defined only on decisions but on pairs (decision, past experience). The obtained criterion is used in the construction of a dynamic choice model under risk. The paper ends with an illustrative example concerning insurance demand. It appears that our (...) model allows to explain modifications in the insurance demand behavior over time observed on the insurance markets for catastrophic risk and difficult to justify with standard models. (shrink)

This essay uses Ulrich Beck’s concept of risk society to understand the threat of catastrophic climate change. It argues that this threat is “abstract-corporeal”, and therefore a special kind of threat that poses special kinds of epistemic and ecological challenges. At the center of these challenges is the problem of human vulnerability, which entails a complex form of trust that both sustains and threatens human survival.

Humanity is threatened by global risks that cross borders and national institutions in a way that has never been experienced before. Disasters, far from being purely natural phenomena, constitute the greatest danger in a context of political, social and scientific uncertainty. A new risk-based governance appears, ranging from the introduction of problematic exceptional measures and an increased vulnerability and inequality in risk exposure. This proposal aims to deepen the analysis of the relationship between law, politics and catastrophe as (...) the focal point of the evolution of risk prevention and management and its significance for the transformations of our understanding of democracy. The research proposes to engage catastrophes by conceiving them as powerful amplifiers and epiphanies of injustices, assuming that the role of the state should be inserted into a wider reading that must include the problem of unequal distribution of risks. (shrink)

In recent years, US policy makers have faced persistent calls for the price of flood and hurricane insurance cover to reflect the true or real risk. The appeal to a true or real measure of risk is rooted in two assumptions. First, scientific research can provide an accurate measure of risk. Second, this information can and should dictate decision-making about the cost of insurance. As a result, contemporary disputes over the cost of catastrophe insurance coverage, hurricane (...) class='Hi'>risk being a prime example, become technical battles over estimating risk. Using examples from the Florida hurricane rate-making decision context, we provide a quantitative investigation of the integrity of these two assumptions. We argue that catastrophe models are politically stylized views of the intractable scientific problem of precise characterization of hurricane risk. Faced with many conflicting scientific theories, model theorists use choice and preference for outcomes to develop a model. Models therefore come to include political positions on relevant knowledge and the risk that society ought to manage. Earnest consideration of model capabilities and inherent uncertainties may help evolve public debate from one focused on “true” or “real” measures of risk, of which there are many, toward one of improved understanding and management of insurance regimes. (shrink)

Socialists have historically thought a lot about the catastrophic risks society faces. Today many DSA chapters have gotten involved in mutual aid to respond to the Covid crisis, generating a debate about how mutual aid fits into socialist work. One form of community engagement that is likely to be increasingly necessary, and is an opportunity for radicalizing angry neighbors, is disaster preparedness. While the prepper subculture is perceived as right-wing, and parts are tied into the militia movement, there are (...) also opportunities for progressive organizations to work with FEMA and local governments. This essay discusses distinctive socialist perspectives on disasters past and future and gives examples of strategic ways socialists can help build disaster-resilient communities. (shrink)

Health security has become a popular way of justifying efforts to control catastrophic threats to public health. Unfortunately, there has been little analysis of the concept of health security, nor the relationship between health security and other potential aims of public health policy. In this paper I develop an account of health security as an aversion to risky policy options. I explore three reasons for thinking risk avoidance is a distinctly worthwhile aim of public health policy: that security (...) is intrinsically valuable, that it is necessary for social planning and that it is an appropriate response to decision-making in contexts of very limited information. Striking the right balance between securing and maximizing population health thus requires a substantive, and hitherto unrecognized, value judgment. Finally, I critically evaluate the current health security agenda in light of this new account of the concept and its relationship to the other aims of public health policy. (shrink)

The possibility of social and technological collapse has been the focus of science fiction tropes for decades, but more recent focus has been on specific sources of existential and global catastrophic risk. Because these scenarios are simple to understand and envision, they receive more attention than risks due to complex interplay of failures, or risks that cannot be clearly specified. In this paper, we discuss the possibility that complexity of a certain type leads to fragility which can function (...) as a source of catastrophic or even existential risk. The paper first reviews a hypothesis by Bostrom about inevitable technological risks, named the vulnerable world hypothesis. This paper next hypothesizes that fragility may not only be a possible risk, but could be inevitable,and would therefore be a subclass or example of Bostrom’s vulnerable worlds. After introducing the titular fragile world hypothesis, the paper details the conditions under which it would be correct, and presents arguments for why the conditions may in fact may apply. Finally, the assumptions and potential mitigations of the new hypothesis are contrasted with those Bostrom suggests. (shrink)

Risk is the most often cited reason for ethical concern about any medical science or technology, particularly those new technologies that are not yet well understood, or create unfamiliar conditions. In fact, while risk and risk-benefit analyses are but one aspect of ethical oversight, ethical review and risk assessment are sometimes taken to mean the same thing. This is not surprising, since both the Common Rule and Food and Drug Administration foreground procedures for minimizing risk (...) for human subjects and require local IRBs to engage in some sort of risk-benefit analysis in decisions to approve or deny proposed research. Existing ethical review and oversight practices are based on the presumption that risk can be clearly identified within the planned activities of the protocol, that metrics can reasonably accurately predict potential hazards, and that mitigation measures can be taken to deal with unintended, harmful, or catastrophic events. (shrink)

’Catastrophes are not accidents.’All complex organizations can take steps to prevent catastrophe by enlisting the new scientific study of danger. The author is a member of the Scientific Committee of 1′Institut Européen de Cindyniques and Directeur général adjoint of L'Union des Assurances de Paris. This article is part of a presentation made to the 1992 Paris Conference of the European Business Ethics Network, of which the author is a Council member. The subject of the Conference was the role of business (...) in the shaping of cities, but the author's reflections on learning from risks are of considerably wider application. (shrink)

’Catastrophes are not accidents.’All complex organizations can take steps to prevent catastrophe by enlisting the new scientific study of danger. The author is a member of the Scientific Committee of 1′Institut Européen de Cindyniques and Directeur général adjoint of L'Union des Assurances de Paris. This article is part of a presentation made to the 1992 Paris Conference of the European Business Ethics Network, of which the author is a Council member. The subject of the Conference was the role of business (...) in the shaping of cities, but the author's reflections on learning from risks are of considerably wider application. (shrink)

Some risks have extremely high stakes. For example, a worldwide pandemic or asteroid impact could potentially kill more than a billion people. Comfortingly, scientific calculations often put very low probabilities on the occurrence of such catastrophes. In this paper, we argue that there are important new methodological problems which arise when assessing global catastrophic risks and we focus on a problem regarding probability estimation. When an expert provides a calculation of the probability of an outcome, they are really providing (...) the probability of the outcome occurring, given that their argument is watertight. However, their argument may fail for a number of reasons such as a flaw in the underlying theory, a flaw in the modeling of the problem, or a mistake in the calculations. If the probability estimate given by an argument is dwarfed by the chance that the argument itself is flawed, then the estimate is suspect. We develop this idea formally, explaining how it differs from the related distinctions of model and parameter uncertainty. Using the risk estimates from the Large Hadron Collider as a test case, we show how serious the problem can be when it comes to catastrophic risks and how best to address it. (shrink)

We describe a significant practical consequence of taking anthropic biases into account in deriving predictions for rare stochastic catastrophic events. The risks associated with catastrophes such as asteroidal/cometary impacts, supervolcanic episodes, and explosions of supernovae/gamma-ray bursts are based on their observed frequencies. As a result, the frequencies of catastrophes that destroy or are otherwise incompatible with the existence of observers are systematically underestimated. We describe the consequences of the anthropic bias for estimation of catastrophic risks, and suggest some (...) directions for future work. (shrink)