Existential Risk

Over one million annual deaths have been attributed to bacterial antimicrobial resistance. Although antibiotics have saved countless other lives, overuse and misuse of antibiotics increases this global threat. Developing new antibiotics and retraining clinicians can be undermined by patients who pressure clinicians to prescribe unnecessary antibiotics. So we validated two low-cost, scalable interventions for improving antibiotic decisions in an online randomized control trial and a pre-registered replication (N = 985). Both first-person vignette experiments found that an infographic and text message (...) caused intermediate to large improvements in antibiotic decisions compared to active controls — even when controlling for a dozen confounds. Notably, these educational interventions also reduced trust in a clinician who offered antibiotics for an upper respiratory infection. These data show how low-cost adjustments to existing communication systems can teach people to reduce the existential risks of antibiotic resistance and how inappropriate antibiotic prescribing can erode trust in health systems. (shrink)

This paper examines and evaluates a range of methodologies that have been proposed for making useful claims about the probability of phenomena that would contribute to existential risk. Section One provides a brief discussion of the nature of such claims, the contexts in which they tend to be made and the kinds of probability that they can contain. Section Two provides an overview of the methodologies that have been developed to arrive at these probabilities and assesses their advantages and disadvantages. (...) Section Three contains four suggestions to improve best practice in existential risk assessment. These suggestions centre on the types of probabilities used in risk assessment, the role of methodology rankings including the ranking of probabilistic information, the extended use of expert elicitation, and the use of confidence measures to better communicate uncertainty in probability assessments. Finally, Section Four provides an annotated literature review of catastrophic and existential risk probability claims as well as the methodologies that were used to produce each of them. (shrink)

When our choice affects some other person and the outcome is unknown, it has been argued that we should defer to their risk attitude, if known, or else default to use of a risk avoidant risk function. This, in turn, has been claimed to require the use of a risk avoidant risk function when making decisions that primarily affect future people, and to decrease the desirability of efforts to prevent human extinction, owing to the significant risks associated with continued human (...) survival. I raise objections to the claim that respect for others’ risk attitudes requires risk avoidance when choosing for future generations. In particular, I argue that there is no known principle of interpersonal aggregation that yields acceptable results in variable population contexts and is consistent with a plausible ideal of respect for others’ risk attitudes in fixed population cases. GPI Working Paper No . 20-2022. (shrink)

Scientific and technological progress might change people’s capabilities or incentives in ways that would destabilize civilization. For example, advances in DIY biohacking tools might make it easy for anybody with basic training in biology to kill millions; novel military technologies could trigger arms races in which whoever strikes first has a decisive advantage; or some economically advantageous process may be invented that produces disastrous negative global externalities that are hard to regulate. This paper introduces the concept of a vulnerable world: (...) roughly, one in which there is some level of technological development at which civilization almost certainly gets devastated by default, i.e. unless it has exited the “semi-anarchic default condition”. Several counterfactual historical and speculative future vulnerabilities are analyzed and arranged into a typology. A general ability to stabilize a vulnerable world would require greatly amplified capacities for preventive policing and global governance. The vulnerable world hypothesis thus offers a new perspective from which to evaluate the risk-benefit balance of developments towards ubiquitous surveillance or a unipolar world order. (shrink)

Since the release of ChatGPT, there has been a lot of debate about whether AI systems pose an existential risk to humanity. This paper develops a general framework for thinking about the existential risk of AI systems. We analyze a two-premise argument that AI systems pose a threat to humanity. Premise one: AI systems will become extremely powerful. Premise two: if AI systems become extremely powerful, they will destroy humanity. We use these two premises to construct a taxonomy of ‘survival (...) stories’, in which humanity survives into the far future. In each survival story, one of the two premises fails. Either scientific barriers prevent AI systems from becoming extremely powerful; or humanity bans research into AI systems, thereby preventing them from becoming extremely powerful; or extremely powerful AI systems do not destroy humanity, because their goals prevent them from doing so; or extremely powerful AI systems do not destroy humanity, because we can reliably detect and disable systems that have the goal of doing so. We argue that different survival stories face different challenges. We also argue that different survival stories motivate different responses to the threats from AI. Finally, we use our taxonomy to produce rough estimates of ‘P(doom)’, the probability that humanity will be destroyed by AI. (shrink)

The ongoing conflict in Ukraine constitutes a poignant reminder of the enduring relevance and potential devastation associated with nuclear weapons. For decades, the possibility of such catastrophic conflict has not seemed so imminent as in the current world affairs. -/- This contribution presents a comprehensive analysis of nuclear strategy for the 21st century. By examining the evolving geostrategic landscape the talk illuminates key concepts such as nuclear posture, credible deterrence, first & second strike capabilities, flexible response, EMP , variable yield, (...) counterforce & countervalue, limited war and escalation dominance, thereby providing a nuanced understanding of the delicate balance of power in the nuclear arena in modern conflicts. Furthermore, it concentrates on the era that started with the war in Ukraine, offering insights into how recent geopolitical developments influence nuclear doctrines. -/- Through conceivable real-world scenarios, we gain perspective on the multifaceted challenges posed by newer technologies and potential escalation dynamics involving global nuclear actors. -/- Such scenarios include: -/- 1. A non-weaponised or sub-critical test within Russia to intimidate opponents into understanding Russia’s resolve and capabilities. Prior norms have restricted testing since 2000, apart from North Korea. In case of a full surface test, Russia could detonate in uninhabited areas within its own vast territory with minimal fallout. While clarifying military strength, demonstrations risk hardening resistance without incentivising surrender. -/- 2. Coercive atomic threat to compel Ukraine or its supporters to make concessions. By leveraging the spectre of nuclear warfare, Russia could expect to achieve its objectives through intimidation and fear. However, the effectiveness of such a strategy is questionable, as it may provoke a strong international response and further galvanise support for Ukraine. -/- 3. Demonstration detonation over/under the Black Sea to signal resolve without targeting Ukraine itself; possibly amidst negotiations to gain diplomatic leverage, whilst limiting human casualties to zero or causing a mild, non-destructive tsunami. -/- 4. Ukraine could decide to launch a pre-emptive strike on Russian nuclear facilities or weapons, fearing an imminent attack. Such an action would have a high risk of failure and might even provoke a nuclear retaliation from Russia. It could also lead to a broader conflict, as neighbouring countries might feel compelled to intervene. -/- 5. Limited tactical strike against Ukrainian military or infrastructure to weaken capacity and morale. Russia’s 2000+ non-strategic arsenal could target positions using short-range missiles with 0.3 – 5 kiloton yields. Objectives like crippling Kharkiv defences might create no-go zones hindering advances. However, Ukraine might maintain drone and cyber offensive capabilities regardless. -/- 6. Electromagnetic Pulse attack designed to disable critical infrastructure in Ukraine and NATO countries without radioactive fallout. This aims to hinder the war effort and economy without provoking the same conventional military response as a blast. However, an EMP attack affecting NATO shall still be considered an act of war warranting retaliation. -/- 7. A weapon could be deployed accidentally or without proper authorisation. This could occur due to a technical malfunction, miscommunication, human error, or the actions of rogue elements. The consequences of such an event could be catastrophic, leading to a rapid escalation of the conflict and potentially triggering a global crisis. -/- 8. Sub-strategic offensive targeting NATO in the European theatre, so as to fracture allied resolve, against arms routes in Poland, Baltic stockpiles, British naval bases, American troop concentrations in Germany etc. Putin may perceive defeat or deposition as an existential threat, viewing such an option as self-preservation despite adverse consequences. -/- . (shrink)

Large-scale, self-sufficient space colonization is a plausible means of efficiently reducing existential risks and ensuring our long-term survival. But humanity is by and large myopic, and as an intergenerational global public good, existential risk reduction is systematically undervalued, hampered by intergenerational discounting. This paper explores how these issues apply to space colonization, arguing that the motivational and psychological barriers to space colonization are a special—and especially strong—case of a more general problem. The upshot is not that large-scale, self-sufficient space colonization (...) will never occur, but that, absent institutional change, the conditions under which it is most likely to occur are precisely those conditions where the threat of suffering risks might be most high. (shrink)

The possibility of social and technological collapse has been the focus of science fiction tropes for decades, but more recent focus has been on specific sources of existential and global catastrophic risk. Because these scenarios are simple to understand and envision, they receive more attention than risks due to complex interplay of failures, or risks that cannot be clearly specified. In this paper, we discuss the possibility that complexity of a certain type leads to fragility which can function as a (...) source of catastrophic or even existential risk. The paper first reviews a hypothesis by Bostrom about inevitable technological risks, named the vulnerable world hypothesis. This paper next hypothesizes that fragility may not only be a possible risk, but could be inevitable,and would therefore be a subclass or example of Bostrom’s vulnerable worlds. After introducing the titular fragile world hypothesis, the paper details the conditions under which it would be correct, and presents arguments for why the conditions may in fact may apply. Finally, the assumptions and potential mitigations of the new hypothesis are contrasted with those Bostrom suggests. (shrink)

Many people believe that it is better to extend the length of a happy life than to create a new happy life, even if the total welfare is the same in both cases. Despite the popularity of this view, one would be hard-pressed to find a fully compelling justification for it in the literature. This paper develops a novel account of why and when extension is better than replacement that applies not just to persons but also to non-human animals and (...) humanity as a whole. (shrink)

What moral reasons, if any, do we have to prevent the extinction of humanity? In “Unfinished Business,” Jonathan Knutzen argues that certain further developments in culture would make our history more ‘collectively meaningful,’ and that premature extinction would be bad because it would close off that possibility. Here, I critically examine this proposal. I argue that if collective meaningfulness is analogous to individual meaningfulness, then our meaning-based reasons to prevent the extinction of humanity are substantially different from the reasons discussed (...) by Knutzen. (shrink)

Abstract: Longtermist Effective Altruists (EAs) aim to mitigate the risk of existential catastrophes. In this paper, I have three goals. First, I identify a catastrophic risk that EAs have completely ignored. I call it religious catastrophe: the threat that (as Christians and Muslims have warned for centuries) billions of people stand in danger of going to hell for all eternity. Second, I argue that, even by secular EA lights, religious catastrophe is at least as bad and at least as probable (...) and, therefore, at least as important as many of the standard EA catastrophic risks (e.g., catastrophic climate change, nuclear winter). Third, I present the following dilemma for secular EAs: either adopt religious catastrophe as an EA cause or ignore religious catastrophe but also ignore catastrophic risks whose mitigation has a similar or lower expected value (i.e., most or all of them). Business as usual—ignoring religious catastrophe while championing the typical EA causes—is not an option consistent with longtermist EA principles. (shrink)

The precautionary principle is invoked in a number of important personal and policy decision contexts. Peterson shows that certain ways of making the principle precise are inconsistent with other criteria of decision-making. Some object that the results do not apply to cases of deep uncertainty or value incommensurability which are alleged to be in the principle’s wheelhouse. First, I show that Peterson’s impossibility results can be generalized considerably to cover cases of both deep uncertainty and incommensurability. Second, I contrast an (...) alternative way of giving voice to the precautionary impulse. (shrink)

Expected value maximization gives plausible guidance for moral decision‐making under uncertainty in many situations. But it has unappetizing implications in ‘Pascalian’ situations involving tiny probabilities of extreme outcomes. This paper shows, first, that under realistic levels of ‘background uncertainty’ about sources of value independent of one's present choice, a widely accepted and apparently innocuous principle—stochastic dominance—requires that prospects be ranked by the expected value of their consequences in most ordinary choice situations. But second, this implication does not hold when differences (...) in expected value are driven by tiny probabilities of extreme outcomes. Stochastic dominance therefore lets us draw a surprisingly principled line between ‘ordinary’ and ‘Pascalian’ situations, providing a powerful justification for de facto expected value maximization in the former context while permitting deviations in the latter. Drawing this distinction is incompatible with an in‐principle commitment to maximizing expected value, but does not require too much departure from decision‐theoretic orthodoxy: it is compatible, for instance, with the view that moral agents must maximize the expectation of a utility function that is an increasing function of moral value. (shrink)

We argue that two long-term goals of AI research stand in tension with one another. The first involves creating AI that is safe, where this is understood as solving the problem of value alignment. The second involves creating artificial general intelligence, meaning AI that operates at or beyond human capacity across all or many intellectual domains. Our argument focuses on the human capacity to make what we call “existential choices”, choices that transform who we are as persons, including transforming what (...) we most deeply value or desire. It is a capacity for a kind of value misalignment, in that the values held prior to making such choices can be significantly different from (misaligned with) the values held after making them. Because of the connection to existentialist philosophers who highlight these choices, we call the resulting form of risk “existentialist risk.” It is, roughly, the risk that results from AI taking an active role in authoring its own values rather than passively going along with the values given to it. On our view, human-like intelligence requires a human-like capacity for value misalignment, which is in tension with the possibility of guaranteeing value alignment between AI and humans. (shrink)

Capitalism is defined as the economic structure in which decisions over production are largely made by or on behalf of individuals in virtue of their private property ownership, subject to the incentives and constraints of market competition. In this paper, I will argue that considerations of long-term welfare, such as those developed by Greaves and MacAskill (2021), support anticapitalism in a weak sense (reducing the extent to which the economy is capitalistic) and perhaps support anticapitalism in a stronger sense (establishing (...) an alternative economic structure in which capitalism is not predominant). I hope to encourage longtermists to give anticapitalism serious consideration, and to encourage anticapitalists to pursue criticisms of capitalism’s efficiency as well as its injustices. (shrink)

What socially beneficial causes should philanthropists prioritize if they give equal ethical weight to the welfare of current and future generations? Many have argued that, because human extinction would result in a permanent loss of all future generations, extinction risk mitigation should be the top priority given this impartial stance. Using standard models of population dynamics, we challenge this conclusion. We first introduce a theoretical framework for quantifying undiscounted cost-effectiveness over the long term. We then show that standard population models (...) imply that there are interventions other than extinction risk mitigation that can produce persistent social benefits. In fact, these social benefits are large enough to render the associated interventions at least as cost-effective as extinction risk mitigation. (shrink)

Longtermists have argued that humanity should significantly increase its efforts to prevent catastrophes like nuclear wars, pandemics, and AI disasters. But one prominent longtermist argument overshoots this conclusion: the argument also implies that humanity should reduce the risk of existential catastrophe even at extreme cost to the present generation. This overshoot means that democratic governments cannot use the longtermist argument to guide their catastrophe policy. In this paper, we show that the case for preventing catastrophe does not depend on longtermism. (...) Standard cost-benefit analysis implies that governments should spend much more on reducing catastrophic risk. We argue that a government catastrophe policy guided by cost-benefit analysis should be the goal of longtermists in the political sphere. This policy would be democratically acceptable, and it would reduce existential risk by almost as much as a strong longtermist policy. (shrink)

A social planner who evaluates risky public policies in light of the other risks with which their society will be faced should judge favourably some such policies even though they would deem them too risky when considered in isolation. I suggest that a longtermist would—or at least should—evaluate risky polices in light of their prediction about future risks; hence, longtermism supports social risk-taking. I consider two formal versions of this argument, discuss the conditions needed for the argument to be valid, (...) and briefly compare these conditions to some risky policy options with which actual public decision-makers are faced. (shrink)

In this paper, we argue that our moral concern for future well-being should reduce over time due to important practical considerations about how humans interact with spacetime. After surveying several of these considerations (around equality, special duties, existential contingency, and overlapping moral concern) we develop a set of core principles that can both explain their moral significance and highlight why this is inherently bound up with our relationship with spacetime. These relate to the equitable distribution of (1) moral concern in (...) a universe that we can only causally affect in one temporal direction; (2) access to the benefits from using spacetime as a resource; and (3) the burdens of care given (1) and (2). We conclude by considering the practical implications of our argument and find that, while it is often assumed that a preference for present over future well-being weakens the case for existential risk mitigation, this likely is not the case. (shrink)

Adoptando una revitalización de la tradición marxista en los Estudios Sociales de Ciencia y Tecnología (STS) y en la filosofía contemporánea, mi trabajo se ocupa de la emergencia de la colapsología en Francia y Bélgica, Europa y el mundo, sus tensiones constituyentes con el campo hegemónico de los societal collapse studies. La tesis realiza un seguimiento minucioso de la colapsología y sus profetas fundacionales, el ingeniero agrónomo y Doctor en Biología, Pablo Servigne, y el administrador ambiental de empresas y Máster (...) en Ciencias Holísticas, Raphaël Stevens, en su intento retórico de constituir un nuevo campo. En el primer capítulo, realizo un diálogo crítico y ensamble de dinámicas y funcionamientos de los campos, estudios de retóricas e ideologías de las ciencias, literaturas especializadas sobre hegemonías de riesgos y catástrofes, reconsiderando los catastrofismos y las finitudes del capitalismo. En el segundo capítulo, en un rango temporal de 2015-2024, en siete momentos analizo los mecanismos de emergencia y reconocimiento, hegemonía y contrahegemonía de los actores colapsólogos, sus repertorios discursivos de catástrofes y disciplinas, con el desglose de su hipótesis fuerte del colapso civilizatorio de la modernidad termo industrial. En las conclusiones, realzo el interés reflexivo del campo mundial STS y la filosofía hacia los campos de estudios catastrofistas. Valiéndome del caso emergente de la colapsología, evidencio los resultados centrales conseguidos y las conexiones con diagnósticos de estados de riesgo y estados de ánimo cultural, tanto de las tecnociencias actuales y sus expertos, como de los públicos, las clases sociales y las sociedades, respecto a la crisis epocal de la civilización capitalista contemporánea. -/- Adopting a revitalization of the Marxist tradition in Science and Technology Studies (STS) and contemporary philosophy, my work addresses the emergence of collapsology in France and Belgium, France and the world, its constituent tensions with the hegemonic field of societal collapse studies. The thesis carefully monitors collapsology and its founding prophets, the agronomer engineer and Doctor in Biology, Pablo Servigne, and the environmental business administrator and Master in Holistic Sciences, Raphaël Stevens, in their rhetorical attempt to constitute a new field. In the first chapter, I carry out a critical dialogue and assembly of dynamics and functioning of the fields, studies of rhetoric and ideologies of sciences, specialized literature on hegemonies of risks and catastrophes, reconsidering catastrophisms and the finiteness of capitalism. In the second chapter, in a time range of 2015-2024, in seven moments I analyze the mechanisms of emergence, recognition, hegemony and counter hegemony of the collapsological actors, their discursive repertoires of catastrophes and disciplines, with the breakdown of their strong hypothesis of the civilizational collapse of thermo-industrial modernity. In the conclusions, I highlight the reflexive interest of the world STS field and philosophy towards the fields of catastrophist studies. Using the emerging case of collapsology, I demonstrate the central results achieved and the connections with diagnoses of risk states and cultural moods, both of current technosciences and their experts, as well as of the public, social classes and societies, regarding the epochal crisis of contemporary capitalist civilization. (shrink)

‘Who argues what’ bears a certain relevance in relation to what is being argued. We are entitled to know those personal circumstances which play a significant role in relation to the argument one supports, so that we can take those circumstances into consideration when evaluating their argument. This is why journals have conflict of interest declarations, and why we value reflexivity in the social sciences. We also often perform double-blind peer review. We recognise that the evaluation of certain statements of (...) fact or value can, and sometimes ought to, be independent of any reference to their authors. Jecker et al argue that we have reasons to prioritise justice concerns over the existential risk (X-Risk) posed by artificial intelligence (AI) as we transition towards more AI-centred societies, despite the warnings about X-Risk that come from the powerful voice of tech leaders, effective altruists and longtermists. Those powerful voices, Jecker et al argue, often have conflicts of interest and are not demographically or epistemically representative. Hence, we should deprioritise their concerns. I believe the opposite may be true. Even though warnings about X-Risk come from sources and in forms we may disagree with, we have reasons to …. (shrink)

Many researchers and intellectuals warn about extreme risks from artificial intelligence. However, these warnings typically came without systematic arguments in support. This paper provides an argument that AI will lead to the permanent disempowerment of humanity, e.g. human extinction, by 2100. It rests on four substantive premises which it motivates and defends: first, the speed of advances in AI capability, as well as the capability level current systems have already reached, suggest that it is practically possible to build AI systems (...) capable of disempowering humanity by 2100. Second, due to incentives and coordination problems, if it is possible to build such AI, it will be built. Third, since it appears to be a hard technical problem to build AI which is aligned with the goals of its designers, and many actors might build powerful AI, misaligned powerful AI will be built. Fourth, because disempowering humanity is useful for a large range of misaligned goals, such AI will try to disempower humanity. If AI is capable of disempowering humanity and tries to disempower humanity by 2100, then humanity will be disempowered by 2100. This conclusion has immense moral and prudential significance. (shrink)

Many of us think human extinction would be a very bad thing, and that we have moral reasons to prevent it. But there is disagreement over what would make extinction so bad, and thus over what grounds these moral reasons. Recently, several theorists have argued that our reasons to prevent extinction stem not just from the value of the welfare of future lives, but also from certain additional values relating to the existence of humanity itself (for example, humanity’s “final” value, (...) or the value of humanity in itself). In this paper, I argue against these “additional value” views. Despite their initial appeal, these views will inevitably face conflicts between the additional values to which they appeal, and the value of the welfare of future lives. And, I argue, the views cannot plausibly resolve these conflicts. In contrast, these conflicts do not arise for a rival view, on which our reasons to prevent extinction stem just from the value of the welfare of future lives. I conclude that this gives us reason to prefer the latter view, despite the greater initial plausibility of additional value views. (shrink)

The notion of existential catastrophe is increasingly appealed to in discussion of risk management around emerging technologies, but it is not completely clear what this notion amounts to. Here, I provide an opinionated survey of the space of plausibly useful definitions of existential catastrophe. Inter alia, I discuss: whether to define existential catastrophe in ex post or ex ante terms, whether an ex ante definition should be in terms of loss of expected value or loss of potential, and what kind (...) of probabilities should be involved in any appeal to expected value. (shrink)

This paper examines a recent argument in favour of strong precautionary action—possibly including working to hasten human extinction—on the basis of a decision-theoretic view that accommodates the risk-attitudes of all affected while giving more weight to the more risk-averse attitudes. First, we dispute the need to take into account other people’s attitudes towards risk at all. Second we argue that a version of the non-identity problem undermines the case for doing so in the context of future people. Lastly, we suggest (...) that we should not work to hasten human extinction, even if significant risk aversion is warranted. (shrink)

> Future people count. There could be a lot of them. We can make their lives better. > > -- William MacAskill, What We Owe The Future > > [Longtermism is] quite possibly the most dangerous secular belief system in the world today. > > -- Émile P. Torres, Against Longtermism Philosophers,1 2 psychologists,3 4 politicians5 and even some tech billionaires6 have sounded the alarm about artificial intelligence (AI) and the dangers it may pose to the long-term future of humanity. (...) Some believe it poses an existential risk (X-Risk)7 to our species, potentially causing our extinction or bringing about the collapse of human civilisation as we know it. The above quote from philosopher Will MacAskill captures the key tenets of ‘longtermism’, an ethical standpoint that places the onus on current generations to prevent AI-related—and other—X-Risks for the sake of people living in the future.1 2 Developing from an adjacent social movement commonly associated with utilitarian philosophy, ‘effective altruism’, longtermism has amassed following of its own. Its supporters argue that preventing X-Risks is at least as morally significant as addressing current challenges like global poverty. However, critics are concerned that such a distant-future focus will sideline efforts to tackle the many pressing moral issues facing humanity now.8 9 Indeed, according to ‘strong’ longtermism,10 future needs arguably should take precedence over present ones. In essence, the claim is that there is greater expected utility to allocating available resources to prevent human extinction in the future than there is to focusing on present lives, since doing so stands to benefit the incalculably large number of people in later generations who will far outweigh existing populations.1 Taken to the extreme, this view suggests it would be morally permissible, or even required, to actively neglect, harm or destroy large swathes of humanity as …. (shrink)

Recent claims that artificial intelligence (AI) systems demonstrate “emergent abilities” have fueled excitement but also fear grounded in the prospect that such systems may enable a wider range of parties to make unprecedented advances in areas that include the development of chemical or biological weapons. Ambiguity surrounding the term “emergent abilities” has added avoidable uncertainty to a topic that has the potential to destabilize the strategic landscape, including the perception of key parties about the viability of nonproliferation efforts. To avert (...) these problems in the future, scientists, developers, policymakers, and other parties should take credible steps to strengthen the health of the scientific ecosystem around AI. (shrink)

Climate change is often described as an existential risk to the human species, but this terminology has generally been avoided in the climate-justice literature in analytic philosophy. I investigate the source of this disconnect and explore the prospects for incorporating the idea of climate change as an existential risk into debates about climate justice. The concept of existential risk does not feature prominently in these discussions, I suggest, because assumptions that structure ‘ideal’ accounts of climate justice ensure that the prospect (...) of climate change as an extinction-level threat does not arise. Given persistent noncompliance with mitigation duties, however, we have reason to revisit these assumptions. I argue that the most promising way for theories of climate justice to account for the significance of existential risk is to look to the practices of protest and resistance in which a concern about extinction or global catastrophe is frequently given expression. (shrink)

When our choice affects some other person and the outcome is unknown, it has been argued that we should defer to their risk attitude, if known, or else default to use of a risk‐avoidant risk function. This, in turn, has been claimed to require the use of a risk‐avoidant risk function when making decisions that primarily affect future people, and to decrease the desirability of efforts to prevent human extinction, owing to the significant risks associated with continued human survival. I (...) raise objections to the claim that respect for others' risk attitudes requires risk‐avoidance when choosing for future generations. In particular, I argue that there is no known principle of interpersonal aggregation that yields acceptable results in variable population contexts and is consistent with a plausible ideal of respect for others' risk attitudes in fixed population cases. (shrink)

Longtermism is the view that the most urgent global priorities, and those to which we should devote the largest portion of our resources, are those that focus on (i) ensuring a long future for humanity, and perhaps sentient or intelligent life more generally, and (ii) improving the quality of the lives that inhabit that long future. While it is by no means the only one, the argument most commonly given for this conclusion is that these interventions have greater expected goodness (...) per unit of resource devoted to them than each of the other available interventions, including those that focus on the health and well-being of the current population. In this paper, I argue that, even if we grant the consequentialist ethics upon which this argument depends, and even if we grant one of the axiologies that are typically paired with that ethics to give the argument, we are not morally required to choose an option that maximises expected utility; indeed, we might not even be permitted to do so. Instead, I will argue, if the argument's consequentialism is correct, we should choose using a decision theory that is sensitive to risk, and allows us to give greater weight to worse-case outcomes than expected utility theory does. And, I will show, such decision theories do not always recommend longtermist interventions. Indeed, sometimes, they recommend exactly the opposite: sometimes, they recommend hastening human extinction. Many, though not all, will take this as a reductio of the consequentialism or the axiology of the argument. I remain agnostic on the conclusion we should draw. (shrink)

Rawls famously argues that the parties in the original position would agree upon the two principles of justice. Among other things, these principles guarantee equal political liberty—that is, democracy—as a requirement of justice. We argue on the contrary that the parties have reason to reject this requirement. As we show, by Rawls’ own lights, the parties would be greatly concerned to mitigate existential risk. But it is doubtful whether democracy always minimizes such risk. Indeed, no one currently knows which political (...) systems would. Consequently, the parties—and we ourselves—have reason to reject democracy as a requirement of justice in favor of political experimentalism, a general approach to political justice which rules in at least some non-democratic political systems which might minimize existential risk. (shrink)

In this commentary on Darrel Moellendorf’s Mobilizing Hope: Climate Change & Global Poverty (Oxford: Oxford University Press, 2022), I discuss his use of the precautionary principle, whether his hope for climate-friendly ‘green growth’ is realistic given the tendency for inequality to accelerate as it gets higher, and what I call his assumption of a liberal baseline. That is, I worry that the audience to whom the book is addressed are those who already accept the environmental and economic values to which (...) Moellendorf appeals, while those who are blocking effective action on climate change in fact reject those values and are therefore unlikely to be moved by the arguments the book presents. (shrink)

I defend the claim that life-suspending technologies can constitute a catastrophic and existential security factor for risks structurally similar to those related to climate change. The gist of the argument is that, under certain conditions, life-suspending technologies such as cryonics can provide self-interested actors with incentives to efficiently tackle such risks—in particular, they provide reasons to overcome certain manifestations of generational egoism, a risk factor of several catastrophic and existential risks. Provided we have reasons to decrease catastrophic and existential risks (...) such as climate change, we also have a (defeasible) reason for investing in developing and making life-suspending technologies (more) widespread. (shrink)

Why, if at all, should we object to economic inequality? Some central arguments – the argument from decreasing marginal utility for example – invoke instrumental reasons and object to inequality because of its effects. Such instrumental arguments, however, often concern only the static effects of inequality and neglect its intertemporal consequences. In this article, we address this striking gap and investigate income inequality's intertemporal consequences, including its potential effects on humanity's (very) long-term future. Following recent arguments around future generations and (...) so-called longtermism, those effects might arguably matter more than inequality's short-term consequences. We assess whether we have instrumental reason to reduce economic inequality based on its intertemporal effects in the short, medium, and the very long term. We find a good short and medium-term instrumental case for lower economic inequality. We then argue, somewhat speculatively, that we have instrumental reasons for inequality reduction from a longtermist perspective too, primarily because greater inequality could increase existential risk. We thus have instrumental reasons to reduce inequality, regardless of which time-horizon we take. We then argue that from most consequentialist perspectives, this pro tanto reason also gives us all-things-considered reason. And even across most non-consequentialist views in philosophy, this argument gives us either an all-things-considered or at least weighty pro tanto reason against inequality. (shrink)

Is the overall value of a world just the sum of values contributed by each value-bearing entity in that world? Additively separable axiologies (like total utilitarianism, prioritarianism, and critical level views) say 'yes', but non-additive axiologies (like average utilitarianism, rank-discounted utilitarianism, and variable value views) say 'no'. This distinction appears to be practically important: among other things, additive axiologies generally assign great importance to large changes in population size, and therefore tend to strongly prioritize the long-term survival of humanity over (...) the interests of the present generation. Non-additive axiologies, on the other hand, need not assign great importance to large changes in population size. We show, however, that when there is a large enough `background population' unaffected by our choices, a wide range of non-additive axiologies converge in their implications with additive axiologies—for instance, average utilitarianism converges with critical-level utilitarianism and various egalitarian theories converge with prioritarianism. We further argue that real-world background populations may be large enough to make these limit results practically significant. This means that arguments from the scale of potential future populations for the astronomical importance of avoiding existential catastrophe, and other arguments in practical ethics that seem to presuppose additive separability, may succeed in practice whether or not we accept additive separability as a basic axiological principle. (shrink)

Longtermists have recently argued that it is overwhelmingly important to do what we can to mitigate existential risks to humanity. I consider three mistakes that are often made in calculating the value of existential risk mitigation. I show how correcting these mistakes pushes the value of existential risk mitigation substantially below leading estimates, potentially low enough to threaten the normative case for existential risk mitigation. I use this discussion to draw four positive lessons for the study of existential risk. -/- (...) . (shrink)

It is plausible that current generations owe something to future generations. One possibility is that we have a duty to not harm them. Another possibility is that we have a duty to protect them. In either case, however, to satisfy the duties to future generations from environmental or political degradation, we need to engage in widespread collective action. But, as we are, we have a limited ability to do so, in part because we lack the self-discipline necessary for successful collective (...) action. Given that having an obligation hinges on being able to satisfy it, the apparent duty to future generations is at odds with our apparent inability to satisfy it. Thus, we either need to rethink our duties to future generations or rethink the nature of our abilities. I argue in this chapter that we should rethink the nature of our abilities. Specifically, we should enhance our self-discipline, which is distinct from enhancements of motivation, judgment, or emotion. Even the most intelligent, motivated, and empathetic actor is likely to slip up, which encourages others to withdraw their cooperation. Without the necessary self-discipline to sustain our public cooperative behavior, collective action is likely to fail, leaving us with one alternative: to abandon our duties to future generations. (shrink)

How can one build AI systems such that they pursue the goals their designers want them to pursue? This is the alignment problem. Numerous authors have raised concerns that, as research advances and systems become more powerful over time, misalignment might lead to catastrophic outcomes, perhaps even to the extinction or permanent disempowerment of humanity. In this paper, I analyze the severity of this risk based on current instances of misalignment. More specifically, I argue that contemporary large language models and (...) game-playing agents are sometimes misaligned. These cases suggest that misalignment tends to have a variety of features: misalignment can be hard to detect, predict and remedy, it does not depend on a specific architecture or training paradigm, it tends to diminish a system’s usefulness and it is the default outcome of creating AI via machine learning. Subsequently, based on these features, I show that the risk of AI alignment magnifies with respect to more capable systems. Not only might more capable systems cause more harm when misaligned, aligning them should be expected to be more difficult than aligning current AI. (shrink)

Recent advances in natural language processing have given rise to a new kind of AI architecture: the language agent. By repeatedly calling an LLM to perform a variety of cognitive tasks, language agents are able to function autonomously to pursue goals specified in natural language and stored in a human-readable format. Because of their architecture, language agents exhibit behavior that is predictable according to the laws of folk psychology: they function as though they have desires and beliefs, and then make (...) and update plans to pursue their desires given their beliefs. We argue that the rise of language agents significantly reduces the probability of an existential catastrophe due to loss of control over an AGI. This is because the probability of such an existential catastrophe is proportional to the difficulty of aligning AGI systems, and language agents significantly reduce that difficulty. In particular, language agents help to resolve three important issues related to aligning AIs: reward misspecification, goal misgeneralization, and uninterpretability. (shrink)

ExcerptRecent advances in the functional power of artificial intelligence (AI) have prompted an urgent warning from industry leaders and researchers concerning its “profound risks to society and humanity.”1 Their open letter is admirable not only for its succinct identification of said risks, which include the mass dissemination of misinformation, loss of jobs, and even the possible extinction of our species, but also for its clear normative framing of the problem: “Should we let machines flood our information channels with propaganda and (...) untruth? Should we automate away all the jobs, including the fulfilling ones? Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization?”2. (shrink)

According to an intriguing though somewhat enigmatic line of thought first proposed by Jonathan Bennett, if humanity went extinct any time soon this would be unfortunate because important business would be left unfinished. This line of thought remains largely unexplored. I offer an interpretation of the idea that captures its intuitive appeal, is consistent with plausible constraints, and makes it non-redundant to other views in the literature. The resulting view contrasts with a welfare-promotion perspective, according to which extinction would be (...) bad because it would come at a massive opportunity cost in terms of feasibly achievable welfare over the lifetime of our species. (shrink)

Our Earth’s climate is changing. Any species living in the Earth’s ecosystem need to thrive to adapt to the new living conditions. Otherwise, extinction will be their outcome. In the race for adaptation, waterbirds (Aequorlitornithes), such as penguins, cormorants, and alcids, seem disadvantageous.

Nuclear weapons are different from every other type of weapons technology. Their awesome destructive potential and the unparalleled consequences of their use oblige us to think critically about the ethics of nuclear possession, planning, and use. Joe Nye has given the ethics of nuclear weapons deep consideration. He posits that we have a basic moral obligation to future generations to preserve roughly equal access to important values, including equal chances of survival, and proposes criteria for achieving conditional or “just deterrence” (...) to minimize the risk of nuclear use and help preserve these values. While Nye's conditions are laudable, they are not achievable. They rely on flawed assumptions about the nature of nuclear weapons and the inherent risks of the nuclear deterrence system. Since the Cold War ended, the strategy and practice of nuclear deterrence has grown riskier, more urgent, more dangerous, and less stable. It is time to rethink how we manage nuclear risks. A new nuclear security system must be built on the design principle that the consequences of system failure cannot threaten to end or fundamentally disrupt civilization. We owe the future a new nuclear security strategy that can prevent an existential global nuclear event. (shrink)

Why, if at all, should we object to economic inequality? Some central arguments – the argument from decreasing marginal utility for example – invoke instrumental reasons and object to inequality because of its effects. Such instrumental arguments, however, often concern only the static effects of inequality and neglect its intertemporal conse- quences. In this article, we address this striking gap and investigate income inequality’s intertemporal consequences, including its potential effects on humanity’s (very) long-term future. Following recent arguments around future generations (...) and so-called longtermism, those effects might arguably matter more than inequality’s short-term con- sequences. We assess whether we have instrumental reason to reduce economic inequality based on its intertemporal effects in the short, medium, and the very long term. We find a good short and medium-term instrumental case for lower economic inequality. We then argue, somewhat speculatively, that we have instrumental reasons for inequality reduction from a longtermist perspective too, primarily because greater inequality could increase existential risk. We thus have instrumental reasons to reduce inequality, regardless of which time-horizon we take. We then argue that from most consequentialist perspectives, this pro tanto reason also gives us all-things-considered reason. And even across most non-consequentialist views in philosophy, this argument gives us either an all-things-considered or at least weighty pro tanto reason against inequality. (shrink)

Longtermists claim that what we ought to do is mainly determined by how our actions might affect the very long-run future. A natural objection to longtermism is that these effects may be nearly impossible to predict — perhaps so close to impossible that, despite the astronomical importance of the far future, the expected value of our present actions is mainly determined by near-term considerations. This paper aims to precisify and evaluate one version of this epistemic objection to longtermism. To that (...) end, I develop two simple models for comparing ‘longtermist’ and ‘neartermist’ interventions, incorporating the idea that it is harder to make a predictable difference to the further future. These models yield mixed conclusions: if we simply aim to maximize expected value, and don’t mind premising our choices on minuscule probabilities of astronomical payoffs, the case for longtermism looks robust. But on some prima facie plausible empirical worldviews, the expectational superiority of longtermist interventions depends heavily on these ‘Pascalian’ probabilities. So the case for longtermism may depend either on plausible but non-obvious empirical claims or on a tolerance for Pascalian fanaticism. (shrink)

Philosophy &Public Affairs, Volume 51, Issue 4, Page 373-412, Fall 2023.

In this article I make two main critiques of Kaczmarek and Beard's article ‘Human Extinction and Our Obligations to the Past’. First, I argue that there is an ambiguity in what it means to realise the benefits of a sacrifice and that this ambiguity affects the persuasiveness of the authors’ arguments and responses to various objections to their view. Second, I argue that their core argument against human extinction depends on an unsupported assumption about the existence and importance of existential (...) benefits. (shrink)