Because of accelerating technological progress, humankind may be rapidly approaching a critical phase in its career. In addition to well-known threats such as nuclear holocaust, the propects of radically transforming technologies like nanotech systems and machine intelligence present us with unprecedented opportunities and risks. Our future, and whether we will have a future at all, may well be determined by how we deal with these challenges. In the case of radically transforming technologies, a better understanding of the transition dynamics from (...) a human to a "posthuman" society is needed. Of particular importance is to know where the pitfalls are: the ways in which things could go terminally wrong. While we have had long exposure to various personal, local, and endurable global hazards, this paper analyzes a recently emerging category: that of existential risks. These are threats that could case our extinction or destroy the potential of Earth - originating intelligent life. Some of these threats are relatively well known while others, including some of the gravest, have gone almost unrecognized. Existential risks have a cluster of features that make ordinary risk management ineffective. A final section of this paper discusses several ethical and policy implications. A clearer understanding of the threat picture will enable us to formulate better strategies. (shrink)

How much value can our decisions create? We argue that unless our current understanding of physics is wrong in fairly fundamental ways, there exists an upper limit of value relevant to our decisions. First, due to the speed of light and the definition and conception of economic growth, the limit to economic growth is a restrictive one. Additionally, a related far larger but still finite limit exists for value in a much broader sense due to the physics of information and (...) the ability of physical beings to place value on outcomes. We discuss how this argument can handle lexicographic preferences, probabilities, and the implications for infinite ethics and ethical uncertainty. (shrink)

The future of humanity is often viewed as a topic for idle speculation. Yet our beliefs and assumptions on this subject matter shape decisions in both our personal lives and public policy – decisions that have very real and sometimes unfortunate consequences. It is therefore practically important to try to develop a realistic mode of futuristic thought about big picture questions for humanity. This paper sketches an overview of some recent attempts in this direction, and it offers a brief discussion (...) of four families of scenarios for humanity’s future: extinction, recurrent collapse, plateau, and posthumanity. (shrink)

Metasystem transitions are events representing the evolutionary emergence of a higher level of organization through the integration of subsystems into a higher “metasystem”. Such events have occurred several times throughout the history of life. The emergence of new levels of organization has occurred within the human system three times; and has resulted in three broadly defined levels of higher control; producing three broadly defined levels of group selection. These are “Human Metasystem Transitions”. Throughout these HMST several common system-level patterns have (...) manifested that are fundamental to understanding the nature and evolution of the human system; as well as our potential future development. First; HMST have been built around the control of three mostly distinct primary energy sources. Second; the control of new energy sources has always been achieved and stabilized by utilizing the evolutionary emergence of a more powerful information-processing medium. Third; new controls emerge with the capability of organizing energy flows over larger expanses of space in shorter durations of time: bands/tribes controlled regional space and stabilized for hundreds of thousand of years; chiefdoms/kingdoms controlled semi-continental expanses of space and stabilized for thousands of years; and nation-states control continental expanses of space and have stabilized for centuries. This space-time component of hierarchical metasystem emergence can be conceptualized as the active compression of space-time-energy-matter enabled by higher informational and energetic properties within the human system; which allow for more complex organization. In this framework; increased information-energy control and feedback; and the consequent metasystem compression of space-time; represent the theoretical pillars of HMST theory. Most importantly; HMST theory may have practical application in modeling the future of the human system and the nature of the next human metasystem. (shrink)

A simple exogenous growth model gives conservative estimates of the economic implications of machine intelligence. Machines complement human labor when they become more productive at the jobs they perform, but machines also substitute for human labor by taking over human jobs. At first, expensive hardware and software does only the few jobs where computers have the strongest advantage over humans. Eventually, computers do most jobs. At first, complementary effects dominate, and human wages rise with computer productivity. But eventually substitution can (...) dominate, making wages fall as fast as computer prices now do. An intelligence population explosion makes per-intelligence consumption fall this fast, while economic growth rates rise by an order of magnitude or more. These results are robust to automating incrementally, and to distinguishing hardware, software, and human capital from other forms of capital. (shrink)

Humans have slowly built more productive societies by slowly acquiring various kinds of capital, and by carefully matching them to each other. Because disruptions can disturb this careful matching, and discourage social coordination, large disruptions can cause a “social collapse,” i.e., a reduction in productivity out of proportion to the disruption. For many types of disasters, severity seems to follow a power law distribution. For some of types, such as wars and earthquakes, most of the expected harm is predicted to (...) occur in extreme events, which kill most people on Earth. So if we are willing to worry about any war or earthquake, we should worry especially about extreme versions. If individuals varied little in their resistance to such disruptions, events a little stronger than extreme ones would eliminate humanity, and our only hope would be to prevent such events. If individuals vary a lot in their resistance, however, then it may pay to increase the variance in such resistance, such as by creating special sanctuaries from which the few remaining humans could rebuild society. (shrink)