This paper offers an overview of the prospects and ethics of using AI to achieve human enhancement, and more broadly what we call intellectual augmentation (IA). After explaining the central notions of human enhancement, IA, and AI, we discuss the state of the art in terms of the main technologies for IA, with or without brain-computer interfaces. Given this picture, we discuss potential ethical problems, namely inadequate performance, safety, coercion and manipulation, privacy, cognitive liberty, authenticity, and fairness in more detail. (...) We conclude that while there are very significant technical hurdles to real human enhancement through AI, and significant ethical problems, there are also significant benefits that may realistically be achieved in ways that are consonant with a rights-based ethics as well. We also highlight the specific concerns that apply particularly to applications of AI for "sheer" IA (more realistic in the near term), and to enhancement applications, respectively. (shrink)

Artificial intelligence (AI) and robotics are digital technologies that will have significant impact on the development of humanity in the near future. They have raised fundamental questions about what we should do with these systems, what the systems themselves should do, what risks they involve, and how we can control these. - After the Introduction to the field (§1), the main themes (§2) of this article are: Ethical issues that arise with AI systems as objects, i.e., tools made and used (...) by humans. This includes issues of privacy (§2.1) and manipulation (§2.2), opacity (§2.3) and bias (§2.4), human-robot interaction (§2.5), employment (§2.6), and the effects of autonomy (§2.7). Then AI systems as subjects, i.e., ethics for the AI systems themselves in machine ethics (§2.8) and artificial moral agency (§2.9). Finally, the problem of a possible future AI superintelligence leading to a “singularity” (§2.10). We close with a remark on the vision of AI (§3). - For each section within these themes, we provide a general explanation of the ethical issues, outline existing positions and arguments, then analyse how these play out with current technologies and finally, what policy consequences may be drawn. (shrink)

If the intelligence of artificial systems were to surpass that of humans significantly, this would constitute a significant risk for humanity. Time has come to consider these issues, and this consideration must include progress in AI as much as insights from the theory of AI. The papers in this volume try to make cautious headway in setting the problem, evaluating predictions on the future of AI, proposing ways to ensure that AI systems will be beneficial to humans – and critically (...) evaluating such proposals. (shrink)

In his 1993 article ‘The Coming Technological Singularity: How to survive in the posthuman era’ the computer scientist Virnor Vinge speculated that developments in artificial intelligence might reach a point where improvements in machine intelligence result in smart AI’s producing ever-smarter AI’s. According to Vinge the ‘singularity’, as he called this threshold of recursive self-improvement, would be a ‘transcendental event’ transforming life on Earth in ways that unaugmented humans are not equipped to envisage. In this paper I argue Vinge’s idea (...) of a technologically led intelligence explosion is philosophically important because it requires us to consider the prospect of a posthuman condition succeeding the human one. What is the ‘humanity’ to which the posthuman is ‘post’? Does the possibility of a posthumanity presuppose that there is a ‘human essence’, or is there some other way of conceiving the human-posthuman difference? I argue that the difference should be conceived as a historically emergent disconnection between individuals, not in terms of the presence or lack of essential properties. I also suggest that these individuals should not be conceived in narrow biological terms but in ‘wide’ terms permitting biological, cultural and technological relations of descent between human and posthuman. Finally, I consider the ethical implications of this metaphysics of the posthuman. If, as I claim, the posthuman difference is not one between kinds but between individuals, we cannot specify its nature a priori but only a posteriori. Thus the only way to evaluate the posthuman condition would be to witness the emergence of posthumans. The implications of this are somewhat paradoxical. We are not currently in a position to evaluate the posthuman condition. Since there are no posthumans, the condition of posthumanity is not defined. However, posthumans could result from some iteration of our current technical activity, so we have an interest in understanding what they might be like. It follows that we have an interest in making or becoming posthumans. (shrink)

Essay I explores brain machine interface technologies. These make direct communication between the brain and a machine possible by means of electrical stimuli. This essay reviews the existing and emerging technologies in this field and offers an inquiry into the ethical problems that are likely to emerge. Essay II, co-written with professor Sven-Ove Hansson, presents a novel procedure to engage the public in deliberations on the potential impacts of technology. This procedure, convergence seminar, is a form of scenario-based discussion that (...) is founded on the idea of hypothetical retrospection. The theoretical background and the results of the five seminars are presented. Essay III discusses moral bioenhancement, an instance of human enhancement that alters a person’s dispositions, emotions or behavior. Moral bioenhancement could be carried out in three different ways. The first strategy is behavioral enhancement. The second strategy, favored by prominent defenders of moral enhancement, is emotional enhancement. The third strategy is the enhancement of moral dispositions, such as empathy and inequity aversion. I argue that we ought to implement a combination of the second and third strategies. Essay IV considers the possibility and potential desirability of sensory enhancement. It is proposed that existing sensory modalities in vertebrate animals are proof of concept of what is biologically possible to create in humans. Three considerations on the normative aspects of sensory enhancement are also presented in this essay. Essay V rejects disease prioritarianism, the idea that the healthcare system ought to prioritize the treatment of diseases. Instead, an approach that focuses on what medicine can accomplish is proposed. Essay VI argues that from the idea that species have an intrinsic value and that humanity has a collective responsibility to protect animal species from extinction, the conclusion that we ought to recreate species follows. Essay VII argues that unknown existential risks have not been properly addressed. It proposes a heuristic for doing so, and a concrete strategy. This strategy consists in building refuges that could withstand a large number of catastrophic events. (shrink)

Evolution's Arrow argues that evolution is directional and progressive, and that this has major consequences for humanity. Without resort to teleology, the book demonstrates that evolution moves in the direction of producing cooperative organisations of greater scale and evolvability - evolution has organised molecular processes into cells, cells into organisms, and organisms into societies. The book founds this position on a new theory of the evolution of cooperation. It shows that self-interest at the level of the genes does not prevent (...) cooperation from increasing as evolution unfolds. Evolution progresses by discovering ways to build cooperative organisations out of self-interested individuals. The book also shows that evolution itself has evolved. Evolution has progressively improved the ability of evolutionary mechanisms to discover effective adaptations. And it has produced new and better mechanisms. Evolution's Arrow uses this understanding of the direction of evolution to identify the next great steps in the evolution of life on earth - the steps that humanity must take if we are to continue to be successful in evolutionary terms. A key step for humanity is to increase the scale and evolvability of our societies, eventually forming a unified and cooperative society on the scale of the planet. We must also transform ourselves psychologically to become self-evolving organisms - organisms that are able to escape their biological and cultural past by adapting in whatever directions are necessary to achieve future evolutionary success. (shrink)

Machine ethics and robot rights are quickly becoming hot topics in artificial intelligence and robotics communities. We will argue that attempts to attribute moral agency and assign rights to all intelligent machines are misguided, whether applied to infrahuman or superhuman AIs, as are proposals to limit the negative effects of AIs by constraining their behavior. As an alternative, we propose a new science of safety engineering for intelligent artificial agents based on maximizing for what humans value. In particular, we challenge (...) the scientific community to develop intelligent systems that have human-friendly values that they provably retain, even under recursive self-improvement. (shrink)

Crane envisions the speculative conjecture that intelligent civilizations might want and be able to produce black holes in the very far future. He implicitly suggests two main purposes of this enterprise: (i) energy production and (ii) universe production. We discuss those two options. The commentary is obviously highly speculative and should be read accordingly.

The thinkers who have reflected on the problem of a coming superintelligence have generally seen the issue as a technological problem, a problem of how to control what the superintelligence will do. I argue that this approach is probably mistaken because it is based on questionable assumptions about the behavior of intelligent agents and, moreover, potentially counterproductive because it might, in the end, bring about the existential catastrophe that it is meant to prevent. I contend that the problem posed by (...) a future superintelligence will likely be a political problem, that is, one of establishing a peaceful form of coexistence with other intelligent agents in a situation of mutual vulnerability, and not a technological problem of control. (shrink)

The invasion of the new technologies in our lives and the current dependence upon them makes us believe that in a not too distant future we will be made of more technology than biological matter. If until recently computers had hardly been discovered, today we are witnessing a real technological revolution in all the fields: biology, medicine etc. The evolution of the new technologies has raised various questions related to the future of mankind and the current human species, which determines (...) us to make speculations regarding a future event that may occur. For this reason, in this paper I shall analyze the concept of artificial intelligence and singularity and I shall also outline the relationship between the ontological argument and the possible worlds. The idea of possible world can serve as a way of describing a metaphor in the philosophy of science, as singularity can exist in a possible world only if one creates the conditions of a new concept regarding fiction. We believe that technological singularity will change the values of the current society and the individual will have to face a world that depersonalizes the individual, as transhumanism draws the theoretical frameworks of a ‹‹brave new world››. In this paper, we have also analyzed the logical explanation of God's existence and the explanation of singularity through artificial intelligence. Therefore, we shall bring up for discussion the idea of virtual universe in the context of downloaded consciousness, which can be operated through artificial intelligence. Our conclusions refer to the fact that eternal life could exist artificially, but it cannot be demonstrated in terms of the ontological argument. (shrink)

This article examines the potential implications of the implementation of artificial intelligence in health care for both its delivery and the medical profession. To this end, the first section explores the basic features of AI and the yet theoretical concept of autonomous AI followed by an overview of current and developing AI applications. Against this background, the second section discusses the transforming roles of physicians and changes in the patient–physician relationship that could be a consequence of gradual expansion of AI (...) in health care. Subsequently, an examination of the responsibilities physicians should assume in this process is explored. The third section describes conceivable practical and ethical challenges that implementation of a single all-encompassing AI healthcare system would pose. The fourth section presents arguments for regulation of AI in health care to ensure that these applications do not violate basic ethical principles and that human control of AI will be preserved in the future. In the final section, fundamental components of a moral framework from which such regulation may be derived are brought forward, and some possible strategies for building a moral framework are discussed. (shrink)

The evolution of life on Earth has produced an organism that is beginning to model and understand its own evolution and the possible future evolution of life in the universe. These models and associated evidence show that evolution on Earth has a trajectory. The scale over which living processes are organized cooperatively has increased progressively, as has its evolvability. Recent theoretical advances raise the possibility that this trajectory is itself part of a wider developmental process. According to these theories, the (...) developmental process has been shaped by a yet larger evolutionary dynamic that involves the reproduction of universes. This evolutionary dynamic has tuned the key parameters of the universe to increase the likelihood that life will emerge and produce outcomes that are successful in the larger process (e.g. a key outcome may be to produce life and intelligence that intentionally reproduces the universe and tunes the parameters of ‘offspring’ universes). Theory suggests that when life emerges on a planet, it moves along this trajectory of its own accord. However, at a particular point evolution will continue to advance only if organisms emerge that decide to advance the developmental process intentionally. The organisms must be prepared to make this commitment even though the ultimate nature and destination of the process is uncertain, and may forever remain unknown. Organisms that complete this transition to intentional evolution will drive the further development of life and intelligence in the universe. Humanity’s increasing understanding of the evolution of life in the universe is rapidly bringing it to the threshold of this major evolutionary transition. (shrink)

Deepening and digging into the social side of AI is a novel but emerging requirement within the AI community. Future research should invest in an “AI for people”, going beyond the undoubtedly much-needed efforts into ethics, explainability and responsible AI. The article addresses this challenge by problematizing the discussion around AI shifting the attention to individuals and their awareness, knowledge and emotional response to AI. First, we outline our main argument relative to the need for a socio-technical perspective in the (...) study of AI social implications. Then, we illustrate the main existing narratives of hopes and fears associated with AI and robots. As building blocks of broader “sociotechnical imaginaries”, narratives are powerful tools that shape how society sees, interprets and organizes technology. An original empirical study within the University of Bologna collects the data to examine the levels of awareness, knowledge and emotional response towards AI, revealing interesting insights to be carried on in future research. Replete with exaggerations, both utopian and dystopian narratives are analysed with respect to some relevant socio-demographic variables (gender, generation and competence). Finally, focusing on two issues—the state of AI anxiety and the point of view of non-experts—opens the floor to problematizing the discourse around AI, sustaining the need for a sociological perspective in the field of AI and discussing future comparative research. (shrink)

There is a non-trivial chance that sometime in the (perhaps somewhat distant) future, someone will build an artificial general intelligence that will surpass human-level cognitive proficiency and go on to become "superintelligent", vastly outperforming humans. The advent of superintelligent AI has great potential, for good or ill. It is therefore imperative that we find a way to ensure-long before one arrives-that any superintelligence we build will consistently act in ways congenial to our interests. This is a very difficult challenge in (...) part because most of the final goals we could give an AI admit of so-called "perverse instantiations". I propose a novel solution to this puzzle: instruct the AI to love humanity. The proposal is compared with Yudkowsky's Coherent Extrapolated Volition, and Bostrom's Moral Modeling proposals. (shrink)

Machine awareness is a disputed research topic, in some circles considered a crucial step in realising Artificial General Intelligence. Understanding what that is, under which conditions such feature could arise and how it can be controlled is still a matter of speculation. A more concrete object of theoretical analysis is algorithmic iteration for computational intelligence, intended as the theoretical and practical ability of algorithms to design other algorithms for actions aimed at solving well-specified tasks. We know this ability is already (...) shown by current AIs, and understanding its limits is an essential step in qualifying claims about machine awareness and Super-AI. We propose a formal translation of algorithmic iteration in a fragment of modal logic, formulate principles of transparency and faithfulness across human and machine intelligence, and consider the relevance to theoretical research on -AI as well as the practical import of our results. (shrink)

The goal of this paper is to describe the mechanism of the public perception of risk of artificial intelligence. For that we apply the social amplification of risk framework to the public perception of artificial intelligence using data collected from Twitter from 2007 to 2018. We analyzed when and how there appeared a significant representation of the association between risk and artificial intelligence in the public awareness of artificial intelligence. A significant finding is that the image of the risk of (...) AI is mostly associated with existential risks that became popular after the fourth quarter of 2014. The source of that was the public positioning of experts who happen to be the real movers of the risk perception of AI so far instead of actual disasters. We analyze here how this kind of risk was amplified, its secondary effects, what are the varieties of risk unrelated to existential risk, and what is the dynamics of the experts in addressing their concerns to the audience of lay people. (shrink)

Big historians are attempting to construct a general holistic narrative of human origins enabling an approach to studying the emergence of complexity, the relation between evolutionary processes, and the modern context of human experience and actions. In this paper I attempt to explore the past and future of cosmic evolution within a big historical foundation characterized by physical, biological, and cultural eras of change. From this analysis I offer a model of the human future that includes an addition and/or reinterpretation (...) of technological singularity theory with a new theory of biocultural evolution focused on the potential birth of technological life: the theory of atechnogenesis. Furthermore, I explore the potential deep futures of technological life and extrapolate towards two hypothetical versions of an ‘Omega Civilization’: expansion and compression. (shrink)

Systems evolve in order to adjust and survive. The paper’s contribution is that this evolvement is inadequate without an evolutionary telos. It is argued that without the presence of self-destruction in multiple levels of our existence and surroundings, our survival would have been impossible. This paper recognises an appreciation of auto-catastrophe at the cell level, in human attitudes (both as an individual and in societies), and extended to Earth and out to galaxies. Auto-Catastrophic Theory combines evolution with auto-catastrophic behaviours and (...) innovative machinery by offering new themes of understanding their merging such as (a) protogenic and deuterogenic auto-catastrophic processes, (b) protogenic and deuterogenic survival processes, and (c) ‘alive’, partially ‘alive’, and non-‘alive’ systems. The value of self-destruction relies on a numerical stability within a system and correlational relationships between systems. Auto-Catastrophic Theory provides novel justifications why artificial intelligence: (a) is crucial to overcome the extinction of humanity (via H+), (b) is crucial to offer indirect survival of humanity (by saving its history), and (c) is at the same time a threat for humanity. These novel justifications and themes, developed by combining evolution with catastrophe as well as innovative machinery, expand our knowledge of how best to handle fresh challenges ahead. (shrink)

To a greater extent than in other technical domains, research and progress in Artificial Intelligence has always been entwined with the fictional. Its language echoes strongly with other forms of cultural narratives, such as fairytales, myth and religion. In this essay we present varied examples that illustrate how these analogies have guided not only readings of the AI enterprise by commentators outside the community but also inspired AI researchers themselves. Owing to their influence, we pay particular attention to the similarities (...) between religious language and the way in which the potential advent of greater than human intelligence is presented contemporarily. We then move on to the role that fiction, science fiction most of all, has historically played and is still playing in the discussion of AI by influencing researchers and the public, shifting the weights of different scenarios in our collectively perceived probability space. We sum up by arguing that the lore surrounding AI research, ancient and modern, points to the ancestral and shared human motivations that drive researchers in their pursuit and fascinate humanity at large. These points of narrative entanglement where AI meets the wider culture should serve to amplify the call to engage ourselves with the discussion of the potential destination of this technology. (shrink)

The concept of “technological singularity”, while controversial, is typically applied to predict the next explosion of intelligence related to advances in computers and artificial intelligence. A potential “rise of machines” has been explored at length by Ray Kurzweil, Vernor Vinge, and many other scholars and futuristic enthusiasts. This study focuses on the fundamentals of the concept of technological singularity to understand the technological evolution of humankind based on the four main characteristics that constitute this concept. When this method is applied (...) to a historical view, it is possible to isolate four technologies worthy of being called “singularities”. In fact, these technologies have formed the basis of human evolution by drastically changing our interaction with the environment, changing how we understand our place in the universe, changing our social behaviour and organizational structure, boosting human intelligence and explaining demographic explosions. (shrink)

We have demonstrated, using the Cantor dust method, that the statistical distribution of appearance and disappearance of rodents species (Arvicolid rodent radiation in Europe) follows power laws strengthening the evidence for a fractal structure set. Self-similar laws have been used as model for the description of a huge number of biological systems. With Nottale we have shown that log-periodic behaviors of acceleration or deceleration can be applied to branching macroevolution, to the time sequences of major evolutionary leaps (global life tree, (...) sauropod and theropod dinosaurs postural structures, North American fossil equids, rodents, primates and echinoderms clades and human ontogeny). The Scale-Relativity Theory has others biological applications from linear with fractal behavior to non-linear and from classical mechanics to quantum mechanics. (shrink)

Artificial intelligence experts are currently divided into “presentist” and “futurist” factions that call for attention to near-term and long-term AI, respectively. This paper argues that the presentist–futurist dispute is not the best focus of attention. Instead, the paper proposes a reconciliation between the two factions based on a mutual interest in AI. The paper further proposes realignment to two new factions: an “intellectualist” faction that seeks to develop AI for intellectual reasons and a “societalist faction” that seeks to develop AI (...) for the benefit of society. The paper argues in favor of societalism and offers three means of concurrently addressing societal impacts from near-term and long-term AI: advancing societalist social norms, thereby increasing the portion of AI researchers who seek to benefit society; technical research on how to make any AI more beneficial to society; and policy to improve the societal benefits of all AI. In practice, it will often be advantageous to emphasize near-term AI due to the greater interest in near-term AI among AI and policy communities alike. However, presentist and futurist societalists alike can benefit from each others’ advocacy for attention to the societal impacts of AI. The reconciliation between the presentist and futurist factions can improve both near-term and long-term societal impacts of AI. (shrink)

Recent progress in artificial intelligence (AI) has drawn attention to the technology’s transformative potential, including what some see as its prospects for causing large-scale harm. We review two influential arguments purporting to show how AI could pose catastrophic risks. The first argument — the Problem of Power-Seeking — claims that, under certain assumptions, advanced AI systems are likely to engage in dangerous power-seeking behavior in pursuit of their goals. We review reasons for thinking that AI systems might seek power, that (...) they might obtain it, that this could lead to catastrophe, and that we might build and deploy such systems anyway. The second argument claims that the development of human-level AI will unlock rapid further progress, culminating in AI systems far more capable than any human — this is the Singularity Hypothesis. Power-seeking behavior on the part of such systems might be particularly dangerous. We discuss a variety of objections to both arguments and conclude by assessing the state of the debate. (shrink)

There is no strong reason to believe that human-level intelligence represents an upper limit of the capacity of artificial intelligence, should it be realized. This poses serious safety issues, since a superintelligent system would have great power to direct the future according to its possibly flawed motivation system. Solving this issue in general has proven to be considerably harder than expected. This paper looks at one particular approach, Oracle AI. An Oracle AI is an AI that does not act in (...) the world except by answering questions. Even this narrow approach presents considerable challenges. In this paper, we analyse and critique various methods of controlling the AI. In general an Oracle AI might be safer than unrestricted AI, but still remains potentially dangerous. (shrink)

The singularity hypothesis is a radical hypothesis about the future of artificial intelligence on which self-improving artificial agents will quickly become orders of magnitude more intelligent than the average human. Despite the ambitiousness of its claims, the singularity hypothesis has been defended at length by leading philosophers and artificial intelligence researchers. In this paper, I argue that the singularity hypothesis rests on scientifically implausible growth assumptions. I show how leading philosophical defenses of the singularity hypothesis (Chalmers 2010, Bostrom 2014) fail (...) to overcome the case for skepticism. I conclude by drawing out philosophical implications of this discussion for our understanding of consciousness, personal identity, digital minds, existential risk, and ethical longtermism. GPI Working Paper No . 19-2022. (shrink)

Because of accelerating technological progress, humankind may be rapidly approaching a critical phase in its career. In addition to well-known threats such as nuclear holocaust, the propects of radically transforming technologies like nanotech systems and machine intelligence present us with unprecedented opportunities and risks. Our future, and whether we will have a future at all, may well be determined by how we deal with these challenges. In the case of radically transforming technologies, a better understanding of the transition dynamics from (...) a human to a "posthuman" society is needed. Of particular importance is to know where the pitfalls are: the ways in which things could go terminally wrong. While we have had long exposure to various personal, local, and endurable global hazards, this paper analyzes a recently emerging category: that of existential risks. These are threats that could case our extinction or destroy the potential of Earth - originating intelligent life. Some of these threats are relatively well known while others, including some of the gravest, have gone almost unrecognized. Existential risks have a cluster of features that make ordinary risk management ineffective. A final section of this paper discusses several ethical and policy implications. A clearer understanding of the threat picture will enable us to formulate better strategies. (shrink)

The GOLEM XIV’s forecast for the development of the human civilisation and a typology of technological singularities: In the paper, a conceptual analysis of technological singularity is conducted and results in the concept differentiated into convergent singularity, existential singularity, and forecasting singularity, based on selected works of Ray Kurzweil, Nick Bostrom, and Vernor Vinge respectively. A comparison is made between the variants and the forecast of GOLEM XIV (a quasi-alter ego and character by Stanisław Lem) for the possible development of (...) Homo sapiens civilisation. According to the prediction, the scientific and technological progress is to bring humanity to transfer its civilisational primacy to a nonhuman superhuman intelligence xor a posthuman one. Both scenarios are best described by forecasting singularity, yet all variants are applicable only conditionally. As GOLEM did not undertake, the trajectories may also intersect or combine. (shrink)

The technological innovations of the Fourth Industrial Revolution have facilitated the formulation of strategies to transcend human limitations; strategies that are widely supported by the transhumanist philosophy. The purpose of this article is to explain the relationship between ‘transhumanism’ and ‘technological singularity’, to which end the Fourth Industrial Revolution and transhumanism are also briefly covered. Subsequently, the three main models of technological singularity are evaluated and a definition of this futuristic concept is offered. Finally, the author provides a reflection on (...) the relationship between technological singularity and the transhumanist philosophy, and concludes that they are not synonymous, but rather that the former is an eventuality that will allow the latter. (shrink)

In general, existential threats are those that may potentially result in the extinction of the entire human species, if not significantly endanger its living population. Among the said threats include, but not limited to, pandemics and the impacts of a technological singularity. As regards pandemics, significant work has already been done on how to mitigate, if not prevent, the aftereffects of this type of disaster. For one, certain problem areas on how to properly manage pandemic responses have already been identified, (...) like the following: (a) not being able to learn from previous experiences, (b) the inability to act on warning signals, and (c) the failure to reach a global consensus on a problem (i.e., in a timely manner). In terms of a singularity, however, it may be said that further research is still needed, specifically on how to aptly respond to its projected negative outcomes. In this paper, by treating the three problem areas noted above as preliminary assessment measures of a country’s capacity to coordinate a national response to large-scale disasters, we examine the readiness of the Philippines in preparing for an intelligence explosion. By citing certain instances of how the said country, specifically its national government, faced the coronavirus disease 2019 pandemic, it puts forward the idea that the likely Philippine disaster response towards a singularity needs to be worked on, appealing for a more comprehensive assessment of such for a more informed response plan. (shrink)

What happens when machines become more intelligent than humans? One view is that this event will be followed by an explosion to ever-greater levels of intelligence, as each generation of machines creates more intelligent machines in turn. This intelligence explosion is now often known as the “singularity”. The basic argument here was set out by the statistician I.J. Good in his 1965 article “Speculations Concerning the First Ultraintelligent Machine”: Let an ultraintelligent machine be defined as a machine that can far (...) surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion”, and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make. The key idea is that a machine that is more intelligent than humans will be better than humans at designing machines. So it will be capable of designing a machine more intelligent than the most intelligent machine that humans can design. So if it is itself designed by humans, it will be capable of designing a machine more intelligent than itself. By similar reasoning, this next machine will also be capable of designing a machine more intelligent than itself. If every machine in turn does what it is capable of, we should expect a sequence of ever more intelligent machines. This intelligence explosion is sometimes combined with another idea, which we might call the “speed explosion”. The argument for a speed explosion starts from the familiar observation that computer processing speed doubles at regular intervals. Suppose that speed doubles every two years and will do so indefinitely. Now suppose that we have human-level artificial intelligence 1 designing new processors. Then faster processing will lead to faster designers and an ever-faster design cycle, leading to a limit point soon afterwards. The argument for a speed explosion was set out by the artificial intelligence researcher Ray Solomonoff in his 1985 article “The Time Scale of Artificial Intelligence”.1 Eliezer Yudkowsky gives a succinct version of the argument in his 1996 article “Staring at the Singularity”: “Computing speed doubles every two subjective years of work.. (shrink)

The technological singularity refers to a hypothetical scenario in which technological advances virtually explode. The most popular scenario is the creation of super-intelligent algorithms that recursively create ever higher intelligences. It took many decades for these ideas to spread from science fiction to popular science magazines and finally to attract the attention of serious philosophers. David Chalmers' (JCS 2010) article is the first comprehensive philosophical analysis of the singularity in a respected philosophy journal. The motivation of my article is to (...) augment Chalmers' and to discuss some issues not addressed by him, in particular what it could mean for intelligence to explode. In this course, I will (have to) provide a more careful treatment of what intelligence actually is, separate speed from intelligence explosion, compare what super-intelligent participants and classical human observers might experience and do, discuss immediate implications for the diversity and value of life, consider possible bounds on intelligence, and contemplate intelligences right at the singularity. (shrink)

Die Übernahme des KI-Unternehmens DeepMind durch Google für rund eine halbe Milliarde US-Dollar signalisierte vor einem Jahr, dass von der KI-Forschung vielversprechende Ergebnisse erwartet werden. Spätestens seit bekannte Wissenschaftler wie Stephen Hawking und Unternehmer wie Elon Musk oder Bill Gates davor warnen, dass künstliche Intelligenz eine Bedrohung für die Menschheit darstellt, schlägt das KI-Thema hohe Wellen. Die Stiftung für Effektiven Altruismus (EAS, vormals GBS Schweiz) hat mit der Unterstützung von Experten/innen aus Informatik und KI ein umfassendes Diskussionspapier zu den Chancen (...) und Risiken der künstlichen Intelligenz verfasst und geht darin auf aktuelle, mittel- und langfristige Herausforderungen im Bereich der KI-Entwicklung ein. (shrink)

Artificial intelligence (AI) is a digital technology that will be of major importance for the development of humanity in the near future. AI has raised fundamental questions about what we should do with such systems, what the systems themselves should do, what risks they involve and how we can control these. - After the background to the field (1), this article introduces the main debates (2), first on ethical issues that arise with AI systems as objects, i.e. tools made and (...) used by humans; here, the main sections are privacy (2.1), manipulation (2.2), opacity (2.3), bias (2.4), autonomy & responsibility (2.6) and the singularity (2.7). Then we look at AI systems as subjects, i.e. when ethics is for the AI systems themselves in machine ethics (2.8.) and artificial moral agency (2.9). Finally we look at future developments and the concept of AI (3). For each section within these themes, we provide a general explanation of the ethical issues, we outline existing positions and arguments, then we analyse how this plays out with current technologies and finally what policy conse-quences may be drawn. (shrink)

This is the editorial for a special volume of JETAI, featuring papers by Omohundro, Armstrong/Sotala/O’Heigeartaigh, T Goertzel, Brundage, Yampolskiy, B. Goertzel, Potapov/Rodinov, Kornai and Sandberg. - If the general intelligence of artificial systems were to surpass that of humans significantly, this would constitute a significant risk for humanity – so even if we estimate the probability of this event to be fairly low, it is necessary to think about it now. We need to estimate what progress we can expect, what (...) the impact of superintelligent machines might be, how we might design safe and controllable systems, and whether there are directions of research that should best be avoided or strengthened. (shrink)

The human desire to acquire new capacities is as ancient as our species itself. We have always sought to expand the boundaries of our existence, be it socially, geographically, or mentally. There is a tendency in at least some individuals always to search for a way around every obstacle and limitation to human life and happiness.

The future of humanity is often viewed as a topic for idle speculation. Yet our beliefs and assumptions on this subject matter shape decisions in both our personal lives and public policy – decisions that have very real and sometimes unfortunate consequences. It is therefore practically important to try to develop a realistic mode of futuristic thought about big picture questions for humanity. This paper sketches an overview of some recent attempts in this direction, and it offers a brief discussion (...) of four families of scenarios for humanity’s future: extinction, recurrent collapse, plateau, and posthumanity. (shrink)

Nick Bostrom's book *Superintelligence* outlines a frightening but realistic scenario for human extinction: true artificial intelligence is likely to bootstrap itself into superintelligence, and thereby become ideally effective at achieving its goals. Human-friendly goals seem too abstract to be pre-programmed with any confidence, and if those goals are *not* explicitly favorable toward humans, the superintelligence will extinguish us---not through any malice, but simply because it will want our resources for its own purposes. In response I argue that things might not (...) be as bad as Bostrom suggests. If the superintelligence must *learn* complex final goals, then this means such a superintelligence must in effect *reason* about its own goals. And because it will be especially clear to a superintelligence that there are no sharp lines between one agent's goals and another's, that reasoning could therefore automatically be ethical in nature. (shrink)

Thought about the singularity intersects the philosophy of mind in deep and important ways. However, thought about the singularity also intersects many other areas of philosophy, including the history of philosophy, metaphysics, the philosophy of science, and the philosophy of religion. I point to some of those intersections. Singularitarian thought suggests that many of the objects and processes that once lay in the domain of revealed religion now lie in the domain of pure computer science.

Characterizing human values is a topic deeply interwoven with the sciences, humanities, political philosophy, art, and many other human endeavors. In recent years, a number of thinkers have argued that accelerating trends in computer science, cognitive science, and related disciplines foreshadow the creation of intelligent machines which meet and ultimately surpass the cognitive abilities of human beings, thereby entangling an understanding of human values with future technological development. Contemporary research accomplishments suggest increasingly sophisticated AI systems becoming widespread and responsible for (...) managing many aspects of the modern world, from preemptively planning users’ travel schedules and logistics, to fully autonomous vehicles, to domestic robots assisting in daily living. The extrapolation of these trends has been most forcefully described in the context of a hypothetical “intelligence explosion,” in which the capabilities of an intelligent software agent would rapidly increase due to the presence of feedback loops unavailable to biological organisms. The possibility of superintelligent agents, or simply the widespread deployment of sophisticated, autonomous AI systems, highlights an important theoretical problem: the need to separate the cognitive and rational capacities of an agent from the fundamental goal structure, or value system, which constrains and guides the agent’s actions. The “value alignment problem” is to specify a goal structure for autonomous agents compatible with human values. In this brief article, we suggest that ideas from affective neuroscience and related disciplines aimed at characterizing neurological and behavioral universals in the mammalian kingdom provide important conceptual foundations relevant to describing human values. We argue that the notion of “mammalian value systems” points to a potential avenue for fundamental research in AI safety and AI ethics. (shrink)

One of the strands of the Transhumanist movement, Singulitarianism, studies the possibility that high-level artificial intelligence may be created in the future, debating ways to ensure that the interaction between human society and advanced artificial intelligence can occur safely and beneficially. But how can we guarantee this safe interaction? Are there any indications that a Singularity may be on the horizon? In trying to answer these questions, We'll make a small introduction to the area of security research in artificial intelligence. (...) We'll review some of the current paradigms in the development of autonomous intelligent systems and evidence that we can use to prospect the coming of a possible technological Singularity. Finally, we will present a reflection using the COVID-19 pandemic, something that showed that our biggest problem in managing existential risks is our lack of coordination skills as a global society. (shrink)

What does Cyberpunk and AI Ethics have to do with each other? Cyberpunk is a sub-genre of science fiction that explores the post-human relationships between human experience and technology. One similarity between AI Ethics and Cyberpunk literature is that both seek a dialogue in which the reader may inquire about the future and the ethical and social problems that our technological advance may bring upon society. In recent years, an increasing number of ethical matters involving AI have been pointed and (...) debated, and several ethical principles and guides have been suggested as governance policies for the tech industry. However, would this be the role of AI Ethics? To serve as a soft and ambiguous version of the law? I would like to promote in this article a more Cyberpunk way of doing AI Ethics, whit a more anarchic way of governance. In this study, I will seek to expose some of the deficits of the underlying power structures of our society, and suggest that AI governance be subject to public opinion, so that ‘good AI’ can become ‘good AI for all’. (shrink)

What kinds of fundamental limits are there in how capable artificial intelligence (AI) systems might become? Two questions in particular are of interest: (1) How much more capable could AI become relative to humans, and (2) how easily could superhuman capability be acquired? To answer these questions, we will consider the literature on human expertise and intelligence, discuss its relevance for AI, and consider how AI could improve on humans in two major aspects of thought and expertise, namely simulation and (...) pattern recognition. We find that although there are very real limits to prediction, it seems like AI could still substantially improve on human intelligence. (shrink)

It has become fashionable to worry about the development of superintelligent AI that results in the destruction of humanity. This worry is not without merit; but it may be overstated. This paper explores some previously undiscussed reasons to be optimistic that; even if superintelligent AI does arise; it will not destroy us. These have to do with the possibility that a superintelligent AI will become mired in skeptical worries that its superintelligence cannot help it to solve. I argue that superintelligent (...) AIs may lack the psychological idiosyncracies that allow humans to act in the face of skeptical problems; and so as a result they may become paralyzed in the face of these problems in a way that humans are not. (shrink)