The thinkers who have reflected on the problem of a coming superintelligence have generally seen the issue as a technological problem, a problem of how to control what the superintelligence will do. I argue that this approach is probably mistaken because it is based on questionable assumptions about the behavior of intelligent agents and, moreover, potentially counterproductive because it might, in the end, bring about the existential catastrophe that it is meant to prevent. I contend that the problem posed by (...) a future superintelligence will likely be a political problem, that is, one of establishing a peaceful form of coexistence with other intelligent agents in a situation of mutual vulnerability, and not a technological problem of control. (shrink)

I survey four categories of factors that might give a digital mind, such as an upload or an artificial general intelligence, an advantage over humans. Hardware advantages include greater serial speeds and greater parallel speeds. Self-improvement advantages include improvement of algorithms, design of new mental modules, and modification of motivational system. Co-operative advantages include copyability, perfect co-operation, improved communication, and transfer of skills. Human handicaps include computational limitations and faulty heuristics, human-centric biases, and socially motivated cognition. The shape of hardware (...) growth curves, as well as the ease of modifying minds, are found to have a major impact on how quickly a digital mind may take advantage of these factors. (shrink)

Humans will not always be the most intelligent agents on Earth, the ones steering the future. What will happen to us when we no longer play that role, and how can we prepare for this transition?

Abstract. Could a person or mind be uploaded—transmitted to a computer or network—and thereby survive bodily death? I argue ‘mind uploading’ is possible only if a mind is an abstract object rather than a concrete particular. Two implications are notable. One, if someone can be uploaded someone can be multiply-instantiated, such that there could be as many instances of a person as copies of a book. Second, mind uploading’s possibility is incompatible with the leading theories of personal identity, insofar as (...) these assume the mind is a concrete particular. Moreover, because David Chalmers (2010; 2012; 2014) defends mind uploading without construing minds as abstract, I show Chalmers’ argument to be unsound. (shrink)

The ‘Mind-Upload’ hypothesis, a radical version of the Brain-in-a-Vat thought experiment, asserts that a whole mind can safely be transferred from a brain to a digital device, after being exactly encoded into substrate independent informational patterns. Prima facie, MU seems the philosophical archenemy of the Embodied Mind theory, which understands embodiment as a necessary and constitutive condition for the existence of a mind and its functions. In truth, whether and why MU and EM are ultimately incompatible is unobvious. This paper, (...) which aims to answer both questions, will not simply confirm that MU and EM actually are incompatible. It will also show the true reason of their incompatibility: while EM implies that a mind’s individual identity is contingent upon the details of its physical constituents, MU presupposes that minds can be relocated from one material vessel to another. A systematic comparison between these conflicting assumptions reveals that the real shortcoming of MU is not the one usually discussed by the philosophical literature: it has nothing to do with MU’s functionalist or computationalist prerequisites, and is only secondarily related to the artificial implementability of consciousness; the real problem is that MU presupposes that minds could still be individuated and numerically identified while being reduced to immaterial formal patterns. EM seems committed to refute this assumption, but does it have sufficient resources to succeed? (shrink)

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may inflict serious damage to human well-being on a global scale ('catastrophic risk'). After summarizing the arguments for why AGI may pose such a risk, we review the fieldʼs proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors and proposals for creating AGIs that are safe due to (...) their internal design. (shrink)

It has become fashionable to worry about the development of superintelligent AI that results in the destruction of humanity. This worry is not without merit; but it may be overstated. This paper explores some previously undiscussed reasons to be optimistic that; even if superintelligent AI does arise; it will not destroy us. These have to do with the possibility that a superintelligent AI will become mired in skeptical worries that its superintelligence cannot help it to solve. I argue that superintelligent (...) AIs may lack the psychological idiosyncracies that allow humans to act in the face of skeptical problems; and so as a result they may become paralyzed in the face of these problems in a way that humans are not. (shrink)