Abstract

Two theses are proposed; regarding the future evolution of the value systems of advanced AGI systems. The Value Learning Thesis is a semi-formalized version of the idea that; if an AGI system is taught human values in an interactive and experiential way as its intelligence increases toward human level; it will likely adopt these human values in a genuine way. The Value Evolution Thesis is a semi-formalized version of the idea that if an AGI system begins with human-like values; and then iteratively modifies itself; it will end up in roughly the same future states as a population of human beings engaged with progressively increasing their own intelligence. Taken together; these theses suggest a worldview in which raising young AGIs to have human-like values is a sensible thing to do; and likely to produce a future that is generally desirable in a human sense. While these two theses are far from definitively proven; I argue that they are more solid and more relevant to the actual future of AGI than Bostrom’s “Instrumental Convergence Thesis” and “Orthogonality Thesis” which are core to the basis of his argument for fearing ongoing AGI development and placing AGI R&D under strict governmental control. In the context of fleshing out this argument; previous publications and discussions by Richard Loosemore and Kaj Sotala are discussed in some detail.