In S. Matthew Liao (ed.), Ethics of Artificial Intelligence. New York, USA: Oxford University Press (2020)

Steve Petersen
Niagara University
Whether it would take one decade or several centuries, many agree that it is possible to create a *superintelligence*---an artificial intelligence with a godlike ability to achieve its goals. And many who have reflected carefully on this fact agree that our best hope for a "friendly" superintelligence is to design it to *learn* values like ours, since our values are too complex to program or hardwire explicitly. But the value learning approach to AI safety faces three particularly philosophical puzzles: first, it is unclear how any intelligent system could learn its final values, since to judge one supposedly "final" value against another seems to require a further background standard for judging. Second, it is unclear how to determine the content of a system's values based on its physical or computational structure. Finally, there is the distinctly ethical question of which values we should best aim for the system to learn. I outline a potential answer to these interrelated puzzles, centering on a "miktotelic" proposal for blending a complex, learnable final value out of many simpler ones.
Keywords ethics of ai  value alignment  ai safety  superintelligence  ethical rationalism  specificationism  coherence
Categories (categorize this paper)
Buy the book Find it on
Edit this record
Mark as duplicate
Export citation
Find it on Scholar
Request removal from index
Revision history

Download options

PhilArchive copy

 PhilArchive page | Other versions
External links

Setup an account with your affiliations in order to access resources via your University's proxy server
Configure custom proxy (use this if your affiliation does not provide a proxy)
Through your library

References found in this work BETA

No references found.

Add more references

Citations of this work BETA

Add more citations

Similar books and articles

Superintelligence as Superethical.Steve Petersen - 2017 - In Patrick Lin, Keith Abney & Ryan Jenkins (eds.), Robot Ethics 2. 0: New Challenges in Philosophy, Law, and Society. New York, USA: Oxford University Press. pp. 322-337.
Computation, Coherence, and Ethical Reasoning.Marcello Guarini - 2007 - Minds and Machines 17 (1):27-46.
Ethical Issues in Teaching.Michael Cholbi - 2013 - International Encyclopedia of Ethics.
Intention, Plans, and Ethical Rationalism.Kieran Setiya - 2014 - In Manuel Vargas & Gideon Yaffe (eds.), Rational and Social Agency: The Philosophy of Michael Bratman. Oxford University Press. pp. 56-82.
Superintelligence as a Cause or Cure for Risks of Astronomical Suffering.Kaj Sotala & Lukas Gloor - 2017 - Informatica: An International Journal of Computing and Informatics 41 (4):389-400.
Ethical Life Cycle of an Innovation.Mari Meel & Maksim Saat - 2002 - Journal of Business Ethics 39 (1-2):21 - 27.


Added to PP index

Total views
317 ( #32,463 of 2,499,061 )

Recent downloads (6 months)
52 ( #15,971 of 2,499,061 )

How can I increase my downloads?


My notes