The moral psychology of AI and the ethical opt-out problem

Abstract

This chapter discusses the limits of normative ethics in new moral domains linked to the development of Artificial Intelligence. In these new domains, people have the possibility to opt out of using a machine, if they do not approve of the ethics that the machine is programmed to follow. In other words, even if normative ethics could determine the best moral programs, these programs would not be adopted (and thus have no positive impact) if they clashed with users' preferences— a phenomenon that we label the ‘Ethical Opt-Out.’ The chapter then explores various ways in which the field of moral psychology can illuminate public perception of moral AI, and inform the regulations of such AI. The chapter’s main focus is on autonomous vehicles, but it also explores the role of psychological science for the study of other moral algorithms.