The purpose of reinforcement learning is to learn an optimal policy in general. However, in 2-players games such as the othello game, it is important to acquire a penalty avoiding policy. In this paper, we focus on formation of a penalty avoiding policy based on the Penalty Avoiding Rational Policy Making algorithm [Miyazaki 01]. In applying it to large-scale problems, we are confronted with the curse of dimensionality. (...) We introduce several ideas and heuristics to overcome the combinational explosion in large-scale problems. First, we propose an algorithm to save the memory by calculation of state transition. Second, we describe how to restrict exploration by two type knowledge; KIFU database and evaluation funcion. We show that our learning player can always defeat against the well-known othello game program KITTY. (shrink)

Reinforcement learning is a kind of machine learning. It aims to adapt an agent to a given environment with a clue to rewards. In general, the purpose of reinforcement learning system is to acquire an optimum policy that can maximize expected reward per an action. However, it is not always important for any environment. Especially, if we apply reinforcement learning system to engineering, environments, we expect the agent to avoid all penalties. In Markov Decision Processes, (...) a pair of a sensory input and an action is called rule. We call a rule penalty if and only if it has a penalty or it can transit to a penalty state where it does not contribute to get any reward. After suppressing all penalty rules, we aim to make a rational policy whose expected reward per an action is larger than zero. In this paper, we propose a suppressing penalty algorithm that can suppress any penalty and get a reward constantly. By applying the algorithm to the tick-tack-toe, its effectiveness is shown. (shrink)

Reinforcement Learning is a kind of machine learning. We know Profit Sharing, the Rational Policy Making algorithm, the Penalty Avoiding Rational Policy Making algorithm and PS-r* to guarantee the rationality in a typical class of the Partially Observable Markov Decision Processes. However they cannot treat continuous state spaces. In this paper, we present a solution to adapt them in continuous state spaces. We give RPM a mechanism to treat continuous state spaces in the (...) environment that has the same type of a reward. We show the effectiveness of the proposed method in numerical examples. (shrink)

Sequential action makes up the bulk of human daily activity, and yet much remains unknown about how people learn such actions. In one motor learning paradigm, the serial reaction time (SRT) task, people are taught a consistent sequence of button presses by cueing them with the next target response. However, the SRT task only records keypress response times to a cued target, and thus it cannot reveal the full time‐course of motion, including predictive movements. This paper describes a mouse movement (...) trajectory SRT task in which the cursor must be moved to a cued location. We replicated keypress SRT results, but also found that predictive movement—before the next cue appears—increased during the experiment. Moreover, trajectory analyses revealed that people developed a centering strategy under uncertainty. In a second experiment, we made prediction explicit, no longer cueing targets. Thus, participants had to explore the response alternatives and learn via reinforcement, receiving rewards and penalties for correct and incorrect actions, respectively. Participants were not told whether the sequence of stimuli was deterministic, nor if it would repeat, nor how long it was. Given the difficulty of the task, it is unsurprising that some learners performed poorly. However, many learners performed remarkably well, and some acquired the full 10‐item sequence within 10 repetitions. Comparing the high‐ and low‐performers’ detailed results in this reinforcement learning (RL) task with the first experiment's cued trajectory SRT task, we found similarities between the two tasks, suggesting that the effects in Experiment 1 are due to predictive, rather than reactive processes. Finally, we found that two standard model‐free reinforcement learning models fit the high‐performing participants, while the four low‐performing participants provide better fit with a simple negative recency bias model. (shrink)

Sequential action makes up the bulk of human daily activity, and yet much remains unknown about how people learn such actions. In one motor learning paradigm, the serial reaction time (SRT) task, people are taught a consistent sequence of button presses by cueing them with the next target response. However, the SRT task only records keypress response times to a cued target, and thus it cannot reveal the full time‐course of motion, including predictive movements. This paper describes a mouse movement (...) trajectory SRT task in which the cursor must be moved to a cued location. We replicated keypress SRT results, but also found that predictive movement—before the next cue appears—increased during the experiment. Moreover, trajectory analyses revealed that people developed a centering strategy under uncertainty. In a second experiment, we made prediction explicit, no longer cueing targets. Thus, participants had to explore the response alternatives and learn via reinforcement, receiving rewards and penalties for correct and incorrect actions, respectively. Participants were not told whether the sequence of stimuli was deterministic, nor if it would repeat, nor how long it was. Given the difficulty of the task, it is unsurprising that some learners performed poorly. However, many learners performed remarkably well, and some acquired the full 10‐item sequence within 10 repetitions. Comparing the high‐ and low‐performers’ detailed results in this reinforcement learning (RL) task with the first experiment's cued trajectory SRT task, we found similarities between the two tasks, suggesting that the effects in Experiment 1 are due to predictive, rather than reactive processes. Finally, we found that two standard model‐free reinforcement learning models fit the high‐performing participants, while the four low‐performing participants provide better fit with a simple negative recency bias model. (shrink)

A deep reinforcement learning-based computational guidance method is presented, which is used to identify and resolve the problem of collision avoidance for a variable number of fixed-wing UAVs in limited airspace. The cooperative guidance process is first analyzed for multiple aircraft by formulating flight scenarios using multiagent Markov game theory and solving it by machine learning algorithm. Furthermore, a self-learning framework is established by using the actor-critic model, which is proposed to train collision avoidance decision-making neural networks. To achieve (...) higher scalability, the neural network is customized to incorporate long short-term memory networks, and a coordination strategy is given. Additionally, a simulator suitable for multiagent high-density route scene is designed for validation, in which all UAVs run the proposed algorithm onboard. Simulated experiment results from several case studies show that the real-time guidance algorithm can reduce the collision probability of multiple UAVs in flight effectively even with a large number of aircraft. (shrink)

Interactions between an intelligent software agent and a human user are ubiquitous in everyday situations such as access to information, entertainment, and purchases. In such interactions, the ISA mediates the user’s access to the content, or controls some other aspect of the user experience, and is not designed to be neutral about outcomes of user choices. Like human users, ISAs are driven by goals, make autonomous decisions, and can learn from experience. Using ideas from bounded rationality, we frame these interactions (...) as instances of an ISA whose reward depends on actions performed by the user. Such agents benefit by steering the user’s behaviour towards outcomes that maximise the ISA’s utility, which may or may not be aligned with that of the user. Video games, news recommendation aggregation engines, and fitness trackers can all be instances of this general case. Our analysis facilitates distinguishing various subcases of interaction, as well as second-order effects that might include the possibility for adaptive interfaces to induce behavioural addiction, and/or change in user belief. We present these types of interaction within a conceptual framework, and review current examples of persuasive technologies and the issues that arise from their use. We argue that the nature of the feedback commonly used by learning agents to update their models and subsequent decisions could steer the behaviour of human users away from what benefits them, and in a direction that can undermine autonomy and cause further disparity between actions and goals as exemplified by addictive and compulsive behaviour. We discuss some of the ethical, social and legal implications of this technology and argue that it can sometimes exploit and reinforce weaknesses in human beings. (shrink)

Recent years have yielded many discussions on how to endow autonomous agents with the ability to make ethical decisions, and the need for explicit ethical reasoning and transparency is a persistent theme in this literature. We present a modular and transparent approach to equip autonomous agents with the ability to comply with ethical prescriptions, while still enacting pre-learned optimal behaviour. Our approach relies on a normative supervisor module, that integrates a theorem prover for defeasible deontic logic within the control loop (...) of a reinforcement learning agent. The supervisor operates as both an event recorder and an on-the-fly compliance checker w.r.t. an external norm base. We successfully evaluated our approach with several tests using variations of the game Pac-Man, subject to a variety of “ethical” constraints. (shrink)

The book reconstructs the history of Western ethics. The approach chosen focuses the endless dialectic of moral codes, or different kinds of ethos, moral doctrines that are preached in order to bring about a reform of existing ethos, and ethical theories that have taken shape in the context of controversies about the ethos and moral doctrines as means of justifying or reforming moral doctrines. Such dialectic is what is meant here by the phrase ‘moral traditions’, taken as a name for (...) threads of moral discourse, made in turn of other interwoven threads, including transmission of shared codes, appeals to reform of prevailing custom, rational argument about the justification of some precept on the basis of some shared general teaching or principle, and rational argument about the ultimate basis for principles and justification of authoritative teaching. That is, the approach adopted to the reading of ethical texts depends on a firm belief in the fact that philosophers hardly created any ethical doctrine out of nothing. The main point this book tries to highlight is how different philosophical theories emerged and followed each other as a result of attempts at accounting for what was going on in the world of moral traditions. Changes were propelled by controversies between different schools, and highly abstract arguments were the unintended effects of moves made by controversialists forced to transform (and occasionally to turn upside down) their own doctrines in order to face the challenge posed by other arguments. This is the reason why the book examines not only texts that already enjoy pride of place in the history of philosophy (Aristotle, Kant, Hegel), but also other texts usually treated in the histories of religions (the Bible, the Talmud, the Quran), and others considered to be much less philosophical (Plutarchus, Pufendorf). -/- 1. Plato and a response to ethical scepticism. Two different traditions of morality in VI-V century Greece are described. The birth of philosophical questioning of traditional morality and temporal and spatial variation of custom is described within the context of the v century crisis, the demise of traditional aristocratic and tyrannical rule and the birth of democracy. Two conflicting answers to the challenge are reconstructed, namely conventionalist or immoralist theories formulated by the Sophists and the eudemonist and intellectualist Socratic theory. Plato’s own reformulation of Socrates answer to the Sophists is reconstructed. His psychological views, his classification of the four cardinal virtues and his political theory are described as parts of a unitary system, culminating in an extremely realist moral ontology identifying the idea of the good with the essence of the (moral and extra-moral) world itself. -/- 2. Aristotle and the invention of practical philosophy. Aristotle’s invention of practical philosophy as a field separated from first philosophy is shown to be an implication of his break with Plato moral ultra-realism. Aristotle’s agenda in his moral works is arguably dependent on a polemical intention, namely dismantling Socratic intellectualism. The semi-inductive or virtuously circular method of practical philosophy is illustrated, starting with the received opinions of the better and wiser individuals and trying dialectically to sift what is left of mistake and inconsistence in such opinions, finally trying to correct mistakes and make the overall practical science more consistent. The chapter illustrates then the relationship of individual ethics, or ‘monastics’, with the art and the science of the pater familias, or ‘economics’, and the science of the ruler and citizen, or politics. The nature of virtues, or better, excellences of character, is discussed, highlighting the basic role of hexis, or ‘disposition’. Prudence, or better practical wisdom, is the focus of the chapter. Its relationship with bouleusis or deliberation is examined, and its autonomous status vis-à-vis theoretical knowledge is stressed. -/- 3. Diogenes and philosophy as a way of life. The chapter provides an overview of Hellenistic ethics, which almost amounts to Hellenistic schools of philosophy, in so far as ‘philosophy’ became in these centuries primarily the name for a way of life. The typical character of the Cynical movement is highlighted, that of a school of life, not a school aimed at providing any kind of intellectual training was to be provided. -/- 4. Epicurus and ethics as self-care. The peculiar character of the Epicurean school is described, a combination of a science of well-being aiming, more than at pleasure as in the popular view, at reduction of useless suffering, of unnecessary needs, and at a balanced selection of pleasures of the best and most durable kind. -/- 5. Epictetus and ethics as therapy of the passions. The various phases of Stoicism are described, and the shifting place given to ethics in the Stoic system of idea, culminating in the paradoxical view of ethics, its impossibility in principle notwithstanding, as the only truly significant and necessary part of philosophy. Cicero is treated, showing how his own synthesis of various Hellenistic trends is as a truly philosophical enterprise, deserving serious consideration after one or two centuries when he was confined to the role of literate. Epictetus is chosen as the best example of what the Stoic tradition could yield, an art of living based on sophisticated introspection, in turn aimed at making a kind of cognitive therapy possible, dismantling obnoxious passions at their root by systematically correcting false representations. -/- 6. Philo and the reconciliation of Torah and Platonism. A reconstruction of basic ideas from a few books of the Hebrew Bible is provided, starting with the Prophetic tradition and the focus on God’s mercy as the source of motivation and standard for human behaviour. Then a comparative analysis is undertaken of a parallel tradition, namely the three codifications of the Torā (Law or, better, Instruction), highlighting how a core of moral ideas may be recognized as a basis and preamble of codification of civil law, cultural practice, and regulation of ritual purity. The importance of Leviticus is stressed as the turning point when emphasis mercy, typical of the Prophetic tradition, is combined with the legal tradition, yielding the change in sensibility of the Second Temple time. Philo of Alexandria is described as one of the three leading figures – together with Rabbi Hillel and Rabbi Yeshua – at the apex the emergence of the mentioned new sensibility, gradually including mercy as an essential part of justice and establishing the starting point of both the Rabbinic and the Christian tradition. This consists precisely in the precept of one’s neighbour’s love or of the golden rule, two eventually identical precepts whose meaning is arguably more sober and sensible than the long-lasting Christian tradition deriving from John and Augustine has made us believe and no novelty vis-à-vis so-called Ancient-Testament teachings. -/- 7. Augustine and Christianity as Neo-Platonism. The first section examines the moral doctrines of so-called 'ethical Treaties' from the Talmud, a group of treaties, among which the best known is Pirqé Avot, that were left out of the six "orders" of the canon as they did not fit in any of the six groups of issues ritual or legal on which the division was based. According to Maimonides, their peculiar theme is provided by the Deòt, 'opinions', i.e., mental dispositions, that is, the translation of the Greek term hexeis and Arab akhlak (in turn providing in this language the name for ethics as such). The three topics I reconstruct are: i) the notion of Torah: The Torah is understood as the world order itself, or as the ‘Wisdom’ that existed even before creation and was ‘the tool by which the world was built’; however, the Torah is an earthly and human entity, as it was "received" by humans, and from that very moment belongs to them; ii) the relationship between love of God and love of neighbour; the treaties require us to study and practice the Torah ‘for its own sake’, that is, require us to act out of love, not out of fear or hope of reward; iii) the idea of sanctification of daily life: having disappeared with the destruction of the Temple the possibility of any conflict between liturgical service and everyday life, the latter is assumed to be in itself divine service: to give food to the poor has the same value as sacrifices in the Temple, and as an implication, the insistence become recurrent on the goodness of created things in themselves along with a polemic against ascetic currents. The conclusions drawn are: i) the moral teachings of the Talmud and those of Yeshua are, rather than similar, virtually identical; one may safely say that the precept of love and the golden rule are central ones for all Talmudic rabbis, that mercy plays an indispensable role alongside with justice, and the latter is not a different thing from one’s neighbour’s love; ii) a peculiarity of Talmud rabbis facing Yeshua is the idea of study as worship, and knowledge as a source of justice; but this is an idea of Judaism after the Temple's destruction that cannot be attributed to the Pharisees of Yeshua’s time; iii) the relation of study and practice in the Talmud parallels that between faith and deeds in Paul's epistles, that is, respectively faith or learning are a necessary and sufficient condition to be recognized as righteous , but deeds are the inevitable effect of either faith or learning. The sayings ascribed to Yeshua are examined first, yielding the conclusion that a close equivalent may be found for every saying in Talmudic literature and yet the whole is ascribed to one rabbi, with rather consistent stress on God’s mercy and unconditional forgiving as the mark of true imitation of God. Thus, Yeshua’s teaching is pure Judaism. The third section describes briefly the galaxy of Gnostic currents and Manichaeism, trying to sketch the profile of moral teachings resulting from an encounter of Asian spiritual traditions, Hellenistic lore and sparks of teachings from apocalyptic Jewish currents. The last section summarizes the turbulent history of several encounters between Christian currents and Hellenistic philosophical schools. The first one was with late Cynicism. Recent, rather controversial, literature discovered the jargon and a few topics from the Stoic-Cynical popular philosophy in a few books from the New Testament itself. This, far from proving that Rabbi Yeshua had been influenced by cynic preachers, is a proof of the necessity felt by Christian preachers to translate the original ‘Christian’ teaching into a Greek lexicon deeply impregnated with cynic terminology. The second was with Platonism, yielding the mild and temperate moral teaching of Clemens of Alexandria, teaching the sanctity of nature and the human body, the joy of moderate fruition of ‘natural’ kinds of pleasures, and the beauty of the marital life – in short, the opposite of the standard picture of Medieval Christianity. Ambrose of Milan brings about a different kind of synthesis, namely with Middle Platonism, where Stoic themes prevail. The most shocking case is Augustine, where an early Manichean education is overcome in a former phase by a synthesis of Plotinian Neo-Platonism and Christian preaching, yielding a sustained polemic with the Manicheans and rather optimistic views on life and Creation, the body and sexuality, and Hebrew-Judaic tradition not far from Clemens of Alexandria. In a later phase, occasioned by controversy on the opposite front, with such Christian currents as the Pelagians and Donatists, Augustine comes back to heavily anti-Judaic and world-denying ascetic attitudes where is earlier Manichean upbringing seems to emerge again. The tragedy of medieval Christianity will be the later Augustine’s overwhelming influence. A final section is dedicated to the monastic tradition where a curious mixture of world-denying asceticism with an astonishingly penetrating ‘science’ of introspection emerges. -/- 8. Al Farabi and the reconciliation of Islam and Platonism. The Qurān and the qadit, that is, collections of sayings ascribed to the prophet Muhammad contain a wealth of precepts and catalogues of virtues mirroring moral contents from the Jewish and the Christian traditions, among them the basic notion of imitation of the Deity, where mercy is assumed to be its basic trait. An important tradition within Islam, namely Sufism, stressed to the utmost degree the role of mercy, turning Islam into a mystic doctrine centred on retreat from the world, abandonment to God’s will, and a peaceful and fraternal attitude to our fellow-beings. A secular literary tradition originating in the Sassanid Empire, the literature of advice to the Prince had for a time a widespread influence in the newly constituted Arabic-Islamic commonwealth. In a later phase, the legacy of Hellenic philosophy made its way into the intellectual elite of the Islamic world. The first important legacy was Platonic, and the Republic became the main text for Islamic Platonism. Al-Farabi wrote an enjoyable remake of the Platonic Republic, arguing that the citizen’s virtues were the basis on which the political building needs to rest. The secular lawgiver is enlightened by the light of reason in his everyday practice of governance, but room is made for the Prophet as the voice of revealed truth that confirms and completes the rational truth that philosophy may afford. In a second phase also Aristotelian and Stoic influence were assimilated. Miskawayh’s treatise was the masterwork at the time Arabic philosophy reached its Zenith. It is a treatment of the soul’s diseases and their remedies, combining the Aristotelian doctrine of the golden mean with the Stoic doctrine of the passions and elements of Galenic medicine. Towards the eleventh-twelve centuries a war raged among theologians and philosophers, finally won by the former with disappearance of philosophy as such. The newly established mainstream, yet, was no kind of intolerant fanaticism. It drew from the work of mystic theologian Al-Ghazālī, the best heir of Sufism, teaching a tolerant and peaceful attitude to our fellow-beings and a passive attitude to destiny as an expression of the Divine will. -/- 9. Moshe ben Maimon and the reconciliation of Torah and Aristotelian ethics. The encounter between the Talmudic tradition and Hellenic philosophy had taken place a first time in Alexandria at the time of Philo but the two traditions had parted way again. In fact, the kind of Platonic Judaism founded by Philo survived only within Christianity, in the fourth Gospel and then in writings by Clemens of Alexandria. The Talmudic literature had absorbed just less striking traits from the Hellenic Philosophy, namely an idea of ethics as care of the self and a role for the education of character as propaedeutic to theoretical knowledge. In the Arabic-Islamic world, a second round started when Jewish authors writing in Arabic undertook the task to prove the full compatibility of the tradition deriving from Torah and Platonic philosophy. The culmination of this attempt is provided by Moshe ben Maimon who tried to use Aristotelian ethics as a language into which the teaching from the Pirké Avot could be translated. -/- 10. Thomas Aquinas and the reconciliation of Christian morality and Aristotelian ethics. In the first section the fresh start is described of philosophical ethics in Latin Europe at the turn of the Millennium. In a first phase, Anselm and Peter Abelard built on a Platonic and Augustinian legacy. In this phase. remarkable innovations are introduced, including Abelard’s claim of the obliging character of erring consciousness. In a second phase, the rediscovery of Nicomachean Ethics thanks to Latin translation of Arabic versions gives birth to a new wave of ethical studies, recovering the very idea of Aristotelian practical philosophy, with the potential implication of full legitimization of natural morality, i.e. ethics without Revelation. Aquinas’s ethics is a theological ethics out of which it would be vain to try to extract a self-contained philosophical ethics. His treatment of topics in philosophical ethics, yet, does not boil down to repetition of Aristotelian arguments but is rather a creative reshaping of such arguments. For example, he introduces also in the practical philosophy a first principle parallel to the principle of non-contradiction; and he also carries out a synthesis of Aristotelianism, Stoicism, and neo-Platonism. Even though it is essentially moral theology, Aquinas’s doctrine - unlike Augustine – grants full citizenship to "natural" morality, firstly by rejecting the claim that the corruption of human nature due to the original sin is so radical as to leave "pure nature" incapable of moral goodness. The doctrine is presented in a more sophisticated formulation in a few of the Quaestiones, such as De Malo and De Veritate, in the Summa contra Gentes and in the commentaries to Aristotle than in the famous Summa Theologica, but the latter work includes the only or the largest exposition of some decisive part of the theory. Thus, the Summa Theologica should be read for what it is more than criticized for not being what it was not meant to be. It was not meant to be the brilliant synthesis of all that Reason he had been able to produce with what Revelation had added about which the Neo-Thomists used to dream, but rather a manual for the training of preachers and confessors, where theoretical claims are not too demanding and a few serious tensions are left. Besides a jump between the Prima Secundae and the Secunda Secundae, being the former an essay in virtue theory and the latter a handbook for confessors, the most serious tension is perhaps the one between the ethics of right reason presented in most of Ia-IIae and the eudemonistic ethics developed in quaestiones 1-5 of the same part; the alternative ethical theory which also may be found in Augustine, the Stoic view of a cosmic reason eventually coincident with the moral law, was believed by Anselm (followed by John Duns Scotus and William of Ockham) to be incompatible with eudemonism. It is questionable whether Thomas could reduce the tension proving that it is just an apparent tension, in so far as the right reason and bliss derived from knowledge of God tend to coincide, but this is just a conjecture. Thomas’s ethics is a virtue ethic, not a law-based one, and moral judgment focuses on the virtues, particularly charity, not on the commandments and even less on absolute prohibitions; Thomas, however, would not have considered a drastic alternative between law and the virtues such as the one which has been advanced in late twentieth-century philosophy to be justified. Nonetheless, when it discusses ‘special’ virtues, it ceases to be an ethics of virtue and becomes a disappointing and often contradictory discussion of legal and illegal acts. Such a discussion takes most of the time ‘reasonable’ middle positions on controversial issues but not the alternative approach that Aristotelianism would have made possible; even when some occasional Aristotelian claim shows up, such as money’s barrenness as a reason against usury, this seems to be made by an author who apparently ignores the Aristotelian Thomas of the Prima Secundae. It is an ethic of human autonomy which recognizes the binding character of the individual conscience and, potentially, even a duty to disobey unjust laws. It is true that what Thomas writes in his discussion of death penalty and persecution of heretics is simply disgusting, and yet we should blame Thomas the man, not Thomist ethical theory. Finally, Thomas’s ethics is not ethical ‘absolutism’, as both traditionalist Catholic theologians and secularist enemies of dogmatic morality tend to believe. It is instead an ethic of prudential judgment. Exceptions to this approach – or better results of logical fallacies – are such claims as the absolute character of negative precepts and of the existence of "intrinsically evil acts", claims that simply cease to make sense in the light of Thomas’s own distinction between human act and natural act, carrying consequences Thomas did not live long enough to draw or was not consistent enough to dare to draw. -/- 11. Francisco de Vitoria and casuistry. The first topic illustrated is the discussion about the notion of pura natura, namely the human condition after the original sin and before divine revelation. The core notion was already there in Aquinas but was later developed by Caietanus (Thomas de Vio) with a number of interesting implications: firstly a view of human nature as such much more positive than Augustine’s and his followers’ view, including both Calvinists and Jansenists; secondly, the implication that the philosopher’s morality, as opposed to the Christian’s morality, is a quite respectable kind of morality; thirdly, that theocracy and prosecution of the unfaithful lacked justification, with more far-fetching consequences than Aquinas himself had dared to draw. The second topic is casuistry, a new name for a comparatively older way of thinking, which was already there in late Stoicism and Cicero as well as in the Talmudic literature. This is an approach aimed at yielding probable enough solutions for doubtful cases even in absence of completely safe staring points. The genre developed from medieval reference books for confessors and became by the sixteenth-century a sophisticated tool-box for dealing with various fields of applied ethics. Francisco Vitoria, the main authority of Baroque Scholasticism, was a controversial figure, among the proponents of the new discipline of casuistry and a consistent proponent of more separation between the religious and the civil authority, stricter limits to the legitimacy of war, innate rights of non-Christian nations in the New World based on the notion of pura natura, providing a standard of ‘natural’ goodness, previous to revelation, on which the Indian nations were judged to live in a naturally good condition, provided with the institutions of family, justice, and religion had accordingly a right to full respect by Christian sovereigns. Bartolomé de Las Casas, arguing along similar lines, was the leader of a historical battle in defence of the rights of the Indios. -/- 12. Michel de Montaigne and the art of living. A short description of one of the Renaissance Phyrronism, one of the classical philosophies revived in the fifteenth and sixteenth century. Montaigne combines the sceptical epistemology with an understanding of ethics – indeed of philosophy as a whole – in terms of an art of living inspired by two basic ideas, sagesse, that is, practical wisdom as the only kind of rationality available after theology, science, and tradition have declared bankrupt, and an aesthetic ideal of self-transformation through the art of writing and introspection. -/- 13. Pierre Nicole and neo-Augustinianism. The chapter illustrates first the vicissitudes of Augustinianism newly born after the late medieval triumph of Thomist Aristotelianism in the alternative Protestant and Catholic versions processed by the Calvinists and the Jansenists. Then it illustrates the doctrines of Pierre Nicole, Blaise Pascal’s best disciple, on the impossibility of introspection, on the ubiquity of self-deception, on the incurably evil character of the passion of love, and on the two moralities, the one of the man of the world, the morality of honesty which is indispensable for granting peace and order but devoid of any true value for eternal salvation, since the same external behaviour may be prompted by opposite motivations, and the morality of charity, the only true one but also useless to society. The third topic is Pierre Malebranche’s view of self-deception as a ubiquitous phenomenon accounting for human action and responsibility and his reformulated theory of self-love, no less ubiquitous than for Nicole and yet not as incurably evil, given the distinction between morally indifferent and even necessary amour de soi and vicious amour proper, a distinction on which the whole of Rousseau’s moral and political theory rests. -/- 14. Samuel von Pufendorf and the new science of morality. The chapter is dedicated to the discovery of the idea of a ‘new moral science’. Hugo Grotius is discussed first highlighting the real character of his project, much closer to the Thomist idea of a natural law accessible to non-corrupted human reason when human beings are living in a state of pura natura. Then Hobbes’s combination of scepticism, voluntarism, and conventionalism is described and both the continuity with Grotius’s new science, in the search for non-revealed rational morality, and the break with him, in the adoption of voluntarism and refusal of an intellectualist view of natural law, are illustrated. Pufendorf’s work is illustrated as a synthesis of the two previous attempts and the – up to Schneewind underestimated – paradigmatic example of the new science of morality. -/- 15. Richard Cumberland and consequentialist voluntarism. The chapter gives an overview of eighteenth-century Anglican ethics, noticing how the Cambridge tradition gave special weight to natural theology as opposed to positive or revealed theology – and how two Cambridge fellows, John Gay and Thomas Brown, elaborated on Cumberland’s (and Malebranche’s, as well as Leibniz’s) strategy for finding a third way between intellectualist view and voluntarist view of the laws of nature. The result of their elaboration was a kind of a rational-choice account of the origins of natural laws, where a law-giver God chooses among a number possible sets of laws on a maximizing criterion, and God’s maximandum is happiness for his creatures. The chapter notices also how such a solution aimed at solving at once the problem of evil and that of the foundation of moral obligation by proving how God’s choice was justified as far as it was the one minimising the amount of suffering in the world. 16. Richard Price and intuitionism. The chapter describes first the doctrines of the Cambridge Platonists, an example of hyper-rationalist reaction to Calvinism. Secondly it describes the sophisticated and universally ignored – from Sidgwick to Anscombe –version of what was later labelled ‘ethical intuitionism’ – showing how it escapes familiar objections and misrepresentations of intuitionism – from Mill to Rawls – in grounding its argument on transcendental arguments while carefully avoiding recourse to introspection and psychological evidence, which has been taken as a too easy target by critics of intuitionism. Thirdly, the chapter discusses Whewell’s ‘philosophy of morality’, as opposed to ‘systematic morality’, not unlike Kant’s distinction between a pure and an empirical moral philosophy. Whewell worked out a systematization of traditional normative ethics as a first step before its rational justification; he believed that the point in the philosophy of morality is justifying a few rational truths about the structure of morality such as to rule hedonism, eudemonism, and consequentialism; yet a system of positive morality cannot be derived solely from such rational truths but requires consideration of the ongoing dialectics between idea and fact in historically given moralities. Whewell’s intuitionism turns out closer to Kantian ethics than commentators have made us believe until now, and quite different from what Sidgwick meant by intuitionism. -/- 17. Adam Smith and the morality of role-switch. The chapter describes first, Hutcheson’s attempt at basing the ‘new science’ of natural law on different bedrock than Pufendorf and the English Platonists, namely a moral sense, a faculty whose existence is assumed to be proved on an empirical basis. The second step is a reconstruction of Hume’s rejoinder in terms of a new science of man including morality on an ‘experimental’ basis, that is, a ‘Newtonian’ hypothetical-deductive approach, distinguished from Hutcheson’s allegedly uncritical descriptive approach to human nature. The third step is a reconstruction of Adam Smith theory of morality understood as emerging from a spontaneous interplay of exchanges of situations (the most basic meaning of the word ‘sympathy’ as construed by this author). -/- 18. Jeremy Bentham and utilitarianism. The bulk of the chapter is dedicated to the doctrines of Jeremy Bentham. The Enlightenment spirit that suggested the idea of a new morality, free from religion, is illustrated. The notion of utility is illustrated as well as the subsequent formulations of the principle of utility. The idea of felicific calculus is discussed, showing how its inner difficulties prompted several reformulations of the principle of utility in order to avoid undesirable implications of proposed formulations. The role of the thesis of spontaneous convergence between interest and duty is discussed, showing how it left numberless questions open, and the distinction between the virtues of prudence, justice and beneficence is described as a way out of the deadlocks of classical utilitarianism. After Bentham, Mill’s reformed utilitarianism is reconstructed, showing how it is a kind of mixed system – as closer to common sense as it gives up Bentham’s claims to consistency and simplicity – resulting as unintended consequences from the controversies in which Mill was too keen to engage. The hidden agenda of the controversy between Utilitarianism and Intuitionism, going beyond the image of the battle between Prejudice and Reason, is described, showing how both competing philosophical outlooks turn out to be more research programs than self-contained doctrinal bodies, and such programs appear to be implemented, and indeed radically transformed while in progress thanks to their enemies no less than to their supporters. -/- 19. Immanuel Kant, practical reason and judgment. The chapter argues that Kant took from Moses Mendelssohn the idea of a distinction between geometry of morals and a practical ethic. He was drastically misunderstood by his followers precisely on this point. He had learned from the sceptics and the Jansenists the lesson that men are prompted to act by deceptive ends, and he was aware that human actions are also empirical phenomena, where laws like the laws of Nature may be detected. His practical ethics made room for judgment as a holistic procedure for assessing the saliency of relevant moral qualities in one given situation; this procedure yields the same results as the geometry of morals does for abstract cases but does so immediately and without balancing conflicting duties with each other, since what makes for the salient quality of a situation is perceived from the very beginning. Kant's practical philosophy is richer than the received image, making room for an ‘empirical moral philosophy’ or moral anthropology including treatment of commerce, needs, value and price, happiness and well-being; the overall social theory and philosophy of history is less different from Adam Smith's than the received image makes believe; the paradox of happiness is central to Kant's philosophy; a distinction between happiness and well-being is clearly drawn; the distinction plays a basic role in establishing a link between the economic and the moral life. -/- 20. Georg Wilhelm Friedrich Hegel and the critique of abstract universalism. The chapter describes first the Romantic movement and the implications some of its concerns, rescuing passions, community, tradition, the individual as the bearer of a unique destiny, and the longing for organic unity between the individual, mankind and nature. Hegel’s contribution is discussed then, highlighting how, on the one hand, he shared a number of these concerns and on the other he had more rationalist leanings. The notion of morality is the pivotal point of the reconstruction, highlighting how Hegel construes this notion as a key to his own diagnosis of the malaise of modernity – the separation of individual and Gemeinschaft – and how his attack on Kant turns around this very idea. -/- 21. Friedrich Nietzsche against Christianity and the Enlightenment. The chapter illustrates first the idea of deconstruction of the back-world of values. Nietzsche claims to be legitimate heir of the French moralists, Leibniz, Kant and Hegel, in so far as he allegedly carries out to its deepest implications their discovery of what lies behind traditional naïve belief in the existence of an objective realm of values just waiting for description by the philosopher. The two exemplars form which genealogy draws inspiration are the classical philologist’s historical reconstructions of lost meanings and the chemist’s decomposition of elements. Then the genealogy of the notions of good and evil carried out in the first dissertation of Genealogy of morals is illustrated with its paradoxical conclusion that will to power is in fact the only ‘genuine’ kind of goodness. The third point illustrated is the dialectic of ascetism and self-realization with its ambiguous outcome. The suggested interpretation of such outcome is that Nietzsche’s normative ethics is a kind of virtue ethics taking an aesthetic ideal as a normative standard -/- 22. George Edward Moore and ideal utilitarianism. The chapter discusses the ideas of common sense and common-sense morality in Sidgwick. I argue that, far from aiming at overcoming common-sense morality, Sidgwick aimed purposely at grounding a consist code of morality by methods allegedly taken from the natural sciences to reach, also in ethics, the same kind of “mature” knowledge as in the natural sciences. His whole polemics with intuitionism was vitiated by the a priori assumption that the widespread ethos of the educated part of humankind, not the theories of the intuitionist philosophers, was what was worth considering as the expression of intuitionist ethics. In spite of a naïve positivist starting-point, Sidgwick was encouraged by his own approach in exploring the fruitfulness of coherentist methods for normative ethics. Thus, Sidgwick left an ambivalent legacy to twentieth-century ethics: the dogmatic idea of a “new” morality of a consequentialist kind, and the fruitful idea that we can argue rationally in normative ethics albeit without shared foundations. Then it reconstructs the background of ideas, concerns and intentions out of which Moore’s early essays, the preliminary version, and then the final version of Principia Ethica originated. I stress the role of religious concerns, as well as that of the Idealist legacy. I argue that PE is more a patchwork of rather diverging contributions than a unitary work, not to say the paradigm of a new school in Ethics. I add a comparison with Rashdall’s almost contemporary ethical work, suggesting that the latter defends the same general claims in a different way, one that manages to pave decisive objections in a more plausible way. I end by suggesting that the emergence of Analytic Ethics was a more ambiguous phenomenon than the received view would make us believe, and that the wheat (or some other gluten-free grain) of this tradition, that is, what logic can do for philosophy, should be separated from the chaff, that is, the confused and mutually incompatible legacies of Utilitarianism and Idealism. -/- 23. Edmund Husserl and the a-priori of action. The chapter illustrates first the idea of phenomenology and the Husserl’s project of a phenomenological ethic as illustrated in his 1908-1914 lectures. The key idea is dismissing psychology and trying to ground a new science of the a priori of action, within which a more restricted field of inquiry will be the science of right actions. Then the chapter illustrates the criticism of modern moral philosophy carried out in the 1920 lectures, where the main target is naturalism, understood in the Kantian meaning of primacy of common sense. The third point illustrate is Adolph Reinach’s theory of social acts as a key the grounding of norms, a view that basically sketches the very ideas ‘discovered’ later by Clarence I. Lewis, John Searle, Karl-Otto Apel and Jürgen Habermas. A final section is dedicated to Nicolai Hartman, who always refused to define himself a phenomenologist and yet developed a more articulated and detailed theory of ‘values’ – with surprising affinities with George E. Moore - than philosophers such as Max Scheler who claimed to Husserl’s legitimate heirs. -/- 24. Bertrand Russell and non-cognitivism. The chapter reconstructs first the discussion after Moore. The naturalistic-fallacy argument was widely accepted but twisted to prove instead that the intuitive character of the definition of ‘good’, its non-cognitive meaning, in a first phase identified with ‘emotive’ meaning. Alfred Julius Ayer is indicated as a typical proponent of such non-cognitivist meta-ethics. More detailed discussion is dedicated to Bertrand Russell’s ethics, a more nuanced and sophisticated doctrine, arguing that non-cognitivism does not condemn morality to arbitrariness and that the project of a rational normative ethics is still possible, heading finally to the justification of a kind of non-hedonist utilitarianism. Stevenson’s theory, another moderate version of emotivism is discussed at some length, showing how the author comes close to the discovery of the role of a pragmatic dimension of language as a basis for ethical argument. Last of all, the discussion from the Forties about Hume’s law is described, mentioning Karl Popper’s argument and Richard Hare’s early non-cognitivist but non-emotivist doctrine named prescriptivism. -/- 25. Elizabeth Anscombe and the revival of virtue ethics. The chapter discusses the three theses defended by Anscombe in 'Modern Moral Philosophy'. I argue that: a) her answer to the question "why should I be moral?" requires a solution of the problem of theodicy, and ignores any attempts to save the moral point of view without recourse to divine retribution; b) her notion of divine law is an odd one more neo-Augustinian than Biblical or Scholastic; c) her image of Kantian ethics and intuitionism is the impoverished image manufactured by consequentialist opponents for polemical purposes and that she seems strangely accept it; d) the difficulty of identifying the "relevant descriptions" of acts is not an argument in favour of an ethics of virtue and against consequentialism or Kantian ethics, and indeed the role of judgment in the latter is a response to the difficulties raised by the case of judgment concerning future action. A short look at further developments in the neo-naturalist current is given trough a reconstruction of Philippa Foot’s and Peter Geach’s critiques to the naturalist-fallacy argument and Alasdair MacIntyre’s grand reconstruction of the origins and allegedly unavoidable failure of the Enlightenment project of an autonomous ethic. -/- 26. Richard Hare and neo-Utilitarianism. The chapter addresses the issue of the complex process of self-transformation Utilitarianism underwent after Sidgwick’s and Moore’s fatal criticism and the unexpected Phoenix-like process of rebirth of a doctrine definitely refuted. A glimpse at this uproarious process is given through two examples. The first is Roy Harrod Wittgensteinian transformation of Utilitarianism in pure normative ethics depurated from hedonism as well as from whatsoever theory of the good. This results in preference utilitarianism combined with a ‘Kantian’ version of rule utilitarianism. The second is Richard Hare’s two-level preference utilitarianism where act utilitarianism plays the function of eventual rational justification of moral judgments and rule-utilitarianism that of action-guiding practical device. -/- 27. Hans-Georg Gadamer and rehabilitation of practical philosophy. The post-war rediscovery of ethics by many German thinkers and its culmination in the Sixties in the movement named ‘Rehabilitation of practical philosophy’ is described. Among the actors of such rehabilitation there were a few of Heidegger’s most brilliant disciples, and Hans-Georg Gadamer is chosen as a paradigmatic example. His reading of Aristotle’s lesson I reconstructed, starting with Heidegger’s lesson but then subtly subverting its outcome thanks to the recovery of the central role of the notion of phronesis. -/- 28. Karl-Otto Apel and the revival of Kantian ethics. Parallel to the neo-Aristotelian trend, there was in the Rehabilitation of practical philosophy a Kantian current. This started, instead than the rehabilitation of prudence, with the discovery of the pragmatic dimension of language carried out by Charles Peirce and the Oxford linguistic philosophy. On this basis, Karl-Otto Apel singled out as the decisive proponent of the linguistic and Kantian turn in German-speaking ethics, worked out the performative-contradiction argument while claiming that this was able to provide a new rational and universal basis for normative ethics. An examination of his argument is some detail is offered, followed by a more cursory reconstruction of Jürgen Habermas’s elaboration on Apel’s theory. -/- 29. John Rawls and public ethics as applied ethics. Rawls’s distinction between a “political” and a “metaphysical” approach to one central part of ethical theory, namely the theory of justice, is interpreted as a formulation of the same basic idea at the root of both the principles approach and neo-casuistry, both discussed in the following chapter, namely that it is possible to reach an agreement concerning positive moral judgments even though the discussion is still open – and in Rawls’ view never will be close – on the basic criteria for judgment. -/- 30. Beauchamp and Childress and bioethics as applied ethics. The chapter presents the revolution of applied ethics while stressing its methodological novelty, exemplified primarily by Beauchamp and Childress principles approach and then by Jonsen and Toulmin’s new casuistry. -/- . (shrink)

Recent research indicates that angry facial expressions are preferentially processed and may facilitate automatic avoidance response, especially in socially anxious individuals. However, few studies have examined whether this bias also expresses itself in more complex cognitive processes and behavior such as decision making. We recently introduced a variation of the Iowa Gambling Task which allowed us to document the influence of task-irrelevant emotional cues on rational decision making. The present study used a modified gambling task to investigate the impact (...) of angry facial expressions on decision making in 38 individuals with a wide range of social anxiety. Participants were to find out which choices were (dis-) advantageous to maximize overall gain. To create a decision conflict between approach of reward and avoidance of fear-relevant angry faces, advantageous choices were associated with angry facial expressions, whereas disadvantageous choices were associated with happy facial expressions. Results indicated that higher social avoidance predicted less advantageous decisions in the beginning of the task, i.e., when contingencies were still uncertain. Interactions with specific skin conductance responses further clarified that this initial avoidance only occurred in combination with elevated responses before choosing an angry facial expressions. In addition, an interaction between high trait anxiety and elevated responses to early losses predicted faster learning of an advantageous strategy. These effects were independent of intelligence, general risky decision-making, self-reported state anxiety, and depression. Thus, socially avoidant individuals who respond emotionally to angry facial expressions are more likely to show avoidance of these faces under uncertainty. This novel laboratory paradigm may be an appropriate analog for central features of social anxiety. (shrink)

In reinforcement learning, a decision maker searching for the most rewarding option is often faced with the question: What is the value of an option that has never been tried before? One way to frame this question is as an inductive problem: How can I generalize my previous experience with one set of options to a novel option? We show how hierarchical Bayesian inference can be used to solve this problem, and we describe an equivalence between the Bayesian model (...) and temporal difference learning algorithms that have been proposed as models of RL in humans and animals. According to our view, the search for the best option is guided by abstract knowledge about the relationships between different options in an environment, resulting in greater search efficiency compared to traditional RL algorithms previously applied to human cognition. In two behavioral experiments, we test several predictions of our model, providing evidence that humans learn and exploit structured inductive knowledge to make predictions about novel options. In light of this model, we suggest a new interpretation of dopaminergic responses to novelty. (shrink)

The paper presents a set of games of competition between two or three players in which reward is jointly determined by a stochastic biased mechanism and players’ choices. More specifically, a resource can be found with unequal probabilities in one of two locations. The first agent is rewarded only if it finds the resource and avoids being found by the next agent in line; the latter is rewarded only if it finds the former. Five benchmarks, based on different psychological (...) and game-theoretic assumptions are derived and their predictions compared to actual behavior of 120, 40, and 48 participants playing repeatedly. Of the five benchmarks—the unique (Nash) equilibrium, reinforcement learning, trust-based efficiency, maximum unpredictability, and regret-based (Impulse Balance) equilibrium—regret for missed opportunities best accounts for the qualitative aspect of participants’ behavior and regret attenuated by randomization best accounts for the quantitative aspect of behavior. (shrink)

The fundamental cognitive problem for active organisms is to decide what to do next in a changing environment. In this article, we analyze motor and action control in computational models that utilize reinforcement learning (RL) algorithms. In reinforcement learning, action control is governed by an action selection policy that maximizes the expected future reward in light of a predictive world model. In this paper we argue that RL provides a way to explicate the so-called action-oriented views (...) of cognitive systems in representational terms. (shrink)

The authors aim at constructing an agent which learns appropriate actions in a Multi-Agent environment with and without social dilemmas. For this aim, the agent must have nonrationality that makes it give up its own profit when it should do that. Since there are many studies on rational learning that brings more and more profit, it is desirable to utilize them for constructing the agent. Therefore, we use a reward-handling manner that makes internal evaluation from the agent's rewards, (...) and then the agent learns actions by a rational learning method with the internal evaluation. If the agent has only a fixed manner, however, it does not act well in the environment with and without dilemmas. Thus, the authors equip the agent with several reward-handling manners and criteria for selecting an effective one for the environmental situation. In the case of humans, what generates the internal evaluation is usually called emotion. Hence, this study also aims at throwing light on emotional activities of humans from a constructive view. In this paper, we divide a Multi-Agent environment into three situations and construct an agent having the reward-handling manners and the criteria. We observe that the agent acts well in all the three Multi-Agent situations composed of homogeneous agents. (shrink)

This paper considers the concept of a good game in terms of its relation to the fair testing of relevant skills and their aesthetic values. As such, it will consider what makes football ‘the beautiful game’ and what part penalty shoot-outs play, or should play, within it. It begins by outlining and refuting Kretchmar’s proposal that games which end following the elapsing of a set amount of time, such as football, are structurally, morally and aesthetically inferior to games which (...) end following the attainment of a designated goal. Instead, it is suggested that the structural deficits of t-games are compensated by the fact that they allow for a fuller test of more valuable open skills than the closed skills that tend to define e-games, such as golf. Moreover, it is the rationing of time in t-games that provides an added aesthetic value that does not occur in e-games. Finally, it is concluded that although it may appear that the existence of the penalty shoot-out supports Kretchmar’s argu.. (shrink)

After generalizing the Archimedean property of real numbers in such a way as to make it adaptable to non-numeric structures, we demonstrate that the real numbers cannot be used to accurately measure non-Archimedean structures. We argue that, since an agent with Artificial General Intelligence (AGI) should have no problem engaging in tasks that inherently involve non-Archimedean rewards, and since traditional reinforcement learning rewards are real numbers, therefore traditional reinforcement learning probably will not lead to AGI. We indicate two (...) possible ways traditional reinforcement learning could be altered to remove this roadblock. (shrink)

The connection between economics and ethics is as old as economics itself, and central to both disciplines. It is an issue that has recently attracted much interest from economists and philosophers. The connection is, in part, a result of the desire of economists to make policy prescriptions, which clearly require some normative criteria. More deeply, much economic theory is founded on the assumption of utility maximization, thereby creating an immediate connection between the foundations of economics and the philosophical literature (...) on utilitarianism and reasons for action. In fact, some influential contemporary approaches to ethics advocate decision-theoretic or game-theoretic foundations of some sort for moral principles, while several economic theorists are now prepared to take into account the ethical dimensions of rational decisions. As a result, it appears that economics and ethics are somehow inextricably linked through theories of rational decision-making. Most of us would probably find it disturbing to concede that there are contradictions between the prescriptions of rationality and the requirements of moral `rightness'. The essays included in the present volume provide a detailed analysis of the connections between ethics and economics as viewed from several different - sometimes conflicting - perspectives. This book, the outcome of a joint meeting of philosophers and economists, has three main themes: the validity of utilitarianism much used by economists, the notion of fairness and equity, and the coherence of the rationality postulate of economics. The book does not reach any final conclusions, but it greatly illuminates the exact areas of possible disagreement and indeed the open-ended nature of ethical reasoning. There is much that economists, and especially welfare economists, can learn from these papers - not least circumspection. (shrink)

In Moderating the Debate: Rationality and the Promise of American Education, Michael Feuer raises concerns about the consequences of basing educational policy on the model of rational choice drawn from economics. Policy making would be better and more realistic, he suggests, if it were based on a newer procedural model drawn from cognitive science. In this essay Eric Bredo builds on Feuer's analysis by offering a more systematic critique of the traditional model of rationality that Feuer criticizes, (...) a more critical evaluation of the procedural model that he favors, and a recommendation that the situational model he does not consider may have some benefits over both. This analysis shows that the traditional model presupposes an actor that cannot learn or develop. While the actor in the procedural model can learn, Bredo contends that it cannot develop, that is, it cannot outgrow its initial assumptions and values. Only the situational model allows for learning and development, important in a model to be used in the field of education. Bredo also considers in his analysis the social‐relational assumptions built into the traditional, procedural, and situational models and the likely ethical consequences of acting on them. (shrink)

The advent of functional brain imaging has revolutionized the ability to understand the biological mechanisms underlying decision-making. Although it has been amply demonstrated that assumptions of rationality often break down in experimental games, there has not been an overarching theory of why this happens. I describe recent advances in functional brain imaging and suggest a framework for considering the function of the human reward system as a discrete agent.

This paper makes a modest point. Suppose one wants to evaluate alternative policies, institutions or even constitutions on the basis of their consequences. To do so, one needs to evaluate their consequences and one needs to know what their consequences are. Let us suppose that the role of economic theories and game theory in particular is mainly to help us to use information we already possess or that we can acquire at a reasonable cost to judge what the consequences will (...) be. We do not necessarily need true theories or theories that provide perfectly precise predictions. What sort of accuracy and precision we need depends on how the consequences of alternative policies are evaluated. The worth of an economic theory in this context of policy assessment depends on the accuracy of the predictions the theory permits, the costs of gathering the needed information, and the costs of learning and using the theory. My only excuse for uttering these truisms is that well known economists and philosophers have denied them. (shrink)

The area of smart power grids needs to constantly improve its efficiency and resilience, to provide high quality electrical power in a resilient grid, while managing faults and avoiding failures. Achieving this requires high component reliability, adequate maintenance, and a studied failure occurrence. Correct system operation involves those activities and novel methodologies to detect, classify, and isolate faults and failures and model and simulate processes with predictive algorithms and analytics. In this paper, we showcase the application of a complex-adaptive, (...) self-organizing modeling method, and Probabilistic Boolean Networks, as a way towards the understanding of the dynamics of smart grid devices, and to model and characterize their behavior. This work demonstrates that PBNs are equivalent to the standard Reinforcement Learning Cycle, in which the agent/model has an interaction with its environment and receives feedback from it in the form of a reward signal. Different reward structures were created to characterize preferred behavior. This information can be used to guide the PBN to avoid fault conditions and failures. (shrink)

In this opinionated review, I draw attention to some of the contributions reinforcement learning can make to questions in the philosophy of mind. In particular, I highlight reinforcement learning's foundational emphasis on the role of reward in agent learning, and canvass two ways in which the framework may advance our understanding of perception and motivation.

The renowned philosopher John Searle reveals the fundamental nature of social reality. What kinds of things are money, property, governments, nations, marriages, cocktail parties, and football games? Searle explains the key role played by language in the creation, constitution, and maintenance of social reality. We make statements about social facts that are completely objective, for example: Barack Obama is President of the United States, the piece of paper in my hand is a twenty-dollar bill, I got married in London, etc. (...) And yet these facts only exist because we think they exist. How is it possible that we can have factual objective knowledge of a reality that is created by subjective opinions? This is part of a much larger question: How can we give an account of ourselves, with our peculiar human traits DS mind, reason, freedom, society - in a world that we know independently consists of mindless, meaningless particles? How can we account for our social and mental existence in a realm of brute physical facts? In answering this question, Searle avoids postulating different realms of being, a mental and a physical, or worse yet, a mental, a physical, and a social. There is just one reality: Searle shows how the human reality fits into that one reality. Mind, language, and civilization are natural products of the basic facts of the physical world described by physics, chemistry and biology. Searle explains how language creates and maintains the elaborate structures of human social institutions. These institutions serve to create and distribute power relations that are pervasive and often invisible. These power relations motivate human actions in a way that provides the glue that holds human civilization together. Searle shows how this account illuminates human rationality, free will, political power, and human rights. Our social world is a world created and maintained by language. (shrink)

Rapid and precise air operation mission planning is a key technology in unmanned aerial vehicles autonomous combat in battles. In this paper, an end-to-end UAV intelligent mission planning method based on deep reinforcement learning is proposed to solve the shortcomings of the traditional intelligent optimization algorithm, such as relying on simple, static, low-dimensional scenarios, and poor scalability. Specifically, the suppression of enemy air defense mission planning is described as a sequential decision-making problem and formalized as a Markov decision process. (...) Then, the SEAD intelligent planning model based on the proximal policy optimization algorithm is established and a general intelligent planning architecture is proposed. Furthermore, three policy training tricks, i.e., domain randomization, maximizing policy entropy, and underlying network parameter sharing, are introduced to improve the learning performance and generalizability of PPO. Experiments results show that the model in this work is efficient and stable, and can be adapted to the unknown continuous high-dimensional environment. It can be concluded that the UAV intelligent mission planning model based on DRL has powerful intelligent planning performance, and provides a new idea for researching UAV autonomy. (shrink)

Successfully explaining and replicating the complexity and generality of human and animal learning will require the integration of a variety of learning mechanisms. Here, we introduce a computational model which integrates associative learning (AL) and reinforcement learning (RL). We contrast the integrated model with standalone AL and RL models in three simulation studies. First, a synthetic grid‐navigation task is employed to highlight performance advantages for the integrated model in an environment where the reward structure is both diverse and (...) dynamic. The second and third simulations contrast the performances of the three models in behavioral experiments, demonstrating advantages for the integrated model in accounting for behavioral data. (shrink)

Toxic speech inflicts individual and group harm, damaging the social fabric upon which we all depend. To understand and combat the harms of toxic speech, philosophers can learn from epidemiology, while epidemiologists can benefit from lessons of philosophy of language. In medicine and public health, research into remedies for toxins pushes in two directions: individual protections (personal actions, avoidances, preventive or reparative tonics) and collective action (specific policies or widespread “inoculations” through which we seek herd immunity). This paper is the (...) beginning of a project of identifying potential inoculations and antidotes to toxic speech. The essay brings a social practice theory of language, with special reliance on language‐games and inferential roles, into conversation with concepts from the study of biologic toxins. Some speech harms are acute while others are chronic and insidious; they have different methods of delivery, come in variable doses, and not everyone is equally susceptible to the power to harm. I argue that of the many kinds of challenges we might issue against toxic speech, challenging its expressive commitments has the greatest potential to stop the damage. The essay explores the different sorts of protections that inoculations and antidotes might offer against discursive toxins and sketches how to imagine these in the practices that govern our speech. The paper does not make policy recommendations, but an epidemiology of discursive toxicity reveals several kinds of “more speech” that might fight against “bad speech.”. (shrink)

This study investigates the ability of individuals with psychopathy to perform passive avoidance learning and whether this ability is modulated by level of reinforcement/punishment. Nineteen psychopathic and 21 comparison individuals, as defined by the Hare Psychopathy Checklist Revised (Hare, 1991), were given a passive avoidance task with a graded reinforcement schedule. Response to each rewarding number gained a point reward specific to that number (i.e., 1, 700, 1400 or 2000 points). Response to each punishing number lost a (...) point punishment specific to that number (i.e., the loss of 1, 700, 1400 or 2000 points). In line with predictions, individuals with psychopathy made more passive avoidance errors than the comparison individuals. In addition, while the performance of both groups was modulated by level of reward, only the performance of the comparison population was modulated by level of punishment. The results are interpreted with reference to a computational account of the emotional learning impairment in individuals with psychopathy. (shrink)

Theories of decision-making and its neural substrates have long assumed the existence of two distinct and competing valuation systems, variously described as goal-directed vs. habitual, or, more recently and based on statistical arguments, as model-free vs. model-based reinforcement-learning. Though both have been shown to control choices, the cognitive abilities associated with these systems are under ongoing investigation. Here we examine the link to cognitive abilities, and find that individual differences in processing speed covary with a shift from model-free to (...) model-based choice control in the presence of above-average working memory function. This suggests shared cognitive and neural processes; provides a bridge between literatures on intelligence and valuation; and may guide the development of process models of different valuation components. Furthermore, it provides a rationale for individual differences in the tendency to deploy valuation systems, which may be important for understanding the manifold neuropsychiatric diseases associated with malfunctions of valuation. (shrink)

Implementing The new NHS and the 1997 NHS (Primary Care) Act will gradually extend cash-limiting into primary health care, especially general practice. UK policy-makers have avoided providing clear, unambivalent direction about how to 'ration' NHS resources. The 'Child B' case became an epitome of public debate about NHS rationing. Among many other decision-making processes which occurred, Cambridge and Huntingdon Health Authority applied an ethical code to this rationing decision. Using new data this paper analyses the rationing criteria NHS managers (...) and clinicians used at local level in the Child B case; and the organisational structures which confronted them with such decisions. Primary Care Groups are likely to confront similar rationing decisions in respect of 'gate-kept' NHS services. However, such rationing processes are not so easily transposed to open-access services such as general practice. NHS rationing decisions, especially in PCGs, will require a much more specific ethical code than hitherto used. (shrink)

Vague predicates, those that exhibit borderline cases, pose a persistent problem for philosophers and logicians. Although they are ubiquitous in natural language, when used in a logical context, vague predicates lead to contradiction. This paper will address a question that is intimately related to this problem. Given their inherent imprecision, why do vague predicates arise in the first place? I discuss a variation of the signaling game where the state space is treated as contiguous, i.e., endowed with a metric that (...) captures a similarity relation over states. This added structure is manifested in payoffs that reward approximate coordination between sender and receiver as well as perfect coordination. I evolve these games using a variation of Herrnstein reinforcement learning that better reflects the generalizing learning strategies real-world actors use in situations where states of the world are similar. In these simulations, signaling can develop very quickly, and the signals are vague in much the way ordinary language predicates are vague—they each exclusively apply to certain items, but for some transition period both signals apply to varying degrees. Moreover, I show that under certain parameter values, in particular when state spaces are large and time is limited, learning generalization of this sort yields strategies with higher payoffs than standard Herrnstein reinforcement learning. These models may then help explain why the phenomenon of vagueness arises in natural language: the learning strategies that allow actors to quickly and effectively develop signaling conventions in contiguous state spaces make it unavoidable. (shrink)

Vague predicates, those that exhibit borderline cases, pose a persistent problem for philosophers and logicians. Although they are ubiquitous in natural language, when used in a logical context, vague predicates lead to contradiction. This paper will address a question that is intimately related to this problem. Given their inherent imprecision, why do vague predicates arise in the first place? I discuss a variation of the signaling game where the state space is treated as contiguous, i.e., endowed with a metric that (...) captures a similarity relation over states. This added structure is manifested in payoffs that reward approximate coordination between sender and receiver as well as perfect coordination. I evolve these games using a variation of Herrnstein reinforcement learning that better reflects the generalizing learning strategies real-world actors use in situations where states of the world are similar. In these simulations, signaling can develop very quickly, and the signals are vague in much the way ordinary language predicates are vague—they each exclusively apply to certain items, but for some transition period both signals apply to varying degrees. Moreover, I show that under certain parameter values, in particular when state spaces are large and time is limited, learning generalization of this sort yields strategies with higher payoffs than standard Herrnstein reinforcement learning. These models may then help explain why the phenomenon of vagueness arises in natural language: the learning strategies that allow actors to quickly and effectively develop signaling conventions in contiguous state spaces make it unavoidable. (shrink)

The stigmatization of some groups of people, whether for some characteristic they possess or some behavior they engage in, will initially strike most of us as wrong. For many years, academic work in public health, which focused mainly on the stigmatization of HIV-positive individuals, reinforced this natural reaction to stigmatization, by pointing out the negative health effects of stigmatization. But more recently, the apparent success of anti-smoking campaigns which employ stigmatization of smokers has raised questions about whether stigmatization may sometimes (...) be justified, because of its positive effects on public health. Discussion of the issue so far has focused on consequences, and on some Kantian considerations regarding the status of the stigmatized. In this article, I argue that further Kantian considerations regarding the treatment of the general public (the potential stigmatizers) also count against any public health policy involving stigmatization. Attempts to encourage stigmatization are likely to fail to appeal to the rational decision-making abilities of the general public, and the creation of stigmatized groups (even if they are stigmatized for their voluntary behavior) is an obstacle to the self-improvement of members of the general public. (shrink)

We reopen the investigation into the formal and conceptual relationship between bidirectional optimality theory (Blutner in J Semant 15(2):115–162, 1998 , J Semant 17(3):189–216, 2000 ) and game theory. Unlike a likeminded previous endeavor by Dekker and van Rooij (J Semant 17:217–242, 2000 ), we consider signaling games not strategic games, and seek to ground bidirectional optimization once in a model of rational step-by-step reasoning and once in a model of reinforcement learning. We give sufficient conditions for equivalence (...) of bidirectional optimality and the former, and show based on numerical simulations that bidirectional optimization may be thought of as a process of reinforcement learning with lateral inhibition. (shrink)

Oxford philosopher Nick Bostrom; in his recent and celebrated book Superintelligence; argues that advanced AI poses a potentially major existential risk to humanity; and that advanced AI development should be heavily regulated and perhaps even restricted to a small set of government-approved researchers. Bostrom’s ideas and arguments are reviewed and explored in detail; and compared with the thinking of three other current thinkers on the nature and implications of AI: Eliezer Yudkowsky of the Machine Intelligence Research Institute ; and David (...) Weinbaum and Viktoras Veitas of the Global Brain Institute. Relevant portions of Yudkowsky’s book Rationality: From AI to Zombies are briefly reviewed; and it is found that nearly all the core ideas of Bostrom’s work appeared previously or concurrently in Yudkowsky’s thinking. However; Yudkowsky often presents these shared ideas in a more plain-spoken and extreme form; making clearer the essence of what is being claimed. For instance; the elitist strain of thinking that one sees in the background in Bostrom is plainly and openly articulated in Yudkowsky; with many of the same practical conclusions. Bostrom and Yudkowsky view intelligent systems through the lens of reinforcement learning – they view them as “reward-maximizers” and worry about what happens when a very powerful and intelligent reward-maximizer is paired with a goal system that gives rewards for achieving foolish goals like tiling the universe with paperclips. Weinbaum and Veitas’s recent paper “Open-Ended Intelligence” presents a starkly alternative perspective on intelligence; viewing it as centered not on reward maximization; but rather on complex self-organization and self-transcending development that occurs in close coupling with a complex environment that is also ongoingly self-organizing; in only partially knowable ways. It is concluded that Bostrom and Yudkowsky’s arguments for existential risk have some logical foundation; but are often presented in an exaggerated way. For instance; formal arguments whose implication is that the “worst case scenarios” for advanced AI development are extremely dire; are often informally discussed as if they demonstrated the likelihood; rather than just the possibility; of highly negative outcomes. And potential dangers of reward-maximizing AI are taken as problems with AI in general; rather than just as problems of the reward-maximization paradigm as an approach to building superintelligence. If one views past; current; and future intelligence as “open-ended;” in the vernacular of Weaver and Veitas; the potential dangers no longer appear to loom so large; and one sees a future that is wide-open; complex and uncertain; just as it has always been. (shrink)

I introduce an extension of the Lewis-Skyrms signaling game, analysed from a dynamical perspective via simple reinforcement learning. In Lewis’ (Convention, Blackwell, Oxford, 1969) conception of a signaling game, salience is offered as an explanation for how individuals may come to agree upon a linguistic convention. Skyrms (Signals: evolution, learning & information, Oxford University Press, Oxford, 2010a) offers a dynamic explanation of how signaling conventions might arise presupposing no salience whatsoever. The extension of the atomic signaling game examined here—which (...) I will refer to as a salience game—introduces a variable parameter into the atomic signaling game which allows for degrees of salience, thus filling in the continuum between Skyrms’ and Lewis’ models. The model does not presuppose any salience at the outset, but illustrates a process by which accidentally evolved salience is amplified, to the benefit of the players. It is shown that increasing degrees of salience allow populations to avoid sub-optimal pooling equilibria and to coordinate upon conventions more quickly. (shrink)

Dual-systems frameworks propose that moral judgments are derived from both an immediate emotional response, and controlled/rational cognition. Recently Cushman (2013) proposed a new dual-system theory based on model-free and model-based reinforcement learning. Model-free learning attaches values to actions based on their history of reward and punishment, and explains some deontological, non-utilitarian judgments. Model-based learning involves the construction of a causal model of the world and allows for far-sighted planning; this form of learning fits well with utilitarian considerations (...) that seek to maximize certain kinds of outcomes. I present three concerns regarding the use of model-free reinforcement learning to explain deontological moral judgment. First, many actions that humans find aversive from model-free learning are not judged to be morally wrong. Moral judgment must require something in addition to model-free learning. Second, there is a dearth of evidence for central predictions of the reinforcement account—e.g., that people with different reinforcement histories will, all else equal, make different moral judgments. Finally, to account for the effect of intention within the framework requires certain assumptions which lack support. These challenges are reasonable foci for future empirical/theoretical work on the model-free/model-based framework. (shrink)

In lieu of an abstract, here is a brief excerpt of the content:Decolonizing Universality:Postcolonial Theory and the Quandary of Ethical AgencyEsha Niyogi De (bio)Living in colonial India, the Bengali thinker and creative writer Rabindranath Tagore (1861-1941) often meditated on ways that "concord" (milan) and "harmony" (sāmanjasya) could be established between persons and cultures [BIC 450-51]. Noting that "ruptures in balance and harmony" (bhār sāmanjasyer abhāv) that once were more localized now affected the whole world, he maintained that these reinforced the (...) progressive message of the modern age that the local "problems of a people are a part and parcel of humankind's" (swajātir samasyā samasta mānusher samasyār antargata) [RC 279]. In his view, the means to reestablish harmony in this unequal world lay in social practices and ideas conducing "freedom" (mukti) rather than enforcing "power" (shakti) and "coercive discipline" (shāsan) [SaS 408-09]. He argued that such progressive trends of practice and thought would be found not in one culture alone but through traveling across different cultural "seas of knowledge (jnān)" [SwSm 699]. For only through such conceptual journeys would we enkindle in ourselves "arguments with and doubts about" (yukti tarka o sandeher udbhav) our own cultures, and also learn from all the "voyagers in the path of progress" (unnatipather yātri) who emerge in different locations and times [SJ 39-40].As these comments reveal, Rabindranath maintained that inequalities could be rectified by making individual and collective norms and actions accountable to and measurable in terms of a speculative notion of universal humanity. While alert to the injustices proliferating from European colonization, he insisted that coercive practices and imbalances of power could not be entirely localized, and that in order to correct multiple forms of coercion we must seek across cultures and societies for (partial) achievements of human freedom and harmony. In so viewing transcultural negotiations of a common humane proclivity as the means to decolonize the world, Rabindranath did not stand alone but in a diverse company of anti-imperialist thinkers who adapted modern humanism to their ends: W. E. B. Dubois—who lauded Rabindranath's appropriation of [End Page 42] such liberal notions as democracy and equal opportunity "for greater ends" than were allowed by the "inconsistencies" of a racist European perspective [74]—Frantz Fanon, Jean-Paul Sartre, and from Rabindranath's own colonial context, the Bengali Anarchist poet Kazi Nazrul Islam (to name but a few). Reflecting upon worldwide experiences of subjection, these radical modern intellectuals tried above all to avoid reinventing separatism by declaring non-Western cultures to be unilaterally superior to Western imperialist culture. Driven by an activist ethics, they attempted instead to correct the roots of separatism through objectively evaluating, against a common frame of reference, right and wrong, relatively successful or failed practices of human empowerment and community.In this essay, I consider if and how we could characterize as "postcolonial" these evaluative ethics that rationally seek to correct coercive relations between persons and societies through referring to an affirmative idea of a human nature shared by all. If we take postcolonial agency to rest on a thorough critique of and resistance to all political and epistemic maneuvers that subordinate or exclude people and cultures, might this form of resistant agency reside in the ethical normativity of anti-imperialist humanism? This question elicits conflicting responses from contemporary postcolonial theorists.The argument, on the one hand, is that if postcolonial theory is to ethically address the multiple and interlocked hierarchies of a postimperial world, it indeed must take into account how people mobilize against separatism by working through critiques of ethnic divisiveness to negotiate a common human ground. Commenting on the universalism of thinkers such as Rabindranath Tagore and W. E. B. DuBois, Edward Said, for example, suggests that "vital" critiques of chauvinistic separatism arise only from viewing "local identity as not exhaustive" but as elaborating in reference to "larger, more generous realities of community between cultures, peoples, and societies" [Culture and Imperialism 217, 229]. His implication is that only through evaluating ourselves on the basis of a notion of noncoercive universality are we able to think against the identity politics that pervade our world, stemming at once from a nativist... (shrink)

With the advent of the era of artificial intelligence, deep reinforcement learning has achieved unprecedented success in high-dimensional and large-scale artificial intelligence tasks. However, the insecurity and instability of the DRL algorithm have an important impact on its performance. The Soft Actor-Critic algorithm uses advanced functions to update the policy and value network to alleviate some of these problems. However, SAC still has some problems. In order to reduce the error caused by the overestimation of SAC, we propose (...) a new SAC algorithm called Averaged-SAC. By averaging the previously learned action-state estimates, it reduces the overestimation problem of soft Q-learning, thereby contributing to a more stable training process and improving performance. We evaluate the performance of Averaged-SAC through some games in the MuJoCo environment. The experimental results show that the Averaged-SAC algorithm effectively improves the performance of the SAC algorithm and the stability of the training process. (shrink)

There has been a movement aiming to teach agents about their privilege by making the information about their privilege as costless as possible. However, some argue that in risk-sensitive frameworks, such as Lara Buchak’s (2013), it can be rational for privileged agents to shield themselves from learning about their privilege, even if the information is costless and relevant. This threatens the efficacy of these information-access efforts in alleviating the problem of elite-group ignorance. In response, I show that even within (...) the same framework, in this case David Kinney and Liam Kofi Bright’s (2021), the rationality of this information avoidance rests on shaky ground in practice. In this framework, whether an agent should avoid information depends on the precise details of (1) how relevant they expect the information to be, (2) their priors about the value of various options, and (3) their risk attitudes. The model suggests that the rationality of elite-group ignorance is a function of structural factors that are pervasive but nonetheless not insurmountable, thus offering a way out of pessimism about elite group education. (shrink)

In lieu of an abstract, here is a brief excerpt of the content:Math Anxiety:Making Room to BreatheValerie Allen (bio) and Todd Stambaugh (bio)"Don't do that to me, Professor," the student said, and everybody laughed, for by this late in the semester, the atmosphere was relaxed. The instructor in question had just reached the point in a worked problem when they could move from reasoning about specific numbers to stating a general principle: x≤y≤z, meaning that y—the value we sought—was always going (...) to be greater than or equal to one value (x) and less than or equal to the other (z). The student who had spoken had earned everyone's affection over the weeks, open about their difficulties with math, ready to risk being wrong, and to ask obvious questions that others may have been reluctant to owning. They had hung in there, despite presenting a classic case of math anxiety, a syndrome long recognized in the scholarship of teaching and learning.Among math anxiety's many symptoms, breathing disruption ranks high: hyperventilation (rapid breathing) and hypoventilation (barely or not breathing). When the energy-conserving parasympathetic nervous system is overtaken by the energy-expending sympathetic nervous system, the long exhalations that decelerate the heart rate are replaced by short, panting puffs from the upper lungs. The timed test provides the perfect environment for math anxiety to flourish. Stress diverts cognitive resources needed for procedural execution, so working memory consequently shrinks.Self-rated metrics for math anxiety have been in place for decades (Richardson and Suinn). Relatively trivial interventions can yield nontrivial results: deep breathing exercises and positive affirmations have a beneficial impact, as does writing about one's nerves immediately prior to an exam (Brunyé et al.). More systemic changes require a reorientation in teaching whereby conceptual mistakes are positively rewarded as opportunities for learning; students compose questions instead of answers; students solve problems together rather than perform solo (Boaler).Return now to that opening scenario to watch it become stereotype incarnate: the student was mature, Black, and female; the teacher, younger, white, and male; her comment was triggered by his shift from a specific [End Page 217] set of numbers into symbolic notation. There is robust evidence to demonstrate that certain demographics are consistently underrepresented in the U.S. math community: women and "African-Americans, Hispanics, American Indians, Alaska Natives, and Pacific Islanders," according to the Mathematical Association of America (MAA) ("Resources for Minority"). Routinely feeling that they do not belong, such groups are particularly vulnerable to math anxiety.Several remedies for such disparity suggest themselves: appointing more women and people of color in the profession; one-on-one tutorial assistance; research projects about the students' personal environment; privileging pre-test, low-stakes work over timed tests; assigning group rather than individual work. When all is said and done, however, such strategies "have little to do with mathematics content" (Martin 304). Mathematicians still need to be able to conceptualize, describe, and manipulate abstract objects, and there seems to be a suspicion that, at heart, abstraction is gendered masculine and colored white.It is possible that the rhetorical immediacy and context-dependence of Black English has misguidedly reinforced such a perception. For poet June Jordan, Black English "devolves from a culture that abhors all abstraction" (Jordan 367). Psychiatrist Alvin Poussaint asks, "Is there a way to say metamorphosis in Ebonics?" (O'Malley). Yet, despite Black English's avoidance of nominalization and similar verbosities that afflict academic prose, abstract logical reasoning flourishes in vernacular idiom, as linguist William Labov demonstrated half a century ago. Our aim is not to explain the asymptote in Black English but rather to consider a feature characteristic of those very minority cultures historically underrepresented in math—storytelling—and to ask how it may be foregrounded in mathematics, where it is decisively sidelined. Storytelling has been claimed by every culture and ethnic group at some point as the repository of wisdom, practical skills, religious beliefs, and shared identity and values; but when it gets corralled into the category of "fiction," and when it gets opposed to "fact," a narrowed concept of truth dispatches storytelling to kindergarten and recreational math. Word problems may supply scenarios but offer none of a plot's satisfactions... (shrink)

The goal of this essay is to explore "the highly contested nature of [decision-making through adopting] a historically comparative and interdisciplinary approach." Internalist history of game theory treats decision theory as a science of making choices to maximize expected gain. Game theory is applied to nuclear deterrence and military strategy, building markets and designing institutions, analyzing collective action, developing jurisprudence, and addressing crime and punishment. This essay draws on recent historiography of Cold War decision-making to draw into focus the constructive (...) aspects of decision theory to argue that the perceived need to avoid worst-case scenarios has instead contributed to a world in which the worst outcome is probabilistically assured to occur over the long duree timeframe. (shrink)

We consider the following signaling game. Nature plays first from the set {1, 2}. Player 1 (the Sender) sees this and plays from the set {A, B}. Player 2 (the Receiver) sees only Player 1’s play and plays from the set {1, 2}. Both players win if Player 2’s play equals Nature’s play and lose otherwise. Players are told whether they have won or lost, and the game is repeated. An urn scheme for learning coordination in this game is as (...) follows. Each node of the desicion tree for Players 1 and 2 contains an urn with balls of two colors for the two possible decisions. Players make decisions by drawing from the appropriate urns. After a win, each ball that was drawn is reinforced by adding another of the same color to the urn. A number of equilibria are possible for this game other than the optimal ones. However, we show that the urn scheme achieves asymptotically optimal coordination. (shrink)

Can humans get arbitrarily capable reinforcement learning agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? This question impacts how far RL can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we study when an RL agent has an instrumental goal to tamper with its reward process, and describe design principles (...) that prevent instrumental goals for two different types of reward tampering. Combined, the design principles can prevent reward tampering from being an instrumental goal. The analysis benefits from causal influence diagrams to provide intuitive yet precise formalizations. (shrink)

The first reference on rationality that integrates accounts from psychology and philosophy, covering descriptive and normative theories from both disciplines. Both analytic philosophy and cognitive psychology have made dramatic advances in understanding rationality, but there has been little interaction between the disciplines. This volume offers the first integrated overview of the state of the art in the psychology and philosophy of rationality. Written by leading experts from both disciplines, The Handbook of Rationality covers the main normative and descriptive theories of (...) rationality—how people ought to think, how they actually think, and why we often deviate from what we can call rational. It also offers insights from other fields such as artificial intelligence, economics, the social sciences, and cognitive neuroscience. The Handbook proposes a novel classification system for researchers in human rationality, and it creates new connections between rationality research in philosophy, psychology, and other disciplines. Following the basic distinction between theoretical and practical rationality, the book first considers the theoretical side, including normative and descriptive theories of logical, probabilistic, causal, and defeasible reasoning. It then turns to the practical side, discussing topics such as decision making, bounded rationality, game theory, deontic and legal reasoning, and the relation between rationality and morality. Finally, it covers topics that arise in both theoretical and practical rationality, including visual and spatial thinking, scientific rationality, how children learn to reason rationally, and the connection between intelligence and rationality. (shrink)

I present a game-theoretical multi-agent system to simulate the evolutionary process responsible for the pragmatic phenomenon division of pragmatic labour (DOPL), a linguistic convention emerging from evolutionary forces. Each agent is positioned on a toroid lattice and communicates via signaling games , where the choice of an interlocutor depends on the Manhattan distance between them. In this framework I compare two learning dynamics: reinforcement learning (RL) and belief learning (BL). An agent’s experiences from previous plays influence his communication behaviour, (...) and RL agents act in a non-rational way whereas BL agents display a small degree of rationality by using best response dynamics . The complete system simulates an evolutionary process of communication strategies, which agents can learn in a structured spatial society. The significant questions are: what circumstances could lead to an evolutionary process that doesn’t result in the expected DOPL convention; and to what extent is interlocutor rationality necessary for the emergence of a society-wide convention à la DOPL? (shrink)

Reinforcement learning approaches to cognitive modeling represent task acquisition as learning to choose the sequence of steps that accomplishes the task while maximizing a reward. However, an apparently unrecognized problem for modelers is choosing when, what, and how much to reward; that is, when (the moment: end of trial, subtask, or some other interval of task performance), what (the objective function: e.g., performance time or performance accuracy), and how much (the magnitude: with binary, categorical, or continuous values). (...) In this article, we explore the problem space of these three parameters in the context of a task whose completion entails some combination of 36 state–action pairs, where all intermediate states (i.e., after the initial state and prior to the end state) represent progressive but partial completion of the task. Different choices produce profoundly different learning paths and outcomes, with the strongest effect for moment. Unfortunately, there is little discussion in the literature of the effect of such choices. This absence is disappointing, as the choice of when, what, and how much needs to be made by a modeler for every learning model. (shrink)

This paper presents a GA-based multi-agent reinforce- ment learning bidding approach (GMARLB) for perform- ing multi-agent reinforcement learning. GMARLB inte- grates reinforcement learning, bidding and genetic algo- rithms. The general idea of our multi-agent systems is as follows: There are a number of individual agents in a team, each agent of the team has two modules: Q module and CQ module. Each agent can select actions to be performed at each step, which are done by the Q module. (...) While the CQ module determines at each step whether the agent should continue or relinquish control. Once an agent relinquishes its control, a new agent is selected by bidding algorithms. We applied GA-based GMARLB to the Backgammon game. The experimental results show GMARLB can achieve a su- perior level of performance in game-playing, outperforming PubEval, while the system uses zero built-in knowledge. (shrink)

Tetris provides a difficult, dynamic task environment within which some people are novices and others, after years of work and practice, become extreme experts. Here we study two core skills; namely, choosing the goal or objective function that will maximize performance and a feature-based analysis of the current game board to determine where to place the currently falling zoid so as to maximize the goal. In Study 1, we build cross-entropy reinforcement learning models to determine whether different goals result (...) in different feature weights. Two of these optimization strategies quickly rise to performance plateaus, whereas two others continue toward higher but more jagged heights. In Study 2, we compare the zoid placement decisions made by our best CERL models with those made by 67 human players. Across 370,131 human game episodes, two CERL models picked the same zoid placements as our lowest scoring human for 43% of the placements and as our three best scoring experts for 65% of the placements. Our findings suggest that people focus on maximizing points, not number of lines cleared or number of levels reached. They also show that goal choice influences the choice of zoid placements for CERLs and suggest that the same is true of humans. Tetris has a repetitive task structure that makes Tetris more tractable and more like a traditional experimental psychology paradigm than many more complex games or tasks. Hence, although complex, Tetris is not overwhelmingly complex and presents a right-sized challenge to cognitive theories, especially those of integrated cognitive systems. (shrink)

We consider Effort Games, a game-theoretic model of cooperation in open environments, which is a variant of the principal-agent problem from economic theory. In our multiagent domain, a common project depends on various tasks; carrying out certain subsets of the tasks completes the project successfully, while carrying out other subsets does not. The probability of carrying out a task is higher when the agent in charge of it exerts effort, at a certain cost for that agent. A central authority, called (...) the principal, attempts to incentivize agents to exert effort, but can only reward agents based on the success of the entire project.We model this domain as a normal form game, where the payoffs for each strategy profile are defined based on the different probabilities of carrying out each task and on the boolean function that defines which task subsets complete the project, and which do not. We view this boolean function as a simple coalitional game, and call this game the underlying coalitional game. We suggest the Price of Myopia as a measure of the influence the model of rationality has on the minimal payments the principal has to make in order to motivate the agents in such a domain to exert effort.We consider the computational complexity of testing whether exerting effort is a dominant strategy for an agent, and of finding a reward strategy for this domain, using either a dominant strategy equilibrium or using iterated elimination of dominated strategies. We show these problems are generally #P-hard, and that they are at least as computationally hard as calculating the Banzhaf power index in the underlying coalitional game. We also show that in a certain restricted domain, where the underlying coalitional game is a weighted voting game with certain properties, it is possible to solve all of the above problems in polynomial time. We give bounds on PoM in weighted voting effort games, and provide simulation results regarding PoM in another restricted class of effort games, namely effort games played over Series-Parallel Graphs. (shrink)

Stanovich & West's target article undervalues the power of implicit learning (particularly reinforcement learning). Implicit learning may allow the learning of more rational responses–and sometimes even generalisation of knowledge–in contexts where explicit, abstract knowledge proves only of limited value, such as for economic decision-making. Four other comments are made.