1. Introduction 2. Reward-Guided Decision Making 3. Content in the Model 4. How to Deflate a Metarepresentational Reading Proust and Carruthers on metacognitive feelings 5. A Deflationary Treatment of RPEs? 5.1 Dispensing with prediction errors 5.2 What is use of the RPE focused on? 5.3 Alternative explanations—worldly correlates 5.4 Contrast cases 6. Conclusion Appendix: Temporal Difference Learning Algorithms.

Can an agent's intelligence level be negative? We extend the Legg-Hutter agent-environment framework to include punishments and argue for an affirmative answer to that question. We show that if the background encodings and Universal Turing Machine (UTM) admit certain Kolmogorov complexity symmetries, then the resulting Legg-Hutter intelligence measure is symmetric about the origin. In particular, this implies reward-ignoring agents have Legg-Hutter intelligence 0 according to such UTMs.

In philosophy, the textbook case for the discussion of human motivation is the examination (and almost always, the refutation) of psychological egoism. The arguments have become part of the folklore of our tribe, from their inclusion in countless introductory texts. [...] One of my central aims has been to define the issues empirically, so we do not just settle them by definition. Although I am inclined at present to put my bets on the reward-event theory, with its internalism, monism, and (...) causal primacy of satisfaction, I think we are very far from knowing enough to settle these questions concerning motivation, human or otherwise. The winds of science will blow where they may. In the meantime, we can be a bit more circumspect about what we put in our tribal folklore. (shrink)

Can humans get arbitrarily capable reinforcement learning agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? This question impacts how far RL can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we study when an RL agent has an instrumental goal to tamper with its reward process, and describe design principles that prevent instrumental (...) goals for two different types of reward tampering. Combined, the design principles can prevent reward tampering from being an instrumental goal. The analysis benefits from causal influence diagrams to provide intuitive yet precise formalizations. (shrink)

When faced with intertemporal choices, which have consequences that unfold over time, we often discount the future, preferring smaller immediate rewards often at the expense of long-term benefits. How psychologically connected one feels to one’s future self-influences such temporal discounting. Psychological connectedness consists in sharing psychological properties with past or future selves, but connectedness comes in degrees. If one feels that one is not psychologically connected to one’s future self, one views that self like a different person and is (...) less likely to wait for the future reward. Increasing perceived psychological connectedness to one’s future self may lead to more far-sighted decisions. Episodic prospection may help in this regard. Episodic prospection is our ability to ‘pre-experience’ the future by mentally simulating it, drawing on information from episodic memory and other sources. Episodic memory and prospection are thought to involve a special form of consciousness, which underpins the capacity to appreciate the connection between one’s past, present, and future selves. Simulating the future self through prospection may increase felt psychological connectedness and support future-oriented decision-making. Yet this is where a puzzle arises. The imagery of episodic memory and prospection is perspectival: often one views the visualised scenario from a detached perspective, seeing oneself from-the-outside as if viewing another person. The aim of this paper is to characterise how the perspectival imagery of prospection relates to psychological connectedness, and to show that even though such imagery involves a detached perspective it can still be used to help reward one’s future self. (shrink)

Systems of logico-probabilistic reasoning characterize inference from conditional assertions that express high conditional probabilities. In this paper we investigate four prominent LP systems, the systems _O, P_, _Z_, and _QC_. These systems differ in the number of inferences they licence _. LP systems that license more inferences enjoy the possible reward of deriving more true and informative conclusions, but with this possible reward comes the risk of drawing more false or uninformative conclusions. In the first part of the paper, we (...) present the four systems and extend each of them by theorems that allow one to compute almost-tight lower-probability-bounds for the conclusion of an inference, given lower-probability-bounds for its premises. In the second part of the paper, we investigate by means of computer simulations which of the four systems provides the best balance of reward versus risk. Our results suggest that system _Z_ offers the best balance. (shrink)

Rewards are reinforcement mechanisms that organizations use to shape desirable employee behaviors. However, rewards may also have unintended consequences, such as building expectations for receiving extra benefits and weakening employee barriers to unethical acts. This article investigates the dark side of the reward–behavior association, and exploring what is referred to as the reward–theft parity effect (RTPE). The authors hypothesize that receiving rewards induces a corresponding type of theft. In Study 1, survey results (n = 634) show initial (...) support for the RTPE between rewards and same‐category theft, and that perceptions of procedural justice attenuate this association. Study 2 involved a 2 (reward) × 2 (theft opportunity) between‐subjects experimental design (n = 300) and provides further support for the RTPE by revealing greater theft under a reward–theft parity condition than under a reward–theft disparity condition. Study 3 replicates these findings in a field experiment (n = 285) and finds that the RTPE is exacerbated when the reward–behavior task is perceived as unjust. Collectively, these findings provide new theoretical insights while also giving managers a more complete understanding of the dark side of behavioral reinforcement mechanisms. (shrink)

According to the reward-prediction error hypothesis of dopamine, the phasic activity of dopaminergic neurons in the midbrain signals a discrepancy between the predicted and currently experienced reward of a particular event. It can be claimed that this hypothesis is deep, elegant and beautiful, representing one of the largest successes of computational neuroscience. This paper examines this claim, making two contributions to existing literature. First, it draws a comprehensive historical account of the main steps that led to the formulation and subsequent (...) success of the RPEH. Second, in light of this historical account, it explains in which sense the RPEH is explanatory and under which conditions it can be justifiably deemed deeper than the incentive salience hypothesis of dopamine, which is arguably the most prominent contemporary alternative to the RPEH. (shrink)

A developing neurobiological/psychological theory of positive motivation gives a key causal role to reward events in the brain which can be directly activated by electrical stimulation (ESB). In its strongest form, this Reward Event Theory (RET) claims that all positive motivation, primary and learned, is functionally dependent on these reward events. Some of the empirical evidence is reviewed which either supports or challenges RET. The paper examines the implications of RET for the concepts of 'motivation', 'desire' and 'reward' or 'pleasure'. (...) It is argued (1) that a 'causal base' as opposed to a functional' concept of motivation has theoretical advantages; (2) that a causal distinction between the focus' and the 'anchor' of desire suggests an ineliminable 'opacity' of desire; and (3) that some affective concept, such as 'pleasure', should play a key role in psychological explanation, distinct from that of motivational (or cognitive) concepts. A concept of 'reward' or 'pleasure' as intrinsically positive affect is defended, and contrasted with the more 'operational' definitions of 'reward' in some of the hypotheses of Roy Wise. (shrink)

This qualitative study of 27 women animal activists examines the risks and rewards that accompany a commitment to animal rights activism. One of the common beliefs about animal rights activists is that their political choices are fanatic and unyielding, resulting in rigid self-denial. Contrary to this notion, the women in this study experienced both the pain and the joy of their transformation toward animal activism. Activism took an enormous toll on their personal relationships, careers, and emotional well being. They (...) struggled as friendships ended and family relationships suffered; some experienced harassment and abuse as a result of their efforts. Yet the women were just as likely to extol the rewards and pleasure gained from their participation in the cause of animal liberation. These included a heightened awareness of political issues, greater self-confidence, the feeling that they were making a difference in the world, and the joy of living a “more meaningful life.”. (shrink)

RÉSUMÉDans Scientific Ontology: Integrating Naturalized Metaphysics and Voluntarist Epistemology, je soutiens que les convictions ontologiques associées à la recherche scientifique sont imprégnées de convictions philosophiques. Les interprétations de l'ontologie scientifique impliquent ce que j'appelle des inférences métaphysiques et, qui plus est, il existe différentes façons de faire ces inférences sur la base de positions épistémiques différentes, mais néanmoins rationnelles. Si cette analyse est juste, elle problématise toute distinction nette entre la métaphysique naturalisée et les autres types de métaphysique, et dissout (...) toute présomption qu'il existe une seule bonne réponse aux questions ontologiques liées aux sciences. Dans cet article, je considère quelques-uns des importants défis qu'Amanda Bryant, Stathis Psillos et Matthew Slater posent à l'endroit de ces affirmations. (shrink)

The brain and emotion treats emotions as states elicited by reinforcers (reward or punishment), but it is unclear how this view can do justice to the diversity of emotions. It is also unclear how such a view distinguishes emotions from states such as hunger and thirst. A complementary approach to understanding emotions may begin by considering emotions as aspects of social cognition.

Since 2010, Section 922 of the Dodd-Frank Act has required the Securities and Exchange Commission to give a significant financial reward to any whistleblower who voluntarily discloses original information concerning fraud or other unlawful activity. How, if at all, might such “incentives” change our understanding of whistleblowing? My answer is that, while incentives should not change the definition of whistleblowing, it should change our understanding of the justification of whistleblowing. We need to distinguish the public justification of whistleblowing, its public (...) defense, from its personal justification, the defense the whistleblower should be able to present to conscience. Motive is relevant only to the personal justification—but, since motive is relevant to that, statutes should be written so as not to interfere with personal justification, for example, by offering incentives when compensation would do. Though the paper’s focus is Section 922, its argument and conclusions are quite general. (shrink)

Reward and punishment sensitivity seem important traits in understanding behavior in general and psychopathology in particular. Though the definitions used for reward and punishment sensitivity differentiate between responsivity and motivation, the measures thus far used to assess these constructs do not. Further, specificity of the type of reward and punishment in questionnaires might result in measurement bias especially when examining the relationship with psychopathology. Therefore, we developed a stimulus-independent multidimensional questionnaire of reward and punishment sensitivity that differentiates between responsivity and (...) motivation. This study addresses the psychometric qualities of this newly developed reward and punishment responsivity and motivation questionnaire. On the basis of exploratory ordinal factor analysis that was used to examine the quality of the initial pool of 39 items, the number of items was reduced to 18. Confirmatory ordinal factor analysis on the remaining items in an independent sample supported a 18-item four-factor model, and showed acceptable to good internal reliability. The relationship between the subscales of the RPRM-Q and often used questionnaires was examined in the combined sample, which showed some first support for the ability of the new questionnaire to differentiate between responsivity and motivation to approach/avoid. The findings indicate that the RPRM-Q might be a helpful instrument to further test the relevance of punishment and reward sensitivity in psychopathology. (shrink)

While it is known that reward induces attentional prioritization, it is not clear what effect reward-learning has when associated with stimuli that are not fully perceived. The masked priming paradigm has been extensively used to investigate the indirect impact of brief stimuli on response behavior. Interestingly, the effect of masked primes is observed even when participants choose their responses freely. While classical theories assume this process to be automatic, recent studies have provided evidence for attentional modulations of masked priming effects. (...) Most such studies have manipulated bottom-up or top-down modes of attentional selection, but the role of “newer” forms of attentional control such as reward-learning and selection history remains unclear. In two experiments, with number and arrow primes, we examined whether reward-mediated attentional selection modulates masked priming when responses are chosen freely. In both experiments, we observed that primes associated with high-reward lead to enhanced free-choice priming compared to primes associated with no-reward. The effect was seen on both proportion of choices and response times, and was more evident in the faster responses. In the slower responses, the effect was diminished. Our study adds to the growing literature showing the susceptibility of masked priming to factors related to attention and executive control. (shrink)

In the absence of explicit punitive sanctions, why do individuals voluntarily participate in intergroup warfare when doing so incurs a mortality risk? Here we consider the motivation of individuals for participating in warfare. We hypothesize that in addition to other considerations, individuals are incentivized by the possibility of rewards. We test a prediction of this “cultural rewards war-risk hypothesis” with ethnographic literature on warfare in small-scale societies. We find that a greater number of benefits from warfare is associated (...) with a higher rate of death from conflict. This provides preliminary support for the relationship between rewards and participation in warfare. (shrink)

The Nuffield Council on Bioethics’ 2011 report, Human Bodies: Donation for Medicine and Research, proposes a system for examining the ethical implications of different types of incentives for the provision of human tissue for use in medicine and research. The cornerstone of this system is the principle of altruism which, the Council recommends, should, where possible, remain the starting point for any such tissue provision. Using the Council's example of ova provision for research as an area in which altruism-based (...) class='Hi'>rewards might be departed from, this article argues that such a system has the potential to become inconsistent and unnecessarily complex. It suggests that the outcomes-focussed and motivations-focussed justifications the Council provides do not sit easily within the fast-paced and unpredictable area of biotechnology research. Further, it may undermine the focus on autonomy that is enshrined in the relevant legislation. This article suggests that a fair system for incentivising and rewarding the provision of human tissue in research should be developed, which focuses on elements of this role that are common to all tissue providers. (shrink)

Humans expect resources to be distributed fairly. They also show biases to construe all acts as intentional. This study investigates whether every unequal distribution is initially assumed to be intentional unfairness. Study 1 presents a control group of adults with a movie showing one individual accidentally receiving less reward than expected for a task. The experimental group was shown the same scenario, except that the individual was now in the presence of an additional person who received the full reward. Despite (...) the similarity of the scenarios, as predicted, participants in the control condition responded as if the disappointing reward was accidental, while those in the experimental condition responded as if the act was intentional: Their tendency to avoid the “perpetrator” did not differ from that of participants in another control condition who saw an intentionally unfair reward distribution. In Study 2, 7- and 8-year-old children’s results replicated those of adults. Implications for social and moral cognition are discussed. (shrink)

According to the reward-prediction error hypothesis (RPEH) of dopamine, the phasic activity of dopaminergic neurons in the midbrain signals a discrepancy between the predicted and currently experienced reward of a particular event. It can be claimed that this hypothesis is deep, elegant and beautiful, representing one of the largest successes of computational neuroscience. This paper examines this claim, making two contributions to existing literature. First, it draws a comprehensive historical account of the main steps that led to the formulation and (...) subsequent success of the RPEH. Second, in light of this historical account, it explains in which sense the RPEH is explanatory and under which conditions it can be justifiably deemed deeper than the incentive salience hypothesis of dopamine, which is arguably the most prominent contemporary alternative to the RPEH. (shrink)

In this paper, we raise and discuss a puzzle about the relationships among goods, reasons, and deontic status. Suppose you have it within your power to give someone something they would enjoy. The following claims seem platitudinous: you can use this power to reward whatever kind of option you want, thereby making that option better and generating a reason for that person to perform it; this reason is then weighed alongside and against the other reasons at play; and altogether, the (...) reasons determine the deontic statuses of that person’s options. We show, however, that in a certain class of cases at least one of these apparent platitudes must be false. In particular, we show that in a certain kind of case wrongfulness cannot be rewarded. In some cases, if one tries to reward wrongfulness, something surprising must go awry: either what you attempt to give as a reward would not, in fact, be good, or it would not generate a reason, or it would have a surprising effect on the deontic status of the relevant options. The upshot is that the relationships among goods, reasons, and deontic status are complicated in ways that have not previously been remarked. (shrink)

Recent philosophical work has praised the reward structure of science, while recent empirical work has shown that many scientific results may not be reproducible. I argue that the reward structure of science incentivizes scientists to focus on speed and impact at the expense of the reproducibility of their work, thus contributing to the so-called reproducibility crisis. I use a rational choice model to identify a set of sufficient conditions for this problem to arise, and I argue that these conditions plausibly (...) apply to a wide range of research situations. Currently proposed solutions will not fully address this problem. Philosophical commentators should temper their optimism about the reward structure of science. (shrink)

People differ in the extent to which they discount the values of future rewards. Behavioural economists measure these differences in terms of functions that describe rates of reduced valuation in the future – temporal discounting – as these vary with time. They measure differences in preference for risk – differing rates of probability discounting – in terms of similar functions that describe reduced valuation of rewards as the probability of their delivery falls. So-called ‘impulsive’ people, including people disposed (...) to addiction, tend to choose rewards in patterns that are described by unusually steeply sloped discount functions. The empirical literature is ambivalent as to whether this applies to pathological gamblers (PGs) who manifest most other behavioural patterns associated with addiction, with different, equally careful studies suggesting opposite conclusions (Petry & Casarella 1999; Holt, Green & Myerson 2003). This puzzle may arise from the fact that most previous experiments on discounting behaviour were not designed so as to allow effects of high temporal discounting to be distinguished from effects of low probability discounting. (Addiction has sometimes been associated with the latter because it seems to involve, at least in the starting addict, under appreciation of risk.) Our research project investigates both forms of discounting behaviour and their relationship to severity of risk of PG in a community sample of South African gamblers. (shrink)

Research on academic pay finds an unexplained gender pay gap that has not fully dissolved over time and that appears to increase with years of experience. In this study, I consider how role congruity bias contributes to this pay gap. Bias is more likely to manifest in a context where there is some ambiguity about performance and where stereotypes are stronger. I predict that bias in the attribution of credit for coauthored research leads to lower returns to research for female (...) professors. To test this prediction, I use a sample of Canadian faculty in accounting, where research is typically coauthored, where females are underrepresented at the most senior ranks, and where many universities evaluate merit in research, teaching, and service to determine salary increases. In regressions of salary on individual and institutional determinants of salary, I find that women earn marginally less than men. However, the pay gap is only evident for women who publish in a selective list of journals; for the subset of faculty with no publications from this list, there are no significant differences in salary. For researching faculty, the pay gap relates specifically to research productivity. While women publish less on average than men, the returns to their research are also lower. In particular, the relation between the individual’s research ranking and salary is significantly lower for women who publish a higher proportion of their work with men, than for all other faculty. Additional analysis of salary and coauthor patterns confirms that women receive significantly less credit for coauthored articles they publish with men than those they publish with other women but that no similar variations in reward are evident for men across publishing patterns. These findings suggest bias in the attribution of credit for coauthored research in the determination of salary, consistent with role congruity theory, and provide an important potential explanation for why salaries for women vary systematically from those of men even after considering productivity. Providing lower rewards for equal work represents a continuing ethical issue in academe and compounds the challenges women already facing in the profession. (shrink)

Rolls's book depends significantly on a definition relating emotion to reward and learning. This definition confuses two separable concepts, and may result in the exclusion of notions of emotion and motivation from lower animals that may possess limited learning capacities. A more useful definition might revolve around the notion that emotions are states that function to optimize the performance of behavior.

Past studies have shown that the perceived time of actions is retrospectively influenced by post-action events. The current study examined whether rewarding performance feedback altered the reported time of action. In Experiment 1, participants performed a speeded button press task and received monetary reward for a presumed “fast,” or a monetary punishment for a presumed “slow” response. Rewarded trials resulted in the false perception that the response action occurred earlier than punished trials. In Experiments 2 and 3, the need for (...) a speeded response and reward were independently manipulated in order to decouple the cognitive and reward components in the feedback signal. When tested independently, neither variable affected the judged time of action. We conclude that meaningful feedback is only used when made salient by reward, to modulate the judged time of an action. (shrink)