Can humans get arbitrarily capable reinforcement learning agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? This question impacts how far RL can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we study when an RL agent has an instrumental goal to tamper with its reward process, and describe design (...) principles that prevent instrumental goals for two different types of reward tampering. Combined, the design principles can prevent reward tampering from being an instrumental goal. The analysis benefits from causal influence diagrams to provide intuitive yet precise formalizations. (shrink)

ABSTRACT The list of proposed addictions has recently grown to include television, videogames, shopping, day trading, kleptomania, and use of the Internet. These activities share with a more established entry, gambling, the property that they require no delivery of a biological stimulus that might be thought to unlock a hardwired brain process. I propose a framework for analyzing that class of incentives that do not depend on the prediction of physically privileged environmental events: people have a great capacity to coin (...) endogenous reward; we learn to cultivate it, and, where it is entrapping, to minimize it, by managing internally generated appetites for it. The basic method of cultivating endogenous reward is to learn cues that predict when best to harvest the reward that has been made possible by the growth of these appetites. This hedonic management occurs in the same motivational marketplace as the instrumental planning that seeks environmental goods in the conventional manner, and presumably obeys the same laws of temporal difference learning; but these laws are no longer limiting. Furthermore, instrumental contingencies often provide the most productive structure for hedonic management as well, for reasons that I discuss; but the needs of hedonic management create incentives both to pursue instrumental goals in a suboptimal manner and to avoid noticing how the hedonic incentive affects this pursuit. The result is the apparent irrationality that is often observed in process addictions. (shrink)

Self-relevance has been demonstrated to impair instrumental learning. Compared to unfamiliar symbols associated with a friend, analogous stimuli linked with the self are learned more slowly. What is not yet understood, however, is whether this effect extends beyond arbitrary stimuli to material with intrinsically meaningful properties. Take, for example, stimulus valence an established moderator of self-bias. Does the desirability of to-be-learned material influence self-learning? Here, in conjunction with computational modelling (i.e. Reinforcement Learning Drift Diffusion Model analysis), (...) a probabilistic selection task was used to establish if and how stimulus valence (i.e. desirable/undesirable posters) impacts the acquisition of knowledge relating to object-ownership (i.e. owned-by-self vs. owned-by-friend). Several interesting results were observed. First, undesirable posters were learned more rapidly for self compared to friend, an effect that was reversed for desirable posters. Second, learning rates were accompanied by associated differences in reward sensitivity toward desirable and undesirable choice selections as a function of ownership. Third, decisional caution was greater for self-relevant (vs. friend relevant) responses. Collectively, these findings inform understanding of self-function and how valence and stimulus relevance mutually influence probabilistic learning. (shrink)

BackgroundOne of the preferences working in the Generation Z is based on their motivational work values. The relevance of job choices with the work values will contribute to student career planning. The work value instrument among generations is one of the popular instruments used to measure final year students' work value, yet few studies of the psychometric properties of non-English language versions of this instrument. This study's objectives were to adapt a questionnaire of work value in Indonesian final year university (...) students.MethodsThe number of participants in this study was 316 students in Indonesia, comprised of final year students from various majors who were selected by quota sampling. The instrument consisted of 5 dimensions of value, including leisure, extrinsic rewards, intrinsic rewards, altruistic rewards, and social rewards. The reliability analysis was performed using McDonald's Omega, the evidence of validity was obtained from test content, internal structure through confirmatory factor analysis, and evidence-based in relation to other variable has conducted the correlation between work value and career development learning using the Pearson's correlation coefficient.ResultsThe results showed that the work values instrument had good psychometric properties, including good reliability, good content validity, and internal structure. In CFA, the two-factor structure showed satisfactory model fit. Moreover, the correlation of work value with career development learning builds stronger validity evidence on this instrument.ConclusionThe adapted instrument can be used practically to identify work value preferences of final year students to help them choose a work preference and setup the career planning before graduating. The result could be of interest for the researcher in work value, motivational work, and career areas in higher education. To the best of our knowledge, there have been no reports about the adaptation of work value instruments in Indonesian final year university students. (shrink)

Across species, animals have an intrinsic drive to approach appetitive stimuli and to withdraw from aversive stimuli. In affective science, influential theories of emotion link positive affect with strengthened behavioural approach and negative affect with avoidance. Based on these theories, we predicted that individuals’ positive and negative affect levels should particularly influence their behaviour when innate Pavlovian approach/avoidance tendencies conflict with learned instrumental behaviours. Here, across two experiments – exploratory Experiment 1 (N = 91) and a preregistered confirmatory Experiment (...) 2 (N = 335) – we assessed how induced positive and negative affect influenced Pavlovian-instrumental interactions in a reward/punishment Go/No-Go task. Contrary to our hypotheses, we found no evidence for a main effect of positive/negative affect on either approach/avoidance behaviour or Pavlovian-instrumental interactions. However, we did find evidence that the effects of induced affect on behaviour were moderated by individual differences in self-reported behavioural inhibition and gender. Exploratory computational modelling analyses explained these demographic moderating effects as arising from positive correlations between demographic factors and individual differences in the strength of Pavlovian-instrumental interactions. These findings serve to sharpen our understanding of the effects of positive and negative affect on instrumental behaviour. (shrink)

There is growing consensus that reward plays an important role in the control of attention. Until recently, reward was thought to influence attention indirectly by modulating task-specific motivation and its effects on voluntary control over selection. Such an account was consistent with the goal-directed (endogenous) versus stimulus-driven (exogenous) framework that had long dominated the field of attention research. Now, a different perspective is emerging. Demonstrations that previously reward-associated stimuli can automatically capture attention even when physically inconspicuous and (...) task-irrelevant challenge previously held assumptions about attentional control. The idea that attentional selection can be value driven, reflecting a distinct and previously unrecognized control mechanism, has gained traction. Since these early demonstrations, the influence of reward learning on attention has rapidly become an area of intense investigation, sparking many new insights. The result is an emerging picture of how the reward system of the brain automatically biases information processing. Here, I review the progress that has been made in this area, synthesizing a wealth of recent evidence to provide an integrated, up-to-date account of value-driven attention and some of its broader implications. (shrink)

People become addicted in different ways, and they respond differently to different interventions. There may nevertheless be a core neural pathology responsible for all distinctively addictive suboptimal behavioral habits. In particular, timing models of reward learning suggest a hypothesis according to which all addiction involves neuroadaptation that attenuates serotonergic inhibition of a mesolimbic dopamine system that has learned that cues for consumption of the addictive target are signals of a high-reward-rate environment.

The issue whether psychopathic offenders are practically rational has attracted philosophical attention. The problem is relevant in theoretical discussions on moral psychology and in those concerning the appropriate social response to the crimes of these individuals. We argue that classical and current experiments concerning the instrumental learning in psychopaths cannot directly support the conclusion that they have impaired instrumental rationality, construed as the ability for transferring the motivation by means-ends reasoning. In fact, we defend the different claim (...) that these experiments appear to show that psychopaths in certain circumstances are not aware of the relevant means for their ends. Moreover, we suggest how further empirical research could help to settle the issue. (shrink)

This study investigates the ability of individuals with psychopathy to perform passive avoidance learning and whether this ability is modulated by level of reinforcement/punishment. Nineteen psychopathic and 21 comparison individuals, as defined by the Hare Psychopathy Checklist Revised (Hare, 1991), were given a passive avoidance task with a graded reinforcement schedule. Response to each rewarding number gained a point reward specific to that number (i.e., 1, 700, 1400 or 2000 points). Response to each punishing number lost a point (...) punishment specific to that number (i.e., the loss of 1, 700, 1400 or 2000 points). In line with predictions, individuals with psychopathy made more passive avoidance errors than the comparison individuals. In addition, while the performance of both groups was modulated by level of reward, only the performance of the comparison population was modulated by level of punishment. The results are interpreted with reference to a computational account of the emotional learning impairment in individuals with psychopathy. (shrink)

This article considers the motivations of teachers to pursue ongoing professional learning. During recent decades, the international policy context has been characterised by high-stakes accountability, but the implications of this agenda for teachers’ motivations toward professional learning remains under-explored. In this mixed methods study, combining a large teacher survey and in-depth teacher interviews, a new and significant concept of ‘instrumental motivation’ is generated to capture how high-stakes performance management policies damage the motivation of teachers to learn professionally. (...) This innovative approach, employing ordinal factor analysis and inductive/deductive hybrid thematic analysis inclusive of self-determination theory, reveals how the ‘instrumental motivation’ of teachers should be constrained and argues for the adoption of alternative motivational strategies to support effective professional learning in schools. (shrink)

The question, “Will science remain human?” expresses a worry that deep learning algorithms will replace scientists in making crucial judgments of classification and inference and that something crucial will be lost if that happens. Ever since the introduction of telescopes and microscopes humans have relied on technologies to “extend” beyond human sensory perception in acquiring scientific knowledge. In this paper I explore whether the ways in which new learning technologies “extend” beyond human cognitive aspects of science can be (...) treated instrumentally. I will consider the norms for determining the reliability of a detection instrument, nuclear magnetic resonance spectroscopy, in predicting models of protein atomic structure. Do the same norms that apply in that case be used to judge the reliability of Artificial Intelligence deep learning algorithms? (shrink)