Reinforcement Learning is a kind of machine learning. We know Profit Sharing, the Rational Policy Making algorithm, the Penalty Avoiding Rational Policy Making algorithm and PS-r* to guarantee the rationality in a typical class of the Partially Observable Markov Decision Processes. However they cannot treat continuous state spaces. In this paper, we present a solution to adapt them in continuous state spaces. We give RPM a mechanism to treat continuous state spaces in (...) the environment that has the same type of a reward. We show the effectiveness of the proposed method in numerical examples. (shrink)

To apply reinforcement learning to difficult classes such as real-environment learning, we need to use a method robust to perceptual aliasing problem. The exploitation-oriented methods such as Profit Sharing can deal with the perceptual aliasing problem to a certain extent. However, when the agent needs to select different actions at the same sensory input, the learning efficiency worsens. To overcome the problem, several state partition methods using history information of state-action pairs are proposed. These methods try to convert (...) a POMDP environment into an MDP environment, and thus they are sometimes very useful. However, their computation cost is very high especially in large state spaces. In contrast, memory-less approaches try to escape from the aliased states by outputting actions stochastically. However, these methods output actions stochastically even in unaliased states, and thus the learning efficiency is bad. If we desire to guarantee the rationality in POMDPs, it is efficient to output actions stochastically only in the aliased states and to output one action deterministically in the other unaliased states. Hence, to discriminate between aliased states and unaliased states, the utilization of χ² -goodness-of-fit test is proposed by Miyazaki et al. They point out that, in aliased states, the distributions of the state transitions by random search and a particular policy are different. This difference doesn't occur owing to non-deterministic actions. Hence, if the agent can collect enough samples to implement the test, the agent can distinguish between aliased states and unaliased states well. However, such a test needs a large amount of data, and it's a problem how the agent collects samples without worsening learning efficiency. If the agent uses random search in the course of learning, the learning efficiency worsens especially in unaliased states. Therefore, in this research, we propose a new method called Extended On-line Profit Sharing with Judgement (EOPSwJ) to detect important incomplete perception, which doesn't need large computation cost and numerous samples. We use two criterions for detecting important incomplete perceptions to attain a task. One is the rate of transitions to each state, and the other is the deterministic rate of actions. We confirm the availability of EOPSwJ using two simulations. (shrink)

Profit Sharing is one of the reinforcement learning methods. An agent, as a learner, selects an action with a state-action value and receives rewards when it reaches a goal state. Then it distributes receiving rewards to state-action values. This paper discusses how to set the initial value of a state-action value. A distribution function ƒ( x ) is called as the reinforcement function. On Profit Sharing, an agent learns a policy by distributing rewards with the (...) reinforcement function. On Markov Decision Processes (MDPs), the reinforcement function ƒ( x ) = 1/ L x is useful, and on Partially Observable Markov Decision Processes (POMDPs), ƒ( x ) = 1/ L w is useful, where L is the sufficient number of rules at each state, and W is the length of an episode. If episodes are always long, the value of the reinforcement function is little. So the differences of rule values become little, and the agent learns little by using the roulette selection as an action selection. This problem is called as Learning Speed Problem. If the value of the reinforcement function for an action is very higher than its state-action value, an agent will not select other action. There is a problem when its action is not a optimal action. This problem is called as Past Experiences Problem. This paper shows that both Learning Speed Problem and Past Experiences Problem are caused by the bad setting between the initial values of a state-action values and the function values of a reinforcement function. We propose how to set the initial values of a state-action values at each state. The experiment shows that an agent can learn correctly even if the length of episode is large. And shows the effectiveness on both MDPs and POMDPs. Our proposed method focuses on the initialization of state-action values and does not limit reinforcement functions. So it can apply to any reinforcement function. (shrink)

Rapid and precise air operation mission planning is a key technology in unmanned aerial vehicles autonomous combat in battles. In this paper, an end-to-end UAV intelligent mission planning method based on deep reinforcement learning is proposed to solve the shortcomings of the traditional intelligent optimization algorithm, such as relying on simple, static, low-dimensional scenarios, and poor scalability. Specifically, the suppression of enemy air defense mission planning is described as a sequential decision-making problem and formalized as a Markov decision (...) process. Then, the SEAD intelligent planning model based on the proximal policy optimization algorithm is established and a general intelligent planning architecture is proposed. Furthermore, three policy training tricks, i.e., domain randomization, maximizing policy entropy, and underlying network parameter sharing, are introduced to improve the learning performance and generalizability of PPO. Experiments results show that the model in this work is efficient and stable, and can be adapted to the unknown continuous high-dimensional environment. It can be concluded that the UAV intelligent mission planning model based on DRL has powerful intelligent planning performance, and provides a new idea for researching UAV autonomy. (shrink)

The purpose of reinforcement learning is to learn an optimal policy in general. However, in 2-players games such as the othello game, it is important to acquire a penalty avoiding policy. In this paper, we focus on formation of a penalty avoiding policy based on the Penalty Avoiding Rational Policy Making algorithm [Miyazaki 01]. In applying it to large-scale problems, we are confronted with the curse of dimensionality. We introduce several ideas and heuristics (...) to overcome the combinational explosion in large-scale problems. First, we propose an algorithm to save the memory by calculation of state transition. Second, we describe how to restrict exploration by two type knowledge; KIFU database and evaluation funcion. We show that our learning player can always defeat against the well-known othello game program KITTY. (shrink)

In this paper, we consider profit sharing that is one of the reinforcement learning methods. An agent learns a candidate solution of a problem from the reward that is received from the environment if and only if it reaches the destination state. A function that distributes the received reward to each action of the candidate solution is called the reinforcement function. On this learning system, the agent can reinforce the set of selected actions when it gets the (...) reward. And the agent should not reinforce the detour actions. First, we will propose a new constraint equation about reinforcement functions to distribute the reinforcement values on the non-detour actions. If we use the reinforcement function to satisfy the constraint equation, the agent can select the non-detour actions directing to the destination state. Next, it is shown that the reinforcement function can be constant after learning process to suppress the selection of detour actions. Lastly, in computer simulations for maze problems, we show that the learning performance of agents does not depend on the size of environment. (shrink)

Reinforcement learning is a kind of machine learning. It aims to adapt an agent to a given environment with a clue to rewards. In general, the purpose of reinforcement learning system is to acquire an optimum policy that can maximize expected reward per an action. However, it is not always important for any environment. Especially, if we apply reinforcement learning system to engineering, environments, we expect the agent to avoid all penalties. In Markov Decision Processes, a (...) pair of a sensory input and an action is called rule. We call a rule penalty if and only if it has a penalty or it can transit to a penalty state where it does not contribute to get any reward. After suppressing all penalty rules, we aim to make a rational policy whose expected reward per an action is larger than zero. In this paper, we propose a suppressing penalty algorithm that can suppress any penalty and get a reward constantly. By applying the algorithm to the tick-tack-toe, its effectiveness is shown. (shrink)

Recent years have yielded many discussions on how to endow autonomous agents with the ability to make ethical decisions, and the need for explicit ethical reasoning and transparency is a persistent theme in this literature. We present a modular and transparent approach to equip autonomous agents with the ability to comply with ethical prescriptions, while still enacting pre-learned optimal behaviour. Our approach relies on a normative supervisor module, that integrates a theorem prover for defeasible deontic logic within the control (...) loop of a reinforcement learning agent. The supervisor operates as both an event recorder and an on-the-fly compliance checker w.r.t. an external norm base. We successfully evaluated our approach with several tests using variations of the game Pac-Man, subject to a variety of “ethical” constraints. (shrink)

We study a joint pricing and inventory control problem for perishables with positive lead time in a finite horizon periodic-review system. Unlike most studies considering a continuous density function of demand, in our paper the customer demand depends on the price of current period and arrives according to a homogeneous Poisson process. We consider both backlogging and lost-sales cases, and our goal is to find a simultaneously ordering and pricing policy to maximize the expected discounted profit over the (...) planning horizon. When there is no fixed ordering cost involved, we design a deep reinforcement learning algorithm to obtain a near-optimal ordering policy and show that there are some monotonicity properties in the learned policy. We also show that our deep reinforcement learning algorithm achieves a better performance than tabular-based Q-learning algorithms. When a fixed ordering cost is involved, we show that our deep reinforcement learning algorithm is effective and efficient, under which the problem of “curse of dimension” is circumvented. (shrink)

Reinforcement Learning is a promising technique for creating agents that can be applied to real world problems. The most important features of RL are trial-and-error search and delayed reward. Thus, agents randomly act in the early learning stage. However, such random actions are impractical for real world problems. This paper presents a novel model of RL agents. A feature of our learning agent model is to integrate the Analytic Hierarchy Process into the standard RL agent model, which consists of (...) three modules: state recognition, learning, and action selecting modules. In our model, the AHP module is designed with {\\it primary knowledge} that humans intrinsically have in a process until a goal state is attained. This integration aims at increasing promising actions instead of completely random actions in the standard RL algorithms. Profit Sharing is adopted as a RL method for our model, since PS is known to be useful even in multi-agent environments. To evaluate our approach in a multi-agent environment, we test a PS RL method with our agent model on a pursuit problem in a grid world. Computational results show that our approach outperforms the standard PS in terms of learning speed in the earlier stages of learning. We also show that the learning performance of our approach is superior at least competitive to that of the standard one in the final stages of learning. (shrink)

In the existing Reinforcement Learning, it is difficult and time consuming to find appropriate the meta-parameters such as learning rate, eligibility traces and temperature for exploration, in particular on a complicated and large-scale problem, the delayed reward often occurs and causes a difficulty in solving the problem. In this paper, we propose a novel method introducing a temperature distribution for reinforcement learning. In addition to the acquirement of policy based on profit sharing, the temperature is given (...) to each state and is trained by hill-climbing method using likelihood function based on success and failure of the task. The proposed method can reduce the parameter setting according to the given problems. We showed the performance on the grid world problem and the control of Acrobot. (shrink)

Recent influential accounts of temporal representation—the use of mental representations with explicit temporal contents, such as before and after relations and durations—sharply distinguish representation from mere sensitivity. A common, important picture of inter-temporal rationality is that it consists in maximizing total expected discounted utility across time. By analyzing reinforcement learning algorithms, this article shows that, given such notions of temporal representation and inter-temporal rationality, it would be possible for an agent to achieve inter-temporal rationality without temporal representation. It then (...) explores potential upshots of this result for theorizing about rationality and representation. (shrink)

Although “rationing” continues to be a dirty word for the public in health policy discourse, Nir Eyal and colleagues handle the concept exactly right in their article in this issue of the Hastings Center Report. They correctly characterize rationing as an ethical requirement, not a moral abomination. They identify the key health policy question as how rationing can best be done, not whether it should be done at all. They make a cogent defense of what they call “rationing (...) through inconvenience” as a justifiable allocational technique. And they wisely call for research on the effectiveness and fairness of this approach and other methods of rationing. I fully agree with their approach to rationing and with their argument that the process they provocatively label “rationing through inconvenience” should not be rejected out of hand. But I believe they have underestimated two ways in which the practical impacts of rationing through inconvenience limit its potential usefulness: the asymmetry of its effect on patients and physicians and the way in which it reduces the capacity of health systems to learn from experience. (shrink)

A deep reinforcement learning-based computational guidance method is presented, which is used to identify and resolve the problem of collision avoidance for a variable number of fixed-wing UAVs in limited airspace. The cooperative guidance process is first analyzed for multiple aircraft by formulating flight scenarios using multiagent Markov game theory and solving it by machine learning algorithm. Furthermore, a self-learning framework is established by using the actor-critic model, which is proposed to train collision avoidance decision-making neural networks. To (...) achieve higher scalability, the neural network is customized to incorporate long short-term memory networks, and a coordination strategy is given. Additionally, a simulator suitable for multiagent high-density route scene is designed for validation, in which all UAVs run the proposed algorithm onboard. Simulated experiment results from several case studies show that the real-time guidance algorithm can reduce the collision probability of multiple UAVs in flight effectively even with a large number of aircraft. (shrink)

Purpose The online economy has not resolved the issue of racial bias in its applications. While algorithms are procedures that facilitate automated decision-making, or a sequence of unambiguous instructions, bias is a byproduct of these computations, bringing harm to historically disadvantaged populations. This paper argues that algorithmic biases explicitly and implicitly harm racial groups and lead to forms of discrimination. Relying upon sociological and technical research, the paper offers commentary on the need for more workplace diversity within high-tech industries (...) and public policies that can detect or reduce the likelihood of racial bias in algorithmic design and execution. Design/methodology/approach The paper shares examples in the US where algorithmic biases have been reported and the strategies for explaining and addressing them. Findings The findings of the paper suggest that explicit racial bias in algorithms can be mitigated by existing laws, including those governing housing, employment, and the extension of credit. Implicit, or unconscious, biases are harder to redress without more diverse workplaces and public policies that have an approach to bias detection and mitigation. Research limitations/implications The major implication of this research is that further research needs to be done. Increasing the scholarly research in this area will be a major contribution in understanding how emerging technologies are creating disparate and unfair treatment for certain populations. Practical implications The practical implications of the work point to areas within industries and the government that can tackle the question of algorithmic bias, fairness and accountability, especially African-Americans. Social implications The social implications are that emerging technologies are not devoid of societal influences that constantly define positions of power, values, and norms. Originality/value The paper joins a scarcity of existing research, especially in the area that intersects race and algorithmic development. (shrink)

In Moderating the Debate: Rationality and the Promise of American Education, Michael Feuer raises concerns about the consequences of basing educational policy on the model of rational choice drawn from economics. Policy making would be better and more realistic, he suggests, if it were based on a newer procedural model drawn from cognitive science. In this essay Eric Bredo builds on Feuer's analysis by offering a more systematic critique of the traditional model of rationality that Feuer (...) criticizes, a more critical evaluation of the procedural model that he favors, and a recommendation that the situational model he does not consider may have some benefits over both. This analysis shows that the traditional model presupposes an actor that cannot learn or develop. While the actor in the procedural model can learn, Bredo contends that it cannot develop, that is, it cannot outgrow its initial assumptions and values. Only the situational model allows for learning and development, important in a model to be used in the field of education. Bredo also considers in his analysis the social‐relational assumptions built into the traditional, procedural, and situational models and the likely ethical consequences of acting on them. (shrink)

In reinforcement learning, a decision maker searching for the most rewarding option is often faced with the question: What is the value of an option that has never been tried before? One way to frame this question is as an inductive problem: How can I generalize my previous experience with one set of options to a novel option? We show how hierarchical Bayesian inference can be used to solve this problem, and we describe an equivalence between the Bayesian model (...) and temporal difference learning algorithms that have been proposed as models of RL in humans and animals. According to our view, the search for the best option is guided by abstract knowledge about the relationships between different options in an environment, resulting in greater search efficiency compared to traditional RL algorithms previously applied to human cognition. In two behavioral experiments, we test several predictions of our model, providing evidence that humans learn and exploit structured inductive knowledge to make predictions about novel options. In light of this model, we suggest a new interpretation of dopaminergic responses to novelty. (shrink)

ABSTRACT The teaching-learning process is of central importance in education. By developing a concept of effective teaching and a corresponding model of the teaching-learning process, it is argued that unless the needs of pupils are to be disregarded, teachers must become co-learners and responsibility for quality of education must be shared. Education is seen as an interactive process in which teachers and pupils participate co-operatively. It is shown that this concept, already implicit in much educational thought and practice, can contribute (...) to more flexible and rational policy-making. (shrink)

We argue that cognitive models can provide a common ground between human users and deep reinforcement learning (Deep RL) algorithms for purposes of explainable artificial intelligence (AI). Casting both the human and learner as cognitive models provides common mechanisms to compare and understand their underlying decision-making processes. This common grounding allows us to identify divergences and explain the learner's behavior in human understandable terms. We present novel salience techniques that highlight the most relevant features in each model's decision- (...) class='Hi'>making, as well as examples of this technique in common training environments such as Starcraft II and an OpenAI gridworld. (shrink)

Theories of decision-making and its neural substrates have long assumed the existence of two distinct and competing valuation systems, variously described as goal-directed vs. habitual, or, more recently and based on statistical arguments, as model-free vs. model-based reinforcement-learning. Though both have been shown to control choices, the cognitive abilities associated with these systems are under ongoing investigation. Here we examine the link to cognitive abilities, and find that individual differences in processing speed covary with a shift from model-free (...) to model-based choice control in the presence of above-average working memory function. This suggests shared cognitive and neural processes; provides a bridge between literatures on intelligence and valuation; and may guide the development of process models of different valuation components. Furthermore, it provides a rationale for individual differences in the tendency to deploy valuation systems, which may be important for understanding the manifold neuropsychiatric diseases associated with malfunctions of valuation. (shrink)

With the advent of the era of artificial intelligence, deep reinforcement learning has achieved unprecedented success in high-dimensional and large-scale artificial intelligence tasks. However, the insecurity and instability of the DRL algorithm have an important impact on its performance. The Soft Actor-Critic algorithm uses advanced functions to update the policy and value network to alleviate some of these problems. However, SAC still has some problems. In order to reduce the error caused by the overestimation of SAC, we propose (...) a new SAC algorithm called Averaged-SAC. By averaging the previously learned action-state estimates, it reduces the overestimation problem of soft Q-learning, thereby contributing to a more stable training process and improving performance. We evaluate the performance of Averaged-SAC through some games in the MuJoCo environment. The experimental results show that the Averaged-SAC algorithm effectively improves the performance of the SAC algorithm and the stability of the training process. (shrink)

The fundamental cognitive problem for active organisms is to decide what to do next in a changing environment. In this article, we analyze motor and action control in computational models that utilize reinforcement learning (RL) algorithms. In reinforcement learning, action control is governed by an action selection policy that maximizes the expected future reward in light of a predictive world model. In this paper we argue that RL provides a way to explicate the so-called action-oriented views of (...) cognitive systems in representational terms. (shrink)

Stanovich & West's target article undervalues the power of implicit learning (particularly reinforcement learning). Implicit learning may allow the learning of more rational responses–and sometimes even generalisation of knowledge–in contexts where explicit, abstract knowledge proves only of limited value, such as for economic decision-making. Four other comments are made.

The subject matter of this thesis can be summarized by a triplet of questions and answers. Showing what these questions and answers mean is, in essence, the goal of my project. The triplet goes like this: Q: How can we make progress in our understanding of social norms and norm compliance? A: Adopting a neurocomputational framework is one effective way to make progress in our understanding of social norms and norm compliance. Q: What could the neurocomputational mechanism of social norm (...) compliance be? A: The mechanism of norm compliance probably consists of Bayesian - Reinforcement Learning algorithms implemented by activity in certain neural populations. Q: What could information about this mechanism tell us about social norms and social norm compliance? A: Information about this mechanism tells us that: a1: Social norms are uncertainty-minimizing devices. a2: Social norm compliance is one trick that agents employ to interact coadaptively and smoothly in their social environment. Most of the existing treatments of norms and norm compliance consist in what Cristina Bicchieri refers to as “rational reconstructions.” A rational reconstruction of the concept of social norm “specifies in which sense one may say that norms are rational, or compliance with a norm is rational”. What sets my project apart from these types of treatments is that it aims, first and foremost, at providing a description of some core aspects of the mechanism of norm compliance. The single most original idea put forth in my project is to bring an alternative explanatory framework to bear on social norm compliance. This is the framework of computational cognitive neuroscience. The chapters of this thesis describe some ways in which central issues concerning social norms can be fruitfully addressed within a neurocomputational framework. In order to qualify and articulate the triplet above, my strategy consists firstly in laying down the beginnings of a model of the mechanism of norm compliance behaviour, and then zooming in on specific aspects of the model. Such a model, the chapters of this thesis argue, explains apparently important features of the psychology and neuroscience of norm compliance, and helps us to understand the nature of the social norms we live by. (shrink)

In recent years there has been an explosion of interest in Artificial Intelligence (AI) both in health care and academic philosophy. This has been due mainly to the rise of effective machine learning and deep learning algorithms, together with increases in data collection and processing power, which have made rapid progress in many areas. However, use of this technology has brought with it philosophical issues and practical problems, in particular, epistemic and ethical. In this paper the authors, with backgrounds in (...) philosophy, maternity care practice and clinical research, draw upon and extend a recent framework for shared decision-making (SDM) that identified a duty of care to the client's knowledge as a necessary condition for SDM. This duty entails the responsibility to acknowledge and overcome epistemic defeaters. This framework is applied to the use of AI in maternity care, in particular, the use of machine learning and deep learning technology to attempt to enhance electronic fetal monitoring (EFM). In doing so, various sub-kinds of epistemic defeater, namely, transparent, opaque, underdetermined, and inherited defeaters are taxonomized and discussed. The authors argue that, although effective current or future AI-enhanced EFM may impose an epistemic obligation on the part of clinicians to rely on such systems' predictions or diagnoses as input to SDM, such obligations may be overridden by inherited defeaters, caused by a form of algorithmic bias. The existence of inherited defeaters implies that the duty of care to the client's knowledge extends to any situation in which a clinician (or anyone else) is involved in producing training data for a system that will be used in SDM. Any future AI must be capable of assessing women individually, taking into account a wide range of factors including women's preferences, to provide a holistic range of evidence for clinical decision-making. (shrink)

This paper uses an improved deep learning algorithm to judge the rationality of the design of landscape image feature recognition. The preprocessing of the image is proposed to enhance the data. The deficiencies in landscape feature extraction are further addressed based on the new model. Then, the two-stage training method of the model is used to solve the problems of long training time and convergence difficulties in deep learning. Innovative methods for zoning and segmentation training of landscape pattern features are (...) proposed, which makes model training faster and generates more creative landscape patterns. Because of the impact of too many types of landscape elements in landscape images, traditional convolutional neural networks can no longer effectively solve this problem. On this basis, a fully convolutional neural network model is designed to perform semantic segmentation of landscape elements in landscape images. Through the method of deconvolution, the pixel-level semantic segmentation is realized. Compared with the 65% accuracy rate of the convolutional neural network, the fully convolutional neural network has an accuracy rate of 90.3% for the recognition of landscape elements. The method is effective, accurate, and intelligent for the classification of landscape element design, which better improves the accuracy of classification, greatly reduces the cost of landscape element design classification, and ensures that the technical method is feasible. This paper classifies landscape behavior based on this model for full convolutional neural network landscape images and demonstrates the effectiveness of using the model. In terms of landscape image processing, the image evaluation provides a certain basis. (shrink)

Oxford philosopher Nick Bostrom; in his recent and celebrated book Superintelligence; argues that advanced AI poses a potentially major existential risk to humanity; and that advanced AI development should be heavily regulated and perhaps even restricted to a small set of government-approved researchers. Bostrom’s ideas and arguments are reviewed and explored in detail; and compared with the thinking of three other current thinkers on the nature and implications of AI: Eliezer Yudkowsky of the Machine Intelligence Research Institute ; and David (...) Weinbaum and Viktoras Veitas of the Global Brain Institute. Relevant portions of Yudkowsky’s book Rationality: From AI to Zombies are briefly reviewed; and it is found that nearly all the core ideas of Bostrom’s work appeared previously or concurrently in Yudkowsky’s thinking. However; Yudkowsky often presents these shared ideas in a more plain-spoken and extreme form; making clearer the essence of what is being claimed. For instance; the elitist strain of thinking that one sees in the background in Bostrom is plainly and openly articulated in Yudkowsky; with many of the same practical conclusions. Bostrom and Yudkowsky view intelligent systems through the lens of reinforcement learning – they view them as “reward-maximizers” and worry about what happens when a very powerful and intelligent reward-maximizer is paired with a goal system that gives rewards for achieving foolish goals like tiling the universe with paperclips. Weinbaum and Veitas’s recent paper “Open-Ended Intelligence” presents a starkly alternative perspective on intelligence; viewing it as centered not on reward maximization; but rather on complex self-organization and self-transcending development that occurs in close coupling with a complex environment that is also ongoingly self-organizing; in only partially knowable ways. It is concluded that Bostrom and Yudkowsky’s arguments for existential risk have some logical foundation; but are often presented in an exaggerated way. For instance; formal arguments whose implication is that the “worst case scenarios” for advanced AI development are extremely dire; are often informally discussed as if they demonstrated the likelihood; rather than just the possibility; of highly negative outcomes. And potential dangers of reward-maximizing AI are taken as problems with AI in general; rather than just as problems of the reward-maximization paradigm as an approach to building superintelligence. If one views past; current; and future intelligence as “open-ended;” in the vernacular of Weaver and Veitas; the potential dangers no longer appear to loom so large; and one sees a future that is wide-open; complex and uncertain; just as it has always been. (shrink)

The authors aim at constructing an agent which learns appropriate actions in a Multi-Agent environment with and without social dilemmas. For this aim, the agent must have nonrationality that makes it give up its own profit when it should do that. Since there are many studies on rational learning that brings more and more profit, it is desirable to utilize them for constructing the agent. Therefore, we use a reward-handling manner that makes internal evaluation from the agent's (...) rewards, and then the agent learns actions by a rational learning method with the internal evaluation. If the agent has only a fixed manner, however, it does not act well in the environment with and without dilemmas. Thus, the authors equip the agent with several reward-handling manners and criteria for selecting an effective one for the environmental situation. In the case of humans, what generates the internal evaluation is usually called emotion. Hence, this study also aims at throwing light on emotional activities of humans from a constructive view. In this paper, we divide a Multi-Agent environment into three situations and construct an agent having the reward-handling manners and the criteria. We observe that the agent acts well in all the three Multi-Agent situations composed of homogeneous agents. (shrink)

This thesis argues for the fundamental importance of the opposition between holistic and reductionistic world-views in economics. Both reductionism and holism may nevertheless underpin laissez-faire policy prescriptions. Scrutiny of the nature of the articulation between micro and macro levels in the writings of economists suggests that invisible hand theories play a key role in reconciling reductionist policy prescriptions with a holistic world. An examination of the prisoners' dilemma in game theory and Arrow's impossibility theorem in social choice (...) theory sets the scene. The prisoners' dilemma epitomises the collective irrationality coordination problems lead to. The source of the dilemma is identified as the combination of interdependence in content and independence in form of the decision making process. Arrovian impossibility has been perceived as challenging traditional views of the relationship between micro and macro levels in economics. Conservative arguments against the possibility in principle of a social welfare function are criticised here as depending on an illicit dualism. The thesis then reviews the standpoints of Smith, Hayek and Keynes. For Smith, the social desirability of individual self-seeking activity is ensured by the 'invisible hand' of a god who has moulded us so to behave, that the quantity of happiness in the world is always maximised. Hayek seeks to re-establish the invisible hand in a secular age, replacing the agency of a deity with an evolutionary mechanism. Hayek's evolutionary theory, criticised here as being based on the exploded notion of group selection, cannot underpin the desirability of spontaneous outcomes. I conclude by arguing that Keynes shares the holistic approach of Smith and Hayek, but without their reliance on invisible hand mechanisms. If spontaneous processes cannot be relied upon to generate desirable social outcomes then we have to take responsibility for achieving this ourselves by establishing the appropriate institutional framework to eliminate macroeconomic prisoners' dilemmas. (shrink)

Algorithms are now routinely used in decision-making; they are potent components in decisions that affect the lives of individuals and the activities of public and private institutions. Although use of algorithms has many benefits, a number of problems have been identified with their use in certain domains, most notably in domains where safety and fairness are important. Awareness of these problems has generated public discourse calling for algorithmic accountability. However, the current discourse focuses largely on algorithms and their opacity. (...) I argue that this reflects a narrow and inadequate understanding of accountability. I sketch an account of accountability that takes accountability to be a social practice constituted by actors, forums, shared beliefs and norms, performativity, and sanctions, and aimed at putting constraints on the exercise of power. On this account, algorithmic accountability is not yet constituted; it is in the making. The account brings to light a set of questions that must be addressed to establish it. (shrink)

The difficulty of handling complex problems has spawned challenges to the traditional paradigm of technical rationality in design, planning, and policy making. One of the most frequently proposed solutions is an interdisciplinary approach, though few writers have described the operational dynamics of such an approach. A global model of interdisciplinary problem-solving is presented based on the premise that the unity of the interdisciplinary approach derives from the creation of an intermediary process that relies on common language, shared information, (...) a mutual sense of stakeholding, and the resolution of disciplinary differences. The theoretical underpinning of this approach is the conceptualization of interdisciplinary problem-solving as a communicative process that requires attention to the rhetorical and political dynamics of working with competing interests, practices, and disciplines. The practice portion is a composite picture of effective models, skills, activities, strategies, and techniques employed by actuals interdisciplinary teams. This global model offers a way of both theoretically and practically visualizing Th. K. van Lohuizen's ideal of achieving unity of town planning, an ideal that has profound implications for the organization of both professional practice and training. (shrink)

It is well-established that algorithms can be instruments of injustice. It is less frequently discussed, however, how current modes of AI deployment often make the very discovery of injustice difficult, if not impossible. In this article, we focus on the effects of algorithmic profiling on epistemic agency. We show how algorithmic profiling can give rise to epistemic injustice through the depletion of epistemic resources that are needed to interpret and evaluate certain experiences. By doing so, we not only demonstrate how (...) the philosophical conceptual framework of epistemic injustice can help pinpoint potential, systematic harms from algorithmic profiling, but we also identify a novel source of hermeneutical injustice that to date has received little attention in the relevant literature: epistemic fragmentation. Epistemic fragmentation is a structural characteristic of algorithmically-mediated environments that isolate individuals, making it more difficult to develop, uptake and apply new epistemic resources. This, in turn, can impede the identification and conceptualisation of emerging harms in these environments. We trace the occurrence of hermeneutical injustice back to the fragmentation of the epistemic experiences of individuals, who are left more vulnerable by the inability to share, compare, and learn from shared experiences. (shrink)

Although seemingly irrational choice abounds, the rules governing these mis‐steps that might provide hints about the factors limiting normative behavior are unclear. We consider three experimental tasks, which probe different aspects of non‐normative choice under uncertainty. We argue for systematic statistical, algorithmic, and implementational sources of irrationality, including incomplete evaluation of long‐run future utilities, Pavlovian actions, and habits, together with computational and statistical noise and uncertainty. We suggest structural and functional adaptations that minimize their maladaptive effects.

Liquidity risk arises from the inability to unwind or hedge trading positions at the prevailing market prices. The risk of liquidity is a wide and complex topic as it depends on several factors and causes. While much has been written on the subject, there exists no clear-cut mathematical description of the phenomena and typical market risk modeling methods fail to identify the effect of illiquidity risk. In this paper, we do not propose a definitive one either, but we attempt to (...) derive novel mathematical algorithms for the dynamic modeling of trading volumes during the closeout period from the perspective of multiple-asset portfolio, as well as for financial entities with different subsidiary firms and multiple agents. The robust modeling techniques are based on the application of initial-value-problem differential equations technique for portfolio selection and risk management purposes. This paper provides some crucial parameters for the assessment of the trading volumes of multiple-asset portfolio during the closeout period, where the mathematical proofs for each theorem and corollary are provided. Based on the new developed econophysics theory, this paper presents for the first time a closed-form solution for key parameters for the estimation of trading volumes and liquidity risk, such as the unwinding constant, half-life, and mean lifetime and discusses how these novel parameters can be estimated and incorporated into the proposed techniques. The developed modeling algorithms are appealing in terms of theory and are promising for practical econophysics applications, particularly in developing dynamic and robust portfolio management algorithms in light of the 2007–2009 global financial crunch. In addition, they can be applied to artificial intelligence and machine learning for the policymaking process, reinforcement machine learning techniques for the Internet of Things data analytics, expert systems in finance, FinTech, and within big data ecosystems. (shrink)

This paper aims to contribute to the existing literature on normative cognition by providing a lineage explanation of human social norm psychology. This approach builds upon theories of goal-directed behavioral control in the reinforcement learning and control literature, arguing that this form of control defines an important class of intentional normative mental states that are instrumental in nature. I defend the view that great ape capacities for instrumental reasoning and our capacity (or family of capacities) for shared intentionality coadapted (...) to each other and argue that the evolution of this capacity has allowed the representation of social norms and the emergence of our capacity for normative guidance. (shrink)

Sequential decision theory formally solves the problem of rational agents in uncertain worlds if the true environmental prior probability distribution is known. Solomonoff's theory of universal induction formally solves the problem of sequence prediction for unknown prior distribution. We combine both ideas and get a parameter-free theory of universal Artificial Intelligence. We give strong arguments that the resulting AIXI model is the most intelligent unbiased agent possible. We outline how the AIXI model can formally solve a number of problem (...) classes, including sequence prediction, strategic games, function minimization, reinforcement and supervised learning. The major drawback of the AIXI model is that it is uncomputable. To overcome this problem, we construct a modified algorithm AIXItl that is still effectively more intelligent than any other time t and length l bounded agent. The computation time of AIXItl is of the order t x 2^l. The discussion includes formal definitions of intelligence order relations, the horizon problem and relations of the AIXI theory to other AI approaches. (shrink)

Reinforcement Learning (RL) handles policy search problems: searching a mapping from state space to action space. However RL is based on gradient methods and as such, cannot deal with problems with multimodal landscape. In contrast, though Genetic Algorithm (GA) is promising to deal with them, it seems to be unsuitable for policy search problems from the viewpoint of the cost of evaluation. Minimal Generation Gap (MGG), used as a generation-alternation model in GA, generates many offspring from two (...) or more parents selected from a population. Therefore, evaluating policies of generated offspring requires much trial and error (i.e. interaction between an agent and an environment). In this paper, we incorporate importance sampling into the framework of MGG in order to reduce the cost of evaluation on policy search. The proposed techniques are applied to Markov Decision Process (MDP) with multimodal landscape. The experimental results show that these techniques can reduce the number of interaction between an agent and an environment, and also mean that MGG and importance sampling are good for each other. (shrink)

In a recent paper on Foundations of Physics, Stephen Boughn reinforces a view that is more shared in the area of the foundations of quantum mechanics than it would deserve, a view according to which quantum mechanics does not require nonlocality of any kind and the common interpretation of Bell theorem as a nonlocality result is based on a misunderstanding. In the present paper I argue that this view is based on an incorrect reading of the presuppositions of the (...) EPR argument and the Bell theorem and, as a consequence, is unfounded. (shrink)

Rationalization occurs when a person has performed an action and then concocts the beliefs and desires that would have made it rational. Then, people often adjust their own beliefs and desires to match the concocted ones. While many studies demonstrate rationalization, and a few theories describe its underlying cognitive mechanisms, we have little understanding of its function. Why is the mind designed to construct post hoc rationalizations of its behavior, and then to adopt them? This may accomplish an important (...) task: transferring information between the different kinds of processes and representations that influence our behavior. Human decision making does not rely on a single process; it is influenced by reason, habit, instinct, norms, and so on. Several of these influences are not organized according to rational choice. Rationalization extracts implicit information – true beliefs and useful desires – from the influence of these non-rational systems on behavior. This is a useful fiction – fiction, because it imputes reason to non-rational psychological processes; useful, because it can improve subsequent reasoning. More generally, rationalization belongs to the broader class of representational exchange mechanisms, which transfer information between many different kinds of psychological representations that guide our behavior. Representational exchange enables us to represent any information in the manner best suited to the particular tasks that require it, balancing accuracy, efficiency, and flexibility in thought. The theory of representational exchange reveals connections between rationalization and theory of mind, inverse reinforcement learning, thought experiments, and reflective equilibrium. (shrink)

Two definitions of algorithm, their uses, and their implied models of computing in society, are reviewed. The first, termed the structural programming definition, aligns more with usage in computer science, and as the name suggests, the intellectual project of structured programming. The second, termed the systemic definition, is more informal and emerges from ethnographic observations of discussions of software in both professional and everyday settings. Specific examples of locating algorithms within modern codebases are shared, as well as code directly impacting (...) social and ethical concerns. The structural distinction between algorithms and social concerns is explained as mirroring the engineering construct of algorithms and data structures. It is proposed that, rather than this separation being an attempt to enforce a professional boundary and evade social responsibility, it is a crucial technical distinction within code which makes it clearer and more transparent. The power structures reinforced by the broader, cultural interpretations of algorithm are reconsidered, along with what it would mean for software to have an inclusive design culture. (shrink)

This article tries to present a broad view on the values and ethicalissues that are at stake in efforts to rationalize health policy on thebasis of economic evaluations (like cost-effectiveness analysis) andrandomly controlled clinical trials. Though such a rationalization isgenerally seen as an objective and `value free' process, moral valuesoften play a hidden role, not only in the production of `evidence', butalso in the way this evidence is used in policy making. For example, thedefinition of effectiveness of (...) medical treatment or health care serviceis heavily dependent on dominant individual or social views about thegoals of the particular treatment or service. There is also a concernthat a reliance on EBM in health policy will occur at the expense ofwidely shared social values like equity and solidarity. Moreover, thereis a concern that when economic considerations and rational proceduresbecome more influential, various `outside' groups third parties likeinsurance companies and policy makers will get a stronger influence onmedical practice which may lead to a change in the patient-providerrelationship. The authors conclude that social values and patientpreference should be explicitly addressed when health policy making isbased on economic and other scientific evidence. (shrink)

Reinforcement learning has recently received much attention as a learning method for complicated systems, e.g., robot systems. It does not need prior knowledge and has higher capability of reactive and adaptive behaviors. However increase in dimensionality of the action-state space makes it diffcult to accomplish learning. The applicability of the existing reinforcement learning algorithms are effective for simple tasks with relatively small action-state space. In this paper, we propose a new reinforcement learning algorithm: “Q-learning with Dynamic Structuring (...) of Exploration Space Based on Genetic Algorithm ”. The algorithm is applicable to systems with high dimensional action and interior state spaces, for example a robot with many redundant degrees of freedom. To demonstrate the effectiveness of the proposed algorithm simulations of obstacle avoidance by a 50 links manipulator have been carried out. It is shown that effective behavior can be learned by using the proposed algorithm. (shrink)

The combination of increased availability of large amounts of fine-grained human behavioral data and advances in machine learning is presiding over a growing reliance on algorithms to address complex societal problems. Algorithmic decision-making processes might lead to more objective and thus potentially fairer decisions than those made by humans who may be influenced by greed, prejudice, fatigue, or hunger. However, algorithmic decision-making has been criticized for its potential to enhance discrimination, information and power asymmetry, and opacity. In this (...) paper, we provide an overview of available technical solutions to enhance fairness, accountability, and transparency in algorithmic decision-making. We also highlight the criticality and urgency to engage multi-disciplinary teams of researchers, practitioners, policy-makers, and citizens to co-develop, deploy, and evaluate in the real-world algorithmic decision-making processes designed to maximize fairness and transparency. In doing so, we describe the Open Algortihms project as a step towards realizing the vision of a world where data and algorithms are used as lenses and levers in support of democracy and development. (shrink)

Whereas classic work in judgment and decision making has focused on the deviation of intuition from rationality, more recent research has focused on the performance of intuition in real-world environments. Borrowing from both approaches, we investigate to which extent competing models of intuitive probabilistic decision making overlap with choices according to the axioms of probability theory and how accurate those models can be expected to perform in real-world environments. Specifically, we assessed to which extent heuristics, models implementing weighted (...) additive information integration (WADD), and the parallel constraint satisfaction (PCS) network model approximate the Bayesian solution and how often they lead to correct decisions in a probabilistic decision task. PCS and WADD outperform simple heuristics on both criteria with an approximation of 88.8% and a performance of 73.7%. Results are discussed in the light of selection of intuitive processes by reinforcement learning. (shrink)

How may nations protect their interests in space if its fragility makes military operations there self-defeating? This essay claims nations are in Prisoners Dilemmas on the matter, and applies David Gauthier’s theories about how it is rational to behave morally—cooperatively—in such dilemmas. Currently space-faring nations should i) enter into co-operative space sharing arrangements with other rational nations, ii) exclude—militarily, but with only terrestrial force—nations irrational or existentially opposed to other nations being in space, and iii) incentivize all nations (...) into co-operation by redistributing some space-generated wealth to nations that might otherwise ruin space. Further, since all nations are rationally incentivized to ever greater dependence on space, and since each, by threatening space, can hold hostage its benefits to all, each can demand resolution of terrestrial discontents with other nations. So each has a rational interest in treating others morally both in space and on terrestrial issues. (shrink)

Recent technological and methodological changes in farming have led to an emerging set of claims about the role of digital technology in food production. Known as precision agriculture, the integration of digital management and surveillance technologies in farming is normatively presented as a revolutionary transformation. Proponents contend that machine learning, Big Data, and automation will create more accurate, efficient, transparent, and environmentally friendly food production, staving off both food insecurity and ecological ruin. This article contributes a critique of these rhetorical (...) and discursive claims to a growing body of critical literature on precision agriculture. It argues precision agriculture is less a revolution than an evolution, an effort to shore up and intensify the conventional farming system responsible for generating many of the social and environmental problems precision agriculture is presented as solving. While precision agriculture advocates portray it as a radical, even democratic epistemological break with the past, this paper locates truth claims surrounding datafication and algorithmic control in farming within deeper historical contexts of the capitalist rationalization of production and efforts to quantify and automate physical and mental labor. Abjuring the growing cultural tendency to treat algorithmic systems as revolutionary in favor of social and historical dimensions of precision agriculture, can help re-frame the discussion about its design and use around real, socially and ecologically oriented change in farming, and so ensure that the possibilities and benefits of precision agriculture are as evenly and effectively shared as possible. (shrink)

Di erent from existing reinforcement learning algorithms that generate only reactive policies and existing probabilis tic planning algorithms that requires a substantial amount of a priori knowledge in order to plan we devise a two stage bottom up learning to plan process in which rst reinforce ment learning dynamic programming is applied without the use of a priori domain speci c knowledge to acquire a reactive policy and then explicit plans are extracted from the learned reactive policy (...) Plan extraction is based on a beam search algorithm that performs temporal projection in a restricted fashion guided by the value functions re sulting from reinforcement learning dynamic programming Experiments and theoretical analysis are presented.. (shrink)

This paper presents a human-computer interaction model with a three layers learning mechanism in a pervasive environment. We begin with a discussion around a number of important issues related to human-computer interaction followed by a description of the architecture for a multi-agent cooperative design system for pervasive computing environment. We present our proposed three- layer HCI model and introduce the group formation algorithm, which is predicated on a dynamic sharing niche technology. Finally, we explore the cooperative reinforcement learning and (...) fusion algorithms; the paper closes with concluding observations and a summary of the principal work and contributions of this paper. (shrink)