Reinforcement Learning is a promising technique for creating agents that can be applied to real world problems. The most important features of RL are trial-and-error search and delayed reward. Thus, agents randomly act in the early learning stage. However, such random actions are impractical for real world problems. This paper presents a novel model of RL agents. A feature of our learning agent model is to integrate the Analytic Hierarchy Process into the standard RL agent (...) model, which consists of three modules: state recognition, learning, and action selecting modules. In our model, the AHP module is designed with {\\it primary knowledge} that humans intrinsically have in a process until a goal state is attained. This integration aims at increasing promising actions instead of completely random actions in the standard RL algorithms. Profit Sharing is adopted as a RL method for our model, since PS is known to be useful even in multi-agent environments. To evaluate our approach in a multi-agent environment, we test a PS RL method with our agent model on a pursuit problem in a grid world. Computational results show that our approach outperforms the standard PS in terms of learning speed in the earlier stages of learning. We also show that the learning performance of our approach is superior at least competitive to that of the standard one in the final stages of learning. (shrink)

In this paper, we consider profit sharing that is one of the reinforcement learning methods. An agent learns a candidate solution of a problem from the reward that is received from the environment if and only if it reaches the destination state. A function that distributes the received reward to each action of the candidate solution is called the reinforcement function. On this learning system, the agent can reinforce the set of selected actions when (...) it gets the reward. And the agent should not reinforce the detour actions. First, we will propose a new constraint equation about reinforcement functions to distribute the reinforcement values on the non-detour actions. If we use the reinforcement function to satisfy the constraint equation, the agent can select the non-detour actions directing to the destination state. Next, it is shown that the reinforcement function can be constant after learning process to suppress the selection of detour actions. Lastly, in computer simulations for maze problems, we show that the learning performance of agents does not depend on the size of environment. (shrink)

Profit Sharing is one of the reinforcement learning methods. An agent, as a learner, selects an action with a state-action value and receives rewards when it reaches a goal state. Then it distributes receiving rewards to state-action values. This paper discusses how to set the initial value of a state-action value. A distribution function ƒ( x ) is called as the reinforcement function. On Profit Sharing, an agent learns a policy by distributing rewards with (...) the reinforcement function. On Markov Decision Processes (MDPs), the reinforcement function ƒ( x ) = 1/ L x is useful, and on Partially Observable Markov Decision Processes (POMDPs), ƒ( x ) = 1/ L w is useful, where L is the sufficient number of rules at each state, and W is the length of an episode. If episodes are always long, the value of the reinforcement function is little. So the differences of rule values become little, and the agent learns little by using the roulette selection as an action selection. This problem is called as Learning Speed Problem. If the value of the reinforcement function for an action is very higher than its state-action value, an agent will not select other action. There is a problem when its action is not a optimal action. This problem is called as Past Experiences Problem. This paper shows that both Learning Speed Problem and Past Experiences Problem are caused by the bad setting between the initial values of a state-action values and the function values of a reinforcement function. We propose how to set the initial values of a state-action values at each state. The experiment shows that an agent can learn correctly even if the length of episode is large. And shows the effectiveness on both MDPs and POMDPs. Our proposed method focuses on the initialization of state-action values and does not limit reinforcement functions. So it can apply to any reinforcement function. (shrink)

AI research is being challenged with ensuring that autonomous agents learn to behave ethically, namely in alignment with moral values. Here, we propose a novel way of tackling the value alignment problem as a two-step process. The first step consists on formalising moral values and value aligned behaviour based on philosophical foundations. Our formalisation is compatible with the framework of (Multi-Objective) Reinforcement Learning, to ease the handling of an agent’s individual and ethical objectives. The second step (...) consists in designing an environment wherein an agent learns to behave ethically while pursuing its individual objective. We leverage on our theoretical results to introduce an algorithm that automates our two-step approach. In the cases where value-aligned behaviour is possible, our algorithm produces a learning environment for the agent wherein it will learn a value-aligned behaviour. (shrink)

To apply reinforcement learning to difficult classes such as real-environment learning, we need to use a method robust to perceptual aliasing problem. The exploitation-oriented methods such as Profit Sharing can deal with the perceptual aliasing problem to a certain extent. However, when the agent needs to select different actions at the same sensory input, the learning efficiency worsens. To overcome the problem, several state partition methods using history information of state-action pairs are proposed. These (...) methods try to convert a POMDP environment into an MDP environment, and thus they are sometimes very useful. However, their computation cost is very high especially in large state spaces. In contrast, memory-less approaches try to escape from the aliased states by outputting actions stochastically. However, these methods output actions stochastically even in unaliased states, and thus the learning efficiency is bad. If we desire to guarantee the rationality in POMDPs, it is efficient to output actions stochastically only in the aliased states and to output one action deterministically in the other unaliased states. Hence, to discriminate between aliased states and unaliased states, the utilization of χ² -goodness-of-fit test is proposed by Miyazaki et al. They point out that, in aliased states, the distributions of the state transitions by random search and a particular policy are different. This difference doesn't occur owing to non-deterministic actions. Hence, if the agent can collect enough samples to implement the test, the agent can distinguish between aliased states and unaliased states well. However, such a test needs a large amount of data, and it's a problem how the agent collects samples without worsening learning efficiency. If the agent uses random search in the course of learning, the learning efficiency worsens especially in unaliased states. Therefore, in this research, we propose a new method called Extended On-line Profit Sharing with Judgement (EOPSwJ) to detect important incomplete perception, which doesn't need large computation cost and numerous samples. We use two criterions for detecting important incomplete perceptions to attain a task. One is the rate of transitions to each state, and the other is the deterministic rate of actions. We confirm the availability of EOPSwJ using two simulations. (shrink)

Collaboration requires agents to coordinate their behavior on the fly, sometimes cooperating to solve a single task together and other times dividing it up into sub‐tasks to work on in parallel. Underlying the human ability to collaborate is theory‐of‐mind (ToM), the ability to infer the hidden mental states that drive others to act. Here, we develop Bayesian Delegation, a decentralized multi‐agent learning mechanism with these abilities. Bayesian Delegation enables agents to rapidly infer the hidden intentions of others by (...) inverse planning. We test Bayesian Delegation in a suite of multi‐agent Markov decision processes inspired by cooking problems. On these tasks, agents with Bayesian Delegation coordinate both their high‐level plans (e.g., what sub‐task they should work on) and their low‐level actions (e.g., avoiding getting in each other's way). When matched with partners that act using the same algorithm, Bayesian Delegation outperforms alternatives. Bayesian Delegation is also a capable ad hoc collaborator and successfully coordinates with other agent types even in the absence of prior experience. Finally, in a behavioral experiment, we show that Bayesian Delegation makes inferences similar to human observers about the intent of others. Together, these results argue for the centrality of ToM for successful decentralized multi‐agent collaboration. (shrink)

The paper presents an approach for developing multi-agent reinforcement learning systems that are made up of a coalition of modular agents. We focus on learning to segment sequences (sequential decision tasks) to create modular structures, through a bidding process that is based on reinforcements received during task execution. The approach segments sequences (and divides them up among agents) to facilitate the learning of the overall task. Notably, our approach does not rely on a priori knowledge or a (...) priori structures. Initial experiments demonstrated the basic promise of the approach. This work shows how bidding and reinforcement learning can be usefully combined, thus pointing to a new research direction. (shrink)

The Internet of Things (IoT) is a fast-growing technology that connects everyday devices to the Internet, enabling wireless, low-consumption and low-cost communication and data exchange. IoT has revolutionized the way devices interact with each other and the internet. The more devices become connected, the greater the risk of security breaches. There is currently a need for new approaches to algorithms that can detect malware regardless of the size of the network and that can adapt to dynamic changes in the network. (...) Through the use of a multi-agent reinforcement learning algorithm, this paper proposes a novel algorithm for malware detection in IoT devices. The proposed algorithm is not strongly dependent on the size of the IoT network due to the that its training is adapted using time differences if the IoT network size is small or Monte Carlo otherwise. To validate the proposed algorithm in an environment as close to reality as possible, we proposed a scenario based on a real IoT network, where we tested different malware propagation models. Different simulations varying the number of agents and nodes in the IoT network have been developed. The result of these simulations proves the efficiency and adaptability of the proposed algorithm in detecting malware, regardless of the malware propagation model. (shrink)

This paper addresses weighting and partitioning in complex reinforcement learning tasks, with the aim of facilitating learning. The paper presents some ideas regarding weighting of multiple agents and extends them into partitioning an input/state space into multiple regions with di erential weighting in these regions, to exploit di erential characteristics of regions and di erential characteristics of agents to reduce the learning complexity of agents (and their function approximators) and thus to facilitate the learning overall. It analyzes, in (...) class='Hi'>reinforcement learning tasks, di erent ways of partitioning a task and using agents selectively based on partitioning. Based on the analysis, some heuristic methods are described and experimentally tested. We nd that some o -line heuristic methods performed the best, signi cantly better than single-agent models. (shrink)

Liquidity risk arises from the inability to unwind or hedge trading positions at the prevailing market prices. The risk of liquidity is a wide and complex topic as it depends on several factors and causes. While much has been written on the subject, there exists no clear-cut mathematical description of the phenomena and typical market risk modeling methods fail to identify the effect of illiquidity risk. In this paper, we do not propose a definitive one either, but we attempt to (...) derive novel mathematical algorithms for the dynamic modeling of trading volumes during the closeout period from the perspective of multiple-asset portfolio, as well as for financial entities with different subsidiary firms and multiple agents. The robust modeling techniques are based on the application of initial-value-problem differential equations technique for portfolio selection and risk management purposes. This paper provides some crucial parameters for the assessment of the trading volumes of multiple-asset portfolio during the closeout period, where the mathematical proofs for each theorem and corollary are provided. Based on the new developed econophysics theory, this paper presents for the first time a closed-form solution for key parameters for the estimation of trading volumes and liquidity risk, such as the unwinding constant, half-life, and mean lifetime and discusses how these novel parameters can be estimated and incorporated into the proposed techniques. The developed modeling algorithms are appealing in terms of theory and are promising for practical econophysics applications, particularly in developing dynamic and robust portfolio management algorithms in light of the 2007–2009 global financial crunch. In addition, they can be applied to artificial intelligence and machine learning for the policymaking process, reinforcement machine learning techniques for the Internet of Things data analytics, expert systems in finance, FinTech, and within big data ecosystems. (shrink)

Massively multiplayer online games are not merely electronic communication systems based on computational databases, but also include artificial intelligence that possesses complex, dynamic structure. Each visible action taken by a component of the multi-agent system appears simple, but is supported by vastly more sophisticated invisible processes. A rough outline of the typical hierarchy has four levels: interaction between two individuals, each either human or artificial, conflict between teams of agents who cooperate with fellow team members, enduring social-cultural (...) groups that seek to accomplish shared goals, and large-scale cultural traditions, often separated into virtual geographic regions. In many MMOs, both magic and religion are represented, in ways that harmonize with a social-scientific theory that defines them in terms of specific versus general psychological compensators. This article draws empirical examples from five diverse MMOs: Dark Age of Camelot, Dungeons and Dragons Online, World of Warcraft, A Tale in the Desert and Gods and Heroes. (shrink)

This paper addresses automatic partitioning in complex reinforcement learning tasks with multiple agents, without a priori domain knowledge regarding task structures. Partitioning a state/input space into multiple regions helps to exploit the di erential characteristics of regions and di erential characteristics of agents, thus facilitating learning and reducing the complexity of agents especially when function approximators are used. We develop a method for optimizing the partitioning of the space through experience without the use of a priori domain knowledge. The (...) method is experimentally tested and compared to a number of other algorithms. As expected, we found that the multi-agent method with automatic partitioning outperformed single-agent learning. (shrink)

The area of smart power grids needs to constantly improve its efficiency and resilience, to provide high quality electrical power in a resilient grid, while managing faults and avoiding failures. Achieving this requires high component reliability, adequate maintenance, and a studied failure occurrence. Correct system operation involves those activities and novel methodologies to detect, classify, and isolate faults and failures and model and simulate processes with predictive algorithms and analytics. In this paper, we showcase the application of a complex-adaptive, self-organizing (...) modeling method, and Probabilistic Boolean Networks, as a way towards the understanding of the dynamics of smart grid devices, and to model and characterize their behavior. This work demonstrates that PBNs are equivalent to the standard Reinforcement Learning Cycle, in which the agent/model has an interaction with its environment and receives feedback from it in the form of a reward signal. Different reward structures were created to characterize preferred behavior. This information can be used to guide the PBN to avoid fault conditions and failures. (shrink)

Can humans get arbitrarily capable reinforcement learning agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? This question impacts how far RL can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we study when an RL agent has an instrumental goal to tamper with its reward process, and describe design principles that (...) prevent instrumental goals for two different types of reward tampering. Combined, the design principles can prevent reward tampering from being an instrumental goal. The analysis benefits from causal influence diagrams to provide intuitive yet precise formalizations. (shrink)

Rapid and precise air operation mission planning is a key technology in unmanned aerial vehicles autonomous combat in battles. In this paper, an end-to-end UAV intelligent mission planning method based on deep reinforcement learning is proposed to solve the shortcomings of the traditional intelligent optimization algorithm, such as relying on simple, static, low-dimensional scenarios, and poor scalability. Specifically, the suppression of enemy air defense mission planning is described as a sequential decision-making problem and formalized as a Markov decision (...) process. Then, the SEAD intelligent planning model based on the proximal policy optimization algorithm is established and a general intelligent planning architecture is proposed. Furthermore, three policy training tricks, i.e., domain randomization, maximizing policy entropy, and underlying network parameter sharing, are introduced to improve the learning performance and generalizability of PPO. Experiments results show that the model in this work is efficient and stable, and can be adapted to the unknown continuous high-dimensional environment. It can be concluded that the UAV intelligent mission planning model based on DRL has powerful intelligent planning performance, and provides a new idea for researching UAV autonomy. (shrink)

We study a joint pricing and inventory control problem for perishables with positive lead time in a finite horizon periodic-review system. Unlike most studies considering a continuous density function of demand, in our paper the customer demand depends on the price of current period and arrives according to a homogeneous Poisson process. We consider both backlogging and lost-sales cases, and our goal is to find a simultaneously ordering and pricing policy to maximize the expected discounted profit over the (...) planning horizon. When there is no fixed ordering cost involved, we design a deep reinforcement learning algorithm to obtain a near-optimal ordering policy and show that there are some monotonicity properties in the learned policy. We also show that our deep reinforcement learning algorithm achieves a better performance than tabular-based Q-learning algorithms. When a fixed ordering cost is involved, we show that our deep reinforcement learning algorithm is effective and efficient, under which the problem of “curse of dimension” is circumvented. (shrink)

I present a game-theoretical multi-agent system to simulate the evolutionary process responsible for the pragmatic phenomenon division of pragmatic labour (DOPL), a linguistic convention emerging from evolutionary forces. Each agent is positioned on a toroid lattice and communicates via signaling games , where the choice of an interlocutor depends on the Manhattan distance between them. In this framework I compare two learning dynamics: reinforcement learning (RL) and belief learning (BL). An agent’s experiences from previous plays (...) influence his communication behaviour, and RL agents act in a non-rational way whereas BL agents display a small degree of rationality by using best response dynamics . The complete system simulates an evolutionary process of communication strategies, which agents can learn in a structured spatial society. The significant questions are: what circumstances could lead to an evolutionary process that doesn’t result in the expected DOPL convention; and to what extent is interlocutor rationality necessary for the emergence of a society-wide convention à la DOPL? (shrink)

Recent studies in pedestrian simulation have been able to construct a highly realistic navigation behaviour in many circumstances. However, when replicating the close interactions between pedestrians, the replicated behaviour is often unnatural and lacks human likeness. One of the possible reasons is that the current models often ignore the cognitive factors in the human thinking process. Another reason is that many models try to approach the problem by optimising certain objectives. On the other hand, in real life, humans do (...) not always take the most optimised decisions, particularly when interacting with other people. To improve the navigation behaviour in this circumstance, we proposed a pedestrian interacting model using reinforcement learning. Additionally, a novel cognitive prediction model, inspired by the predictive system of human cognition, is also incorporated. This helps the pedestrian agent in our model to learn to interact and predict the movement in a similar practice as humans. In our experimental results, when compared to other models, the path taken by our model’s agent is not the most optimised in certain aspects like path lengths, time taken and collisions. However, our model is able to demonstrate a more natural and human-like navigation behaviour, particularly in complex interaction settings. (shrink)

This theoretical paper takes an agency-theoretic approach to questions of corporate social responsibility (CSR). A comparison of various extant frameworks focusses on how CSR agency emerges in complex multi-agent and multi-sector stakeholder networks. The discussion considers the respective capabilities and relevance of these frameworks – culminating in an integrative CSR practice model. A short literature review of the evolution of CSR since the 1950’s provides the backdrop for understanding multi-agent cross-sectoral stakeholder coalitions as a strategic (...) determinant of today’s organizational behavior. The paper turns to Werhane’s coupling of moral imagination and systems thinking and forging stakeholder coalitions in problem contexts that were traditionally deemed intractable by for-profit organizations. However, it identifies the problem that the systems approach treats macro-agents (organizations, stakeholders) as given (“black-boxed”) and shies away from more radically re-imagining the possibilities of reassembling agency from the bottom up. Actor Network Theory (ANT) provides such a method, which strictly commits to treating organizational behavior as a product of technological, human and environmental micro-processes. ANT, however, is lacking a genuine moral deliberative stance in designing complex CSR coalitions. In an attempt to capitalize on the respective strengths of these frameworks (Systems thinking and ANT) the paper tends to a recent iterative series of “situational transactive” models that are rooted in the US pragmatist tradition and seek to capture intelligent planning processes in complex problematic contexts. The contribution proposes a new CSR practice model, which assigns specific roles to the theoretical contributions of ANT, system thinking and pragmatism in complex deliberation processes. This model can be industry-tested in a future study. (shrink)

The purpose of this work is to elaborate an empirically grounded mathematical model of the magnitude of consequences component of “moral intensity” (Jones, Academy of Management Review 16 (2),366, 1991) that can be used to evaluate different ethical situations. The model is built using the analytical hierarchy process (AHP) (Saaty, The Analytic Hierarchy Process , 1980) and empirical data from the legal profession. One contribution of our work is that it illustrates how AHP can be applied in (...) the field of ethics. Following a review of the literature, we discuss the development of the model. We then illustrate how the model can be used to rank-order three well-known ethical reasoning cases in terms of the magnitude of consequences. The work concludes with implications for theory, practice, and future research. Specifically we discuss how this work extends the previous work by Collins ( Journal of Business Ethics 8 , 1, 1989) regarding the nature of harm variable. We also discuss the contribution this work makes in the development of ethical scenarios used to test hypotheses in the field of business ethics. Finally, we discuss how the model can be used for after-action review, contribute to organizational learning, train employees in ethical reasoning, and aid in the design and development of decision support systems that support ethical reasoning. (shrink)

The purpose of this work is to elaborate an empirically grounded mathematical model of the magnitude of consequences component of "moral intensity", 366, 1991) that can be used to evaluate different ethical situations. The model is built using the analytical hierarchy process and empirical data from the legal profession. One contribution of our work is that it illustrates how AHP can be applied in the field of ethics. Following a review of the literature, we discuss the development of the (...) model. We then illustrate how the model can be used to rank-order three well-known ethical reasoning cases in terms of the magnitude of consequences. The work concludes with implications for theory, practice, and future research. Specifically we discuss how this work extends the previous work by Collins regarding the nature harm variable. We also discuss the contribution this work makes in the development of ethical scenarios used to test hypotheses in the field of business ethics. Finally, we discuss how the model can be used for after-action review, contribute to organizational learning, train employees in ethical reasoning, and aid in the design and development of decision support systems that support ethical reasoning. (shrink)

To address the prevalence and complexities of sustainable development challenges around the world, organizations in the business, government, and non-profit sectors are increasingly collaborating via multi-stakeholder partnerships. Because complex problems can be neither understood nor addressed by a single organization, it is necessary to bring together the knowledge and resources of many stakeholders. Yet, how these partnerships coordinate their collaborative activities to achieve mutual and organization-specific goals is not well understood. This study takes an organization design perspective of (...) collaborative decision-making processes to explore how they impact the effectiveness of multi-stakeholder partnerships. We compare the decision-making processes of 94 sustainability-focused multi-stakeholder partnerships and find that collaborative decision-making has an indirect and positive impact on partnership capacity through systems that keep partners informed, coordinate partner interactions, and facilitate ongoing learning. The implications of this study for multi-stakeholder partnership research and practice are that partnership capacity is contingent on the design of decision-making processes, as well as internal mechanisms that coordinate and monitor collaborative activities. (shrink)

Although virtue ethics has repeatedly been proposed as a suitable framework for the development of artificial moral agents, it has been proven difficult to approach from a computational perspective. In this work, we present the first technical implementation of artificial virtuous agents in moral simulations. First, we review previous conceptual and technical work in artificial virtue ethics and describe a functionalistic path to AVAs based on dispositional virtues, bottom-up learning, and top-down eudaimonic reward. We then provide the details of a (...) technical implementation in a moral simulation based on a tragedy of the commons scenario. The experimental results show how the AVAs learn to tackle cooperation problems while exhibiting core features of their theoretical counterpart, including moral character, dispositional virtues, learning from experience, and the pursuit of eudaimonia. Ultimately, we argue that virtue ethics provides a compelling path toward morally excellent machines and that our work provides an important starting point for such endeavors. (shrink)

Reinforcement Learning is a kind of machine learning. We know Profit Sharing, the Rational Policy Making algorithm, the Penalty Avoiding Rational Policy Making algorithm and PS-r* to guarantee the rationality in a typical class of the Partially Observable Markov Decision Processes. However they cannot treat continuous state spaces. In this paper, we present a solution to adapt them in continuous state spaces. We give RPM a mechanism to treat continuous state spaces in the environment that has the same (...) type of a reward. We show the effectiveness of the proposed method in numerical examples. (shrink)

Conflicts in land exploration are incisive social problems which have been the subject in many studies. Risk assessment of land conflicts is effective to resolve such problems. Specifically, fuzzy mathematics and the analytic hierarchy process were combined together to evaluate risk in land conflicts in our work, which is proved useful to solve uncertainty and imprecision problems. Based on the analysis of the principles for the risk assessment of a land conflicts index system, a set of risk assessment (...) indexes using the fuzzy analytic hierarchy process was presented. The results show that the overall risk is at medium level, and the risk of the feasibility index and controllability index need to be paid more attention to. The contribution of this article is reflected in two aspects: the application of FAHP in risk assessments of land conflicts is effective and valid; it is helpful for governments to establish a stricter management system of work safety for conflicts in land exploration based on the risk assessment results. (shrink)

In the existing Reinforcement Learning, it is difficult and time consuming to find appropriate the meta-parameters such as learning rate, eligibility traces and temperature for exploration, in particular on a complicated and large-scale problem, the delayed reward often occurs and causes a difficulty in solving the problem. In this paper, we propose a novel method introducing a temperature distribution for reinforcement learning. In addition to the acquirement of policy based on profit sharing, the temperature is (...) given to each state and is trained by hill-climbing method using likelihood function based on success and failure of the task. The proposed method can reduce the parameter setting according to the given problems. We showed the performance on the grid world problem and the control of Acrobot. (shrink)

The rainwater system is an important part of the urban infrastructure as well as a key hub for maintaining the dynamic operation of the city and a clear indicator of the level of urban development. With the rapid development of urbanization, the hardened area of roads and residential areas has increased, and the construction of rainwater systems is so far insufficient, causing the urban waterlogging and water pollution problems to become increasingly serious. Accordingly, combined with the “sponge city” construction concept (...) of the six-character policy of “seepage, retention, storage, use, purification, and drainage,” we propose to adopt measures for the local conditions and to reasonably select sponge city engineering measures to increase rainwater utilization, effectively reduce rainwater runoff, and alleviate the city waterlogging and water pollution problems. We used the analytic hierarchy process to evaluate the effect of a sponge city “pocket park” rainwater system in Chaohu City before and after the transformation. The results showed that the pocket park after the renovation was well controlled, the waterlogging was basically eliminated, the water quality pollution was clearly improved, and the ecological environment was significantly improved. (shrink)

Reinforcement Learning (RL) handles policy search problems: searching a mapping from state space to action space. However RL is based on gradient methods and as such, cannot deal with problems with multimodal landscape. In contrast, though Genetic Algorithm (GA) is promising to deal with them, it seems to be unsuitable for policy search problems from the viewpoint of the cost of evaluation. Minimal Generation Gap (MGG), used as a generation-alternation model in GA, generates many offspring from two or more (...) parents selected from a population. Therefore, evaluating policies of generated offspring requires much trial and error (i.e. interaction between an agent and an environment). In this paper, we incorporate importance sampling into the framework of MGG in order to reduce the cost of evaluation on policy search. The proposed techniques are applied to Markov Decision Process (MDP) with multimodal landscape. The experimental results show that these techniques can reduce the number of interaction between an agent and an environment, and also mean that MGG and importance sampling are good for each other. (shrink)

The authors aim at constructing an agent which learns appropriate actions in a Multi-Agent environment with and without social dilemmas. For this aim, the agent must have nonrationality that makes it give up its own profit when it should do that. Since there are many studies on rational learning that brings more and more profit, it is desirable to utilize them for constructing the agent. Therefore, we use a reward-handling manner that makes internal (...) evaluation from the agent's rewards, and then the agent learns actions by a rational learning method with the internal evaluation. If the agent has only a fixed manner, however, it does not act well in the environment with and without dilemmas. Thus, the authors equip the agent with several reward-handling manners and criteria for selecting an effective one for the environmental situation. In the case of humans, what generates the internal evaluation is usually called emotion. Hence, this study also aims at throwing light on emotional activities of humans from a constructive view. In this paper, we divide a Multi-Agent environment into three situations and construct an agent having the reward-handling manners and the criteria. We observe that the agent acts well in all the three Multi-Agent situations composed of homogeneous agents. (shrink)

This paper presents a GA-based multi-agent reinforce- ment learning bidding approach (GMARLB) for perform- ing multi-agent reinforcement learning. GMARLB inte- grates reinforcement learning, bidding and genetic algo- rithms. The general idea of our multi-agent systems is as follows: There are a number of individual agents in a team, each agent of the team has two modules: Q module and CQ module. Each agent can select actions to be performed at each (...) step, which are done by the Q module. While the CQ module determines at each step whether the agent should continue or relinquish control. Once an agent relinquishes its control, a new agent is selected by bidding algorithms. We applied GA-based GMARLB to the Backgammon game. The experimental results show GMARLB can achieve a su- perior level of performance in game-playing, outperforming PubEval, while the system uses zero built-in knowledge. (shrink)

This paper presents a human-computer interaction model with a three layers learning mechanism in a pervasive environment. We begin with a discussion around a number of important issues related to human-computer interaction followed by a description of the architecture for a multi-agent cooperative design system for pervasive computing environment. We present our proposed three- layer HCI model and introduce the group formation algorithm, which is predicated on a dynamic sharing niche technology. Finally, we explore the cooperative reinforcement (...) learning and fusion algorithms; the paper closes with concluding observations and a summary of the principal work and contributions of this paper. (shrink)

David Hume’s skeptical solution to the problem of induction was grounded in his belief that we learn by means of custom. We consider here how a form of reinforcement learning like custom may allow an agent to learn how to learn in other ways as well. Specifically, an agent may learn by simple reinforcement to adopt new forms of learning that work better than simple reinforcement in the context of specific tasks. We will consider (...) how such a bootstrapping process may lead to a system that includes trial-and-error forms of learning like win-stay/lose-shift, probe and adjust, and simple reinforcement itself together with higher-rationality inferential tools. (shrink)

How do we come to like the things that we do? Each one of us starts from a relatively similar state at birth, yet we end up with vastly different sets of aesthetic preferences. These preferences go on to define us both as individuals and as members of our cultures. Therefore, it is important to understand how aesthetic preferences form over our lifetimes. This poses a challenging problem: to understand this process, one must account for the many factors at (...) play in the formation of aesthetic values and how these factors influence each other over time. A general framework based on basic neuroscientific principles that can also account for this process is needed. Here, we present such a framework and illustrate it through a model that accounts for the trajectories of aesthetic values over time. Our framework is inspired by meta-analytic data of neuroimaging studies of aesthetic appraisal. This framework incorporates effects of sensory inputs, rewards, and motivational states. Crucially, each one of these effects is probabilistic. We model their interactions under a reinforcement-learning circuitry. Simulations of this model and mathematical analysis of the framework lead to three main findings. First, different people may develop distinct weighing of aesthetic variables because of individual variability in motivation. Second, individuals from different cultures and environments may develop different aesthetic values because of unique sensory inputs and social rewards. Third, because learning is stochastic stemming from probabilistic sensory inputs, motivations, and rewards, aesthetic values vary in time. These three theoretical findings account for different lines of empirical research. Through our study, we hope to provide a general and unifying framework for understanding the various aspects involved in the formation of aesthetic values over time. (shrink)

Our proposed cognitive distance learning problem solver generates sequence of actions from initial state to goal states in problem state space. This problem solver learns cognitive distance of arbitrary combination of two states. Action generation at each state is selection of next state that has minimum cognitive distance to the goal, like Q-learning agent. In this paper, first, we show that our proposed method reduces search cost than conventional search method by analytical simulation in spherical state (...) space. Second, we show that an average search cost is more reduced more the prior learning term is long and our problem solver is familiar to the environment, by a computer simulation in a tile world state space. Third, we showed that proposed problem solver is superior to the reinforcement learning techniques when goal is changed by a computer simulation. Forth, we found that our simulation result consist with psychological experimental results. (shrink)

We investigate how Reinforcement Learning methods can be used to solve the production selection and production ordering problem in ACT‐R. We focus on four algorithms from the Q learning family, tabular Q and three versions of Deep Q Networks, as well as the ACT‐R utility learning algorithm, which provides a baseline for the Q algorithms. We compare the performance of these five algorithms in a range of lexical decision tasks framed as sequential decision problems.

In a multiagent and multi-cultural world, the fine-grained analysis of agents’ dynamic behaviour, i.e. of their activities, is essential. Dynamic activities are actions that are characterized by an agent who executes the action and by other participants of the action. Wh-questions on the participants of the actions pose a difficult particular challenge because the variability of the types of possible answers to such questions is huge. To deal with the problem, we propose the analysis and classification of (...) Wh-questions apt for agents’ communication in a multiagent system (MAS). Our proposal of such a system consists of agents who communicate with their fellow agents by messaging so that each autonomous agent, though resource-bounded, can make less or more rational decisions to meet its own and collective goals. In addition, by communicating with other fellow agents and their environment, agents can learn new concepts and enrich their ontology so that their behaviour is dynamic. We aim to make a general proposal of the system so that the ‘envelope’ of agents’ messages can be formalized in any MAS standard, be it The Foundation for Intelligent Physical Agents - Agent Communication Language (FIPA-ACL) or Knowledge Query and Manipulation Language (KQML). Yet, the content of messages is encoded in a formalized natural language. To this end, we apply Transparent Intensional Logic (TIL) with its procedural semantics which is particularly apt for a fine-grained analysis in which all the semantically salient features of natural language can be plausibly formalized. In this paper, we concentrate on analysing the content of query messages, particularly the content of those that encode Wh-questions and the answers to them. We also summarize TIL deduction system that makes it possible to answer such questions in an intelligent way. Linguists distinguish several subtypes of Wh-questions. Though the linguistic classification is helpful, it is not always suitable for agents’ communication and reasoning. We need the classification based on a logical analysis of Wh-questions so that the agents can infer possible answers to such questions rather than only looking for them by keywords. This paper aims to apply an appropriate classification of the logical types of Wh-questions and the analysis of such questions; we concentrate in particular on questions concerning the participants of activities. The application of these results to the analysis of processes and events based on verb valency frames is another novelty of the paper. (shrink)

One most common problem in reinforcement learning systems (e.g. Q-learning) is to reduce the number of trials to converge to an optimal policy. As one of the solution to the problem, k-certainty exploration method was proposed. Miyazaki reported that this method could determine an optimal policy faster than Q-learning in Markov decision processes (MDPs). This method is very efficient learning method. But, we propose an improvement plan that makes this method more efficient. In k-certainty exploration method, in (...) case there is no k-uncertainty rule (rules which aren’t selected more than k times) in current state, an agent sometimes walks randomly until it finds a state where it can select a k-uncertainty rule. We think that this behavior is not efficient. To reduce this useless behavior, we propose combining k-certainty exploration method with Dynamic programming (DP). Miyazaki’s system uses DP after all rules are executed at least k times. But, our method uses k-certainty exploration method along with DP during learning. Our method, takes two pattern actions. In case an agent can select k-uncertainty rules, one of these rules is selected at random as k-certainty exploration method. In another case there is no k-uncertainty rule, behavior of agent is different from the behavior of k-certainty exploration method. In that case, our method uses DP to compute an optimal policy for moving from a current state to a state in which some k-uncertainty rules remain. The model for DP is constructed by using only known states. The outline will be described below. First, an agent makes a map constructed by using only known states. In the map, goals are states in which there are k-uncertainty rules and arbitrary state values are set in these states. Point to which attention should be paid is that the map is not given from outside. It is made from only experience. Next, each state values of states in which there are only k-certainty rules in the map are computed by DP (we used Policy Iteration). Finally, an agent continues to select greedy action until it arrives at a state in which it can select a k-uncertainty rule. By this improvement, we expect it can determine an optimal policy faster than k-certainty exploration method. And we have verified that our exploration method can determine an optimal policy faster than k-certainty exploration method by computer simulation. (shrink)

This article proposes an Analytic Hierarchy Process Dempster-Shafer and similarity-based network selection algorithm for the scenario of dynamic changes in user requirements and network environment; combines machine learning with network selection and proposes a decision tree-based network selection algorithm; combines multiattribute decision-making and genetic algorithm to propose a weighted Gray Relation Analysis and genetic algorithm-based network access decision algorithm. Firstly, the training data is obtained from the collaborative algorithm, and it is used as the training set, and the (...) network attributes are used as the attribute set, and the continuous attributes are discretized by dichotomization, and the attribute that can make the greatest information gain is selected as the division feature, and a decision tree with strong generalization ability is finally obtained, which is used as the decision basis for network access selection. The simulation results show that the algorithm proposed in this thesis can effectively improve user service quality under three services, and the algorithm is simple and effective with low complexity. It first uses AHP-DS hierarchical analysis to establish a recursive hierarchy for the network selection problem and obtains the subjective weights of network attributes through the judgment matrix. Then, it uses a genetic algorithm to adjust the subjective weight, defines the fitness function in the genetic algorithm-based on gray correlation analysis, adjusts the weights of the selection operator, crossover operator, and variation operator in the genetic algorithm, and gets the network with the largest fitness as the target network, which can effectively improve the user service quality. (shrink)

Connected vehicle fleets have formed significant component of industrial internet of things scenarios as part of Industry 4.0 worldwide. The number of vehicles in these fleets has grown at a steady pace. The vehicles monitoring with machine learning algorithms has significantly improved maintenance activities. Predictive maintenance potential has increased where machines are controlled through networked smart devices. Here, benefits are accrued considering uptimes optimization. This has resulted in reduction of associated time and labor costs. It has also provided significant increase (...) in cost benefit ratios. Considering vehicle fault trends in this research predictive maintenance problem is addressed through hybrid deep learning-based ensemble method (HDLEM). The ensemble framework which acts as predictive analytics engine comprises of three deep learning algorithms viz modified cox proportional hazard deep learning (MCoxPHDL), modified deep learning embedded semi supervised learning (MDLeSSL) and merged LSTM (MLSTM) networks. Both sensor as well as historical maintenance data are collected and prepared using benchmarking methods for HDLEM training and testing. Here, times between failures (TBF) modeling and prediction on multi-source data are successfully achieved. The results obtained are compared with stated deep learning models. This ensemble framework offers great potential towards achieving more profitable, efficient and sustainable vehicle fleet management solutions. This helps better telematics data implementation which ensures preventative management towards desired solution. The ensemble method's superiority is highlighted through several experimental results. (shrink)

The role to be played by multi-stakeholder partnerships in addressing the ‘wicked problems’ of sustainable development is made explicit by the seventeenth Sustainable Development Goal. But how do these partnerships really work? Based on the analysis of four sustainability-oriented innovation initiatives implemented in Belgium, Italy, Germany, and France, this study explores the roles and mechanisms that collaborating actors may enact to facilitate the pursuit of sustainable development, with a particular focus on non-profit organizations. The results suggest that (...) collaborative innovations for sustainability contribute simultaneously to the fulfilment of different Sustainable Development Goals, reaching beyond their original intent, and that the value being created has the potential to reinforce such roles and mechanisms. These partnerships are prompted and managed by non-profit organizations that act as metagovernors of collaborative innovation processes as they play the roles of cultural spreaders, enablers, relational brokers, service provides, and influencers. These findings will help policy-makers and practitioners in the public and non-profit sector to identify and utilize emerging opportunities for value creation through collaborative innovation, and to better design existing and prospective collaborative efforts aimed at sustainable objectives, thereby supporting progress towards the implementation of Agenda 2030. (shrink)

A deep reinforcement learning-based computational guidance method is presented, which is used to identify and resolve the problem of collision avoidance for a variable number of fixed-wing UAVs in limited airspace. The cooperative guidance process is first analyzed for multiple aircraft by formulating flight scenarios using multiagent Markov game theory and solving it by machine learning algorithm. Furthermore, a self-learning framework is established by using the actor-critic model, which is proposed to train collision avoidance decision-making neural networks. To (...) achieve higher scalability, the neural network is customized to incorporate long short-term memory networks, and a coordination strategy is given. Additionally, a simulator suitable for multiagent high-density route scene is designed for validation, in which all UAVs run the proposed algorithm onboard. Simulated experiment results from several case studies show that the real-time guidance algorithm can reduce the collision probability of multiple UAVs in flight effectively even with a large number of aircraft. (shrink)

In this paper I will discuss a certain philosophical and conceptual program -- that I have called philosophy of social action writ large -- and also show in detail how parts of the program have been, and is currently being carried out. In current philosophical research the philosophy of social action can be understood in a broad sense to encompass such central research topics as action occurring in a social context (this includes multi-agent action); shared we-attitudes (such as (...) we-intention, mutual belief) and other social attitudes expressing collective intentionality and needed for the explication and explanation of social action; social macro-notions, such as actions performed by social groups and properties of social groups such as their goals and beliefs; social practices, and institutions (see e.g. Tuomela, 1995, 2000a, 2001). The theory of social action understood analogously in a broad sense would then involve not only philosophical but all other relevant theorizing about social action. Thus, in this sense, such fields of Artificial Intelligence (AI) as Distributed AI (DAI) and the theory of Multi-Agent Systems (MAS) fall within the scope of the theory of social action. DAI studies the social side of computer systems and includes various well-known areas ranging from human-computer interaction, computer-supported cooperative work, organizational processing, and distributed problem solving to the simulation of social systems. (shrink)

Photo by Thought Catalog on Unsplash INTRODUCTION The coronavirus pandemic is a challenge to educators, policy makers, and ordinary people. In facing the threat from COVID-19, school systems and global institutions need “to address the essential matter of each human being and how they are interacting with, and affected by, a much wider set of biological and technical conditions.”[1] Educators must grapple with the societal issues that come with the intent of ensuring the safety of the public. To some, “these (...) are actually as important as the biological concerns of people.”[2] The current global crisis shows that “scientifically, socially, and politically the economy and technosphere are not just related, they are integral to a comprehensive response to major challenges.”[3] In developing these responses, scientists, government leaders, and policy makers need to consider the vulnerabilities of people, especially those in “thrown away” groups.[4] Jerome Ravetz explains that “microscopic viral predators cull our populations, as ever, but with a selection that is not natural but social and political.”[5] Educators must address the underlying vulnerabilities and evaluate the virus as a threat to academic experiences and access to a fair education. ANALYSIS • Critical Pedagogy By definition, the critical approach to teaching is about the problem-posing method of education developed by Paolo Freire. In his Pedagogy of the Oppressed, he proposed a paradigm shift away from the banking method of learning wherein teachers deposit knowledge into the minds of students. Critical pedagogy is an educational approach that challenges students to develop the ability to recognize and criticize dominating theories and evaluate them in their social context. Teachers press students to recognize oppression and try to remedy oppression in their culture.[6] Despite the lack of in-person interaction between the teacher and the students, the effort to use innovative teaching techniques like critical pedagogy should continue. Online learning is not just about the use of technology, although the internet is crucial in the delivery of content. Since human beings are creators of value, they determine the meaning and purpose of technology. In this way, the set of values people have will influence online learning. Teachers cannot be more concerned about outcomes than about the process itself. The process is crucial since the ability of the student to think critically is developed in the exchange between the student and the teacher. The teacher cannot simply dump loads of information (deposit knowledge) but must pose problems to test the analytical and critical skills of students.[7] Education is about how people humanize the world. Policy makers miss the point when they focus on the delivery but do not pay attention to the substantive aspect of learning, which is human empowerment. Education is meant to expand the freedom of people. Education should be seen as an integrative activity. Learning is a formative process that aims to develop the human person. Without the face-to-face encounter between teacher and students, the challenge is finding ways to make learning an effective means to mold the values of young people. • Online Critical Pedagogy & the Role of Technology Physical distance appears to be an impediment in realizing the ideals of learning. The lack of contact between the teacher and the students may prevent a more meaningful interaction since online instruction is impersonal. It can be argued that there is no alternative to some classroom activities, especially laboratory experiments in science courses. The total classroom environment naturally influences the behavior of students when it comes to academic work: the look in the eyes of the professor, the caring ways of a teacher, or the pressure while taking exams contribute to an experience that only a classroom setting can provide. If implemented properly, technology can facilitate the personal relationships between teachers and students while providing meaningful experiences. With the current need for online learning during the COVID-19 pandemic, gadgets are indispensable. Some would even consider gadgets an extension of the human body. A mobile phone is not just any instrument; it has evolved into a novel way of being in the world. Our gadgets are a means of seeing how the outside world unfolds. Modern technological tools allow the interface of people in many fields of experience in a globalized environment. Computers and other digital devices extend the meaning and value of human freedom. For example, a laptop provided to a poor child can redefine the meaning of and what the future might hold for that child. The device is crucial to the whole learning process. Modern tools are critical to self-discovery and greater freedom. The values that people embrace will matter in the new normal. The internet has provided a new democratic space that empowers groups and individuals to express themselves and to understand the world.[8] In an online class, students and teachers alike need to analyze big picture questions and layers of information. For example, the student in an ethics or philosophy class can reflect on the realities of life. With the proper guidance, online education should help define the meaning of moral commitment and human responsibility. CONCLUSION Before the pandemic, policy makers had been pursuing the goals of a globalized economy. They had been fashioning and promoting programs that cater to the interests of a consumer-driven world that has deprived the poor of opportunity. When the pandemic struck, globalization suddenly came to a halt and people realized the things that truly matter in life – family, love, and life itself. The new normal must now emphasize the role of education as a source of inner strength that can empower the person to live well reinforcing values based on a social consciousness. Critical pedagogy is possible under the new normal. The distance between the teacher and student does not make the educational process less real and in the absence of a vaccine, online learning is the safest strategy. Governments cannot freeze an entire school year since education is the only way out from poverty for millions. The COVID-19 pandemic provides a reason to re-imagine teaching using technology to encourage all students to question systems of oppression or greed, as the persistent pursuit of the truth is what education is all about. [1] Hartsell, Layne, Krabbe, Alexander, & Pastreich, Emmanuel. “Covid-19, Global Justice, and a New Biopolitics of the Anthropocene.” Social Ethics Society Journal of Applied Philosophy 6, no. 2 (2020), 3. [2] Ravetz, Jerome. “Science for a Proper Recovery: Post-Normal, not New Normal.” Issues in Science and Technology [Internet] July 15, 2020. https://issues.org/post-normal-science-for-pandemic-recovery/ [3] Hartsell, et al. “Covid-19, Global Justice, and a New Biopolitics of the Anthropocene,” 6. [4] Ravetz, “Science for a Proper Recovery: Post-Normal, not New Normal.” [5] Ibid. [6] Freire, Paolo. Pedagogy of the Oppressed. (New York: Continuum Books, 1993). [7] Freire writes that in such a situation “instead of communicating, the teacher issues communiqués and makes deposits which the students patiently receive, memorize, and repeat.” See his Pedagogy of the Oppressed. [8] Bakardjieva, Maria. Internet Society. (London: Sage Publications, 2005). (shrink)

We argue that cognitive models can provide a common ground between human users and deep reinforcement learning (Deep RL) algorithms for purposes of explainable artificial intelligence (AI). Casting both the human and learner as cognitive models provides common mechanisms to compare and understand their underlying decision-making processes. This common grounding allows us to identify divergences and explain the learner's behavior in human understandable terms. We present novel salience techniques that highlight the most relevant features in each model's decision-making, as (...) well as examples of this technique in common training environments such as Starcraft II and an OpenAI gridworld. (shrink)

Evolution and learning are two analytically distinct concepts. People learn yet evolution (`change') does not necessarily take place. To clarify this problem the concept of learning is explicated. The first problem addressed is the question of who is learning. Here a shift from the single actor perspective to an interaction perspective is proposed (using Habermas and Luhmann as theoretical arguments for such a shift). Both, however, idealize the preconditions that interactants share while learning collectively. Against rationalist assumptions it (...) is argued that in order to learn people need a narratively based shared world. What do they learn? They acquire knowledge and they learn how to learn. This still does not solve the problem why they learn. Learning, it is argued, does not guarantee evolution but provides the mutations for evolutionary processes to take place. (shrink)

A fundamental question in reading research concerns whether attention is allocated strictly serially, supporting lexical processing of one word at a time, or in parallel, supporting concurrent lexical processing of two or more words (Reichle, Liversedge, Pollatsek, & Rayner, 2009). The origins of this debate are reviewed. We then report three simulations to address this question using artificial reading agents (Liu & Reichle, 2010; Reichle & Laurent, 2006) that learn to dynamically allocate attention to 1–4 words to “read” as efficiently (...) as possible. These simulation results indicate that the agents strongly preferred serial word processing, although they occasionally attended to more than one word concurrently. The reason for this preference is discussed, along with implications for the debate about how humans allocate attention during reading. (shrink)

Theories of decision-making and its neural substrates have long assumed the existence of two distinct and competing valuation systems, variously described as goal-directed vs. habitual, or, more recently and based on statistical arguments, as model-free vs. model-based reinforcement-learning. Though both have been shown to control choices, the cognitive abilities associated with these systems are under ongoing investigation. Here we examine the link to cognitive abilities, and find that individual differences in processing speed covary with a shift from model-free to (...) model-based choice control in the presence of above-average working memory function. This suggests shared cognitive and neural processes; provides a bridge between literatures on intelligence and valuation; and may guide the development of process models of different valuation components. Furthermore, it provides a rationale for individual differences in the tendency to deploy valuation systems, which may be important for understanding the manifold neuropsychiatric diseases associated with malfunctions of valuation. (shrink)