In this paper, we discuss guidelines for a reward design problem that defines when and what amount of reward should be given to the agent/s, within the context of reinforcement learning approach. We would like to take keepaway soccer as a standard task of the multiagent domain which requires skilled teamwork. The difficulties of designing reward for this task are due to its features as follows: i) since it belongs to the continuing task which (...) has no explicit goal to achieve, it is hard to tell when reward should be given to the agent/s. ii) since it is a multiagent cooperative task, it is hard to decide what is a fair share of reward for each agent's contribution to achieve the goal. Through some experiments, we show that the reward design have a major effect on the agent's behavior, and introduce the successful reward function that makes agents perform keepaway better and more interesting than the conventional one does. Finally, we explore the relationship between `reward design' and `acquired behaviors' from the viewpoint of teamwork. (shrink)

Autonomous underwater vehicles are widely used to accomplish various missions in the complex marine environment; the design of a control system for AUVs is particularly difficult due to the high nonlinearity, variations in hydrodynamic coefficients, and external force from ocean currents. In this paper, we propose a controller based on deep reinforcement learning in a simulation environment for studying the control performance of the vectored thruster AUV. RL is an important method of artificial intelligence that can learn behavior (...) through trial-and-error interactions with the environment, so it does not need to provide an accurate AUV control model that is very hard to establish. The proposed RL algorithm only uses the information that can be measured by sensors inside the AUVs as the input parameters, and the outputs of the designed controller are the continuous control actions, which are the commands that are set to the vectored thruster. Moreover, a reward function is developed for deep RL controller considering different factors which actually affect the control accuracy of AUV navigation control. To confirm the algorithm’s effectiveness, a series of simulations are carried out in the designed simulation environment, which is a method to save time and improve efficiency. Simulation results prove the feasibility of the deep RL algorithm applied to the control system for AUV. Furthermore, our work also provides an optional method for robot control problems to deal with improving technology requirements and complicated application environments. (shrink)

As a universal language in the world, English has become a necessary language communication tool under the globalization of trade. Intelligent, efficient, and reasonable English language-assisted learning system helps to further improve the English ability of language learners. English online learning dictionary, as an important query tool for English learners, is an important part of English online learning. This paper will optimize the design of English online learning dictionary system based on multiagent architecture. Based on the hybrid (...) class='Hi'>multiagent cooperative algorithm, this paper will improve the disadvantages of the online English learning dictionary system and propose an appropriate dictionary application evaluation function. At the same time, an improved reinforcement learning algorithm is introduced into the corresponding English online learning dictionary navigation problem so as to improve the efficiency of the online English learning dictionary system. English online learning dictionary is more intelligent and efficient. In this paper, the new online learning dictionary system optimization algorithm is proposed and compared with the traditional system algorithm. The experimental results show that the algorithm proposed in this paper solves the collaborative confusion problem of English learning online dictionary to a certain extent and further solves the corresponding navigation problem so as to improve the efficiency. (shrink)

A tracking control algorithm of nonlinear multiple agents with undirected communication is studied for each multiagent system affected by external interference and input saturation. A control design scheme combining iterative learning and adaptive control is proposed to perform parameter adaptive time-varying adjustment and prove the effectiveness of the control protocol by designing Lyapunov functions. Simulation results show that the high-precision tracking control problem of the nonlinear multiagent system based on adaptive iterative learning control can be well realized (...) even when the input is saturated. Finally, the validity of the proposed algorithm is verified by numerical analysis. (shrink)

Can humans get arbitrarily capable reinforcement learning agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? This question impacts how far RL can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we study when an RL agent has an instrumental goal to tamper with its reward process, and describe design (...) principles that prevent instrumental goals for two different types of reward tampering. Combined, the design principles can prevent reward tampering from being an instrumental goal. The analysis benefits from causal influence diagrams to provide intuitive yet precise formalizations. (shrink)

The area of smart power grids needs to constantly improve its efficiency and resilience, to provide high quality electrical power in a resilient grid, while managing faults and avoiding failures. Achieving this requires high component reliability, adequate maintenance, and a studied failure occurrence. Correct system operation involves those activities and novel methodologies to detect, classify, and isolate faults and failures and model and simulate processes with predictive algorithms and analytics. In this paper, we showcase the application of a complex-adaptive, self-organizing (...) modeling method, and Probabilistic Boolean Networks, as a way towards the understanding of the dynamics of smart grid devices, and to model and characterize their behavior. This work demonstrates that PBNs are equivalent to the standard Reinforcement Learning Cycle, in which the agent/model has an interaction with its environment and receives feedback from it in the form of a reward signal. Different reward structures were created to characterize preferred behavior. This information can be used to guide the PBN to avoid fault conditions and failures. (shrink)

The fundamental cognitive problem for active organisms is to decide what to do next in a changing environment. In this article, we analyze motor and action control in computational models that utilize reinforcement learning (RL) algorithms. In reinforcement learning, action control is governed by an action selection policy that maximizes the expected future reward in light of a predictive world model. In this paper we argue that RL provides a way to explicate the so-called action-oriented views of (...) cognitive systems in representational terms. (shrink)

A deep reinforcement learning-based computational guidance method is presented, which is used to identify and resolve the problem of collision avoidance for a variable number of fixed-wing UAVs in limited airspace. The cooperative guidance process is first analyzed for multiple aircraft by formulating flight scenarios using multiagent Markov game theory and solving it by machine learning algorithm. Furthermore, a self-learning framework is established by using the actor-critic model, which is proposed to train collision avoidance decision-making neural networks. To (...) achieve higher scalability, the neural network is customized to incorporate long short-term memory networks, and a coordination strategy is given. Additionally, a simulator suitable for multiagent high-density route scene is designed for validation, in which all UAVs run the proposed algorithm onboard. Simulated experiment results from several case studies show that the real-time guidance algorithm can reduce the collision probability of multiple UAVs in flight effectively even with a large number of aircraft. (shrink)

This paper suggests an online solution for the optimal tracking control of robotic systems based on a single critic neural network -based reinforcement learning method. To this end, we rewrite the robotic system model as a state-space form, which will facilitate the realization of optimal tracking control synthesis. To maintain the tracking response, a steady-state control is designed, and then an adaptive optimal tracking control is used to ensure that the tracking error can achieve convergence in an optimal sense. (...) To solve the obtained optimal control via the framework of adaptive dynamic programming, the command trajectory to be tracked and the modified tracking Hamilton-Jacobi-Bellman are all formulated. An online RL algorithm is the developed to address the HJB equation using a critic NN with online learning algorithm. Simulation results are given to verify the effectiveness of the proposed method. (shrink)

Most current online distributed machine learning algorithms have been studied in a data-parallel architecture among agents in networks. We study online distributed machine learning from a different perspective, where the features about the same samples are observed by multiple agents that wish to collaborate but do not exchange the raw data with each other. We propose a distributed feature online gradient descent algorithm and prove that local solution converges to the global minimizer with a sublinear rate O 2 T. Our (...) algorithm does not require exchange of the primal data or even the model parameters between agents. Firstly, we design an auxiliary variable, which implies the information of the global features, and estimate at each agent by dynamic consensus method. Then, local parameters are updated by online gradient descent method based on local data stream. Simulations illustrate the performance of the proposed algorithm. (shrink)

The large scale, time varying, and diversification of physically coupled networked infrastructures such as power grid and transportation system lead to the complexity of their controller design, implementation, and expansion. For tackling these challenges, we suggest an online distributed reinforcement learning control algorithm with the one-layer neural network for each subsystem or called agents to adapt the variation of the networked infrastructures. Each controller includes a critic network and action network for approximating strategy utility function and desired control (...) law, respectively. For avoiding a large number of trials and improving the stability, the training of action network introduces supervised learning mechanisms into reduction of long-term cost. The stability of the control system with learning algorithm is analyzed; the upper bound of the tracking error and neural network weights are also estimated. The effectiveness of our proposed controller is illustrated in the simulation; the results indicate the stability under communication delay and disturbances as well. (shrink)

We study a joint pricing and inventory control problem for perishables with positive lead time in a finite horizon periodic-review system. Unlike most studies considering a continuous density function of demand, in our paper the customer demand depends on the price of current period and arrives according to a homogeneous Poisson process. We consider both backlogging and lost-sales cases, and our goal is to find a simultaneously ordering and pricing policy to maximize the expected discounted profit over the planning horizon. (...) When there is no fixed ordering cost involved, we design a deep reinforcement learning algorithm to obtain a near-optimal ordering policy and show that there are some monotonicity properties in the learned policy. We also show that our deep reinforcement learning algorithm achieves a better performance than tabular-based Q-learning algorithms. When a fixed ordering cost is involved, we show that our deep reinforcement learning algorithm is effective and efficient, under which the problem of “curse of dimension” is circumvented. (shrink)

Successfully explaining and replicating the complexity and generality of human and animal learning will require the integration of a variety of learning mechanisms. Here, we introduce a computational model which integrates associative learning (AL) and reinforcement learning (RL). We contrast the integrated model with standalone AL and RL models in three simulation studies. First, a synthetic grid‐navigation task is employed to highlight performance advantages for the integrated model in an environment where the reward structure is both diverse and (...) dynamic. The second and third simulations contrast the performances of the three models in behavioral experiments, demonstrating advantages for the integrated model in accounting for behavioral data. (shrink)

The generation of the pitch control signal in a wind turbine (WT) is not straightforward due to the nonlinear dynamics of the system and the coupling of its internal variables; in addition, they are subjected to the uncertainty that comes from the random nature of the wind. Fuzzy logic has proved useful in applications with changing system parameters or where uncertainty is relevant as in this one, but the tuning of the fuzzy logic controller (FLC) parameters is neither straightforward nor (...) an easy task. On the other hand, reinforcement learning (RL) allows systems to automatically learn, and this capability can be exploited to tune the FLC. In this work, a WT pitch control architecture that uses RL to tune the membership functions and scale the output of a fuzzy controller is proposed. The RL strategy calculates the fuzzy controller gains in order to reduce the output power error of the WT according to the wind speed. Different reward mechanisms based on the output power error have been considered. Simulation results with different wind profiles show that this architecture performs better (123.7 W) in terms of power errors than an FLC without RL (133.2 W) or a simpler PID (208.8 W). Even more, it provides a smooth response and outperforms other hybrid controllers such as RL-PID and radial basis function neural network control. (shrink)

As one of the most effective medical technologies for the infertile patients, in vitro fertilization has been more and more widely developed in recent years. However, prolonged waiting for IVF procedures has become a problem of great concern, since this technology is only mastered by the large general hospitals. To deal with the insufficiency of IVF service capacity, this paper studies an IVF queuing network in an integrated cloud healthcare system, where the two key medical services, that is, egg retrieval (...) and transplantation, are assigned to accomplish in the general hospital, while the routine medical tests are assigned into the community hospital. Based on continuous-time Markov procedure, a dynamic large-scale server scheduling problem in this complicated service network is modeled with consideration of different arrival rates of multiple type of patients and different service capacities of multiple servers that can be defined as doctors of the general hospital. To solve this model, a reinforcement learning algorithm is proposed, where the reward functions are designed for four conflicting subcosts: setup cost, patient waiting cost, penalty cost for unsatisfied patient personal preferences, and medical cost of patient. The experimental results show that the optimal service rule of each server’s queue obtained by the RL method is significantly superior to the traditional service rule. (shrink)

Recent work in the cognitive sciences has argued that beliefs sometimes acquire signaling functions in virtue of their ability to reveal information that manipulates “mindreaders.” This paper sketches some of the evolutionary and design considerations that could take agents from solipsistic goal pursuit to beliefs that serve as social signals. Such beliefs will be governed by norms besides just the traditional norms of epistemology. As agents become better at detecting the agency of others, either through evolutionary history or individual (...) learning, the candidate pool for signaling expands. This logic holds for natural and artificial agents that find themselves in recurring social situations that reward the sharing of one’s thoughts. (shrink)

This paper presents a human-computer interaction model with a three layers learning mechanism in a pervasive environment. We begin with a discussion around a number of important issues related to human-computer interaction followed by a description of the architecture for a multi-agent cooperative design system for pervasive computing environment. We present our proposed three- layer HCI model and introduce the group formation algorithm, which is predicated on a dynamic sharing niche technology. Finally, we explore the cooperative reinforcement learning (...) and fusion algorithms; the paper closes with concluding observations and a summary of the principal work and contributions of this paper. (shrink)

In this paper, to solve the problem of slow design of arts and crafts and to improve design efficiency and aesthetics, the existing big data and 3D technology are used to conduct an in-depth analysis of the optimization of the rapid design system of arts and crafts machine salt baking. In the system requirement analysis, the functional modules of this system are identified as nine functional modules such as design terminology management system and external information import (...) function according to the actual usage requirements. In the system design, the overall structure design, database design, and functional module design of the system are comprehensively elaborated, and the key issues such as 3D display and home layout generation algorithm based on reinforcement learning are analyzed and designed. In the implementation part of the system, the overall construction of the system and the composition of functional modules are introduced in detail and the main functional modules of the system are presented with interface diagrams. In the system implementation part, the overall system construction and functional module composition are introduced in detail, the main functional modules of the system are shown with interface diagrams, codes, and algorithms, and the specific implementation process of 3D display and soft layout algorithms are also explained in detail. The process of Surface Mount Technology big data processing and analysis is designed, and the design of SMT production line data collection scheme and real-time data processing architecture is completed. Based on the characteristics of SMT production line data, the K-means algorithm is used to detect data outliers and verify the accuracy of the method; also, the Spark-based association rule printing parameter recommendation model is designed, and the efficiency of the Apriori algorithm is significantly improved by parallelization. (shrink)

Reinforcement learning is a kind of machine learning. It aims to adapt an agent to a given environment with a clue to rewards. In general, the purpose of reinforcement learning system is to acquire an optimum policy that can maximize expected reward per an action. However, it is not always important for any environment. Especially, if we apply reinforcement learning system to engineering, environments, we expect the agent to avoid all penalties. In Markov Decision Processes, a (...) pair of a sensory input and an action is called rule. We call a rule penalty if and only if it has a penalty or it can transit to a penalty state where it does not contribute to get any reward. After suppressing all penalty rules, we aim to make a rational policy whose expected reward per an action is larger than zero. In this paper, we propose a suppressing penalty algorithm that can suppress any penalty and get a reward constantly. By applying the algorithm to the tick-tack-toe, its effectiveness is shown. (shrink)

Reinforcement learning approaches to cognitive modeling represent task acquisition as learning to choose the sequence of steps that accomplishes the task while maximizing a reward. However, an apparently unrecognized problem for modelers is choosing when, what, and how much to reward; that is, when (the moment: end of trial, subtask, or some other interval of task performance), what (the objective function: e.g., performance time or performance accuracy), and how much (the magnitude: with binary, categorical, or continuous values). (...) In this article, we explore the problem space of these three parameters in the context of a task whose completion entails some combination of 36 state–action pairs, where all intermediate states (i.e., after the initial state and prior to the end state) represent progressive but partial completion of the task. Different choices produce profoundly different learning paths and outcomes, with the strongest effect for moment. Unfortunately, there is little discussion in the literature of the effect of such choices. This absence is disappointing, as the choice of when, what, and how much needs to be made by a modeler for every learning model. (shrink)

GoalThis paper presents an immersive Virtual Reality system to analyze and train Executive Functions of soccer players. EFs are important cognitive functions for athletes. They are a relevant quality that distinguishes amateurs from professionals.MethodThe system is based on immersive technology, hence, the user interacts naturally and experiences a training session in a virtual world. The proposed system has a modular design supporting the extension of various so-called game modes. Game modes combine selected game mechanics with specific simulation content to (...) target particular training aspects. The system architecture decouples selection/parameterization and analysis of training sessions via a coaching app from an Unity3D-based VR simulation core. Monitoring of user performance and progress is recorded by a database that sends the necessary feedback to the coaching app for analysis.ResultsThe system is tested for VR-critical performance criteria to reveal the usefulness of a new interaction paradigm in the cognitive training and analysis of EFs. Subjective ratings for overall usability show that the design as VR application enhances the user experience compared to a traditional desktop app; whereas the new, unfamiliar interaction paradigm does not negatively impact the effort for using the application.ConclusionThe system can provide immersive training of EF in a fully virtual environment, eliminating potential distraction. It further provides an easy-to-use analyzes tool to compare user but also an automatic, adaptive training mode. (shrink)

This article describes a brief overview of systems design concepts, and provides an example of the use of one very simple framework for utilizing systems design. Its purpose is to demonstrate the value of even the simplest of systems design models in clarifying the issues behind what are often perceived to be organizational conflicts. The example provided is that of a medical function within an industrial organization, but the implications apply to almost any support function or department (...) found within a large organization. (shrink)

The paper presents an approach for developing multi-agent reinforcement learning systems that are made up of a coalition of modular agents. We focus on learning to segment sequences (sequential decision tasks) to create modular structures, through a bidding process that is based on reinforcements received during task execution. The approach segments sequences (and divides them up among agents) to facilitate the learning of the overall task. Notably, our approach does not rely on a priori knowledge or a priori structures. (...) Initial experiments demonstrated the basic promise of the approach. This work shows how bidding and reinforcement learning can be usefully combined, thus pointing to a new research direction. (shrink)

Reinforcement learning relies on the reward prediction error (RPE) signals conveyed by the midbrain dopamine system. Previous studies showed that dopamine plays an important role in both positive and negative reinforcement. However, whether various reinforcement processes will induce distinct learning signals is still unclear. In a probabilistic learning task, we examined RPE signals in different reinforcement types using an electrophysiology index, namely, the feedback-related negativity (FRN). Ninety-four participants were randomly assigned into four groups: base (no (...) money incentive), positive reinforcement (presentation of money rewards), negative reinforcement (removal of money losses), and combined reinforcement (money rewards and removal of money losses) groups. In addition, in order to evaluate the engagement of emotional activity in the different reinforcement processes, Positive and Negative Affect Schedule-Expanded Form (PANAS-X) scales were applied before and after the experiment to detect the emotional changes. The results showed that there was no difference between groups in the dopamine-related learning bias. However, compared to the other three groups, negative reinforcement elicited smaller FRN (the difference-wave measure) during the learning, stronger positive affect and joviality, and less fatigue after the learning, in which the difference between the negative and positive reinforcement groups was smaller. The results indicated that pure avoidance motivation may induce distinct emotional fluctuations, which influence the feedback processing. (shrink)

In lieu of an abstract, here is a brief excerpt of the content:Editor’s NoteJessica HeybachThis final installation of Education and Culture’s special theme issue on Dewey, Data, and Technology coincides with what feels like a technological paradigm shift. As I sat down to write this editor’s note, a former student forwarded me Stephen Marche’s December 6, 2022 piece in The Atlantic titled “The College Essay is Dead” wherein he offers a critique of the humanities as dependent on traditional forms of (...) assignments and assessment (i.e. writing), and their historically persistent inability to sufficiently respond to technology and be in conversation with the technologists. Ultimately Marche argues that the humanists and technologists need each other now more than ever with the development of naturallanguage processing. In November 2022, OpenAI released ChatGPT (GPT stands for generative pre-trained transformer), a large language model chatbot capable of answering any question posed by a user. From “What was John Dewey’s theory of experience?” to chocolate cake recipes, ChatGPT offers users answers to their questions by predicting text (similar to autocorrect). The chatbot can answer fairly sophisticated questions, translate text, and write code, all of which previously used to require a human curator to achieve. Interestingly, ChatGPT was trained through a model of learning reflective of what B.F. Skinner (1938) proposed in The Behavior of Organisms: An Experimental Analysis.1 Specifically, to train ChatGPT, creators utilized reinforcement learning and a system of rewards: “a human ‘labeler’ assigns rewards to various outputs the model generates by ranking them from the best to the worst. Thus, ChatGPT is trained by incorporating human intuition—specifically in terms of preferences in verbal combinations—and learning from it.”2Like all educational technologies, ChatGPT has been met with polarizing reactions—you either love it or hate it. Yet without much debate, educators and educational consultants immediately absorbed the technology as an educational good and started proselytizing. With little regard as to how the technology works, who created it, or why it was created, many educators have become early adopters. What is most curious about educators’ response is how many are not alarmed by the inaccuracy of the text produced by ChatGPT (imagine if calculators gave us wrong answers), or the issues of privacy inherent in becoming a user of AI (imagine if the overhead projector sold our behavioral data to other corporate entities), or the notion of academic integrity and thought itself in the face of AI (imagine if the blackboard became a coauthor). Rather, our addiction to novelty and technological fantasies is on display in this technological paradigm shift. Most universities now have webpages filled with links to resources and have filled their professional development calendars with AI-related webinars related to revising [End Page 1] assignments and assessments and creating syllabus policies for AI-produced materials.Although there is much to debate and decades of scholarship will interrogate ChatGPT in all disciplines, for the purposes of this editor’s note I want to pause and focus on the reactive early adopters of ChatGPT. As a regular observer of the educational Twitter-verse, I’m struck by how quickly this technology has been championed by school leaders and classroom educators. Since educators are users only, rather than the designers of said technology, ChatGPT in the classroom can be seen as the ultimate form of outsourcing available to date. I want to emphasize that ChatGPT is different than the calculator, overhead projector, or blackboard, as it is part of a culture already in motion that is interested in the destruction of the teaching profession writ large. Of course, Dewey would caution us against either/or thinking and the fatalism embedded in my own early reaction to ChatGPT, but I implore educators to consider ChatGPT in light of recent political attacks in P-20 educational settings and the increasing legislative moves to destroy progressive and reconstructive educational aims of education.One educational consultant declared just one month after its release: “#ChatGPT is here to stay and efforts against it are futile.” This consultant goes on to list its usefulness in the following tasks: “generating lesson plans, creating quizzes, providing feedback on student writing, summarizing and paraphrasing texts, personalized learning, grading student work, generating... (shrink)

There is an apparent connection between reinforcement learning and agency. Artificial entities controlled by reinforcement learning algorithms are standardly referred to as agents, and the mainstream view in the psychology and neuroscience of agency is that humans and other animals are reinforcement learners. This article examines this connection, focusing on artificial reinforcement learning systems and assuming that there are various forms of agency. Artificial reinforcement learning systems satisfy plausible conditions for minimal agency, and those which (...) use models of the environment to perform forward search are capable of a form of agency which may reasonably be called action for reasons. (shrink)

Animals routinely adapt to changes in the environment in order to survive. Though reinforcement learning may play a role in such adaptation, it is not clear that it is the only mechanism involved, as it is not well suited to producing rapid, relatively immediate changes in strategies in response to environmental changes. This research proposes that counterfactual reasoning might be an additional mechanism that facilitates change detection. An experiment is conducted in which a task state changes over time and (...) the participants had to detect the changes in order to perform well and gain monetary rewards. A cognitive model is constructed that incorporates reinforcement learning with counterfactual reasoning to help quickly adjust the utility of task strategies in response to changes. The results show that the model can accurately explain human data and that counterfactual reasoning is key to reproducing the various effects observed in this change detection paradigm. (shrink)

How can one build AI systems such that they pursue the goals their designers want them to pursue? This is the alignment problem. Numerous authors have raised concerns that, as research advances and systems become more powerful over time, misalignment might lead to catastrophic outcomes, perhaps even to the extinction or permanent disempowerment of humanity. In this paper, I analyze the severity of this risk based on current instances of misalignment. More specifically, I argue that contemporary large language models and (...) game-playing agents are sometimes misaligned. These cases suggest that misalignment tends to have a variety of features: misalignment can be hard to detect, predict and remedy, it does not depend on a specific architecture or training paradigm, it tends to diminish a system’s usefulness and it is the default outcome of creating AI via machine learning. Subsequently, based on these features, I show that the risk of AI alignment magnifies with respect to more capable systems. Not only might more capable systems cause more harm when misaligned, aligning them should be expected to be more difficult than aligning current AI. (shrink)

In this opinionated review, I draw attention to some of the contributions reinforcement learning can make to questions in the philosophy of mind. In particular, I highlight reinforcement learning's foundational emphasis on the role of reward in agent learning, and canvass two ways in which the framework may advance our understanding of perception and motivation.

This paper investigates a distributed coordination control for a class of high-order uncertain multiagent systems. Under the framework of iterative learning control, a novel fully distributed learning protocol is devised for the coordination problem of MASs including time-varying parameter uncertainties as well as actuator saturations. Meanwhile, the learning updating laws of various parameters are proposed. Utilizing Lyapunov theory and combining with Graph theory, the proposed algorithm can make each follower track a leader completely over a limited time interval even (...) though each follower is without knowing the dynamics of the leader. Moreover, the extension to formation control is made. The validity and feasibility of the algorithm are verified conclusively by two examples. (shrink)

This paper presents a GA-based multi-agent reinforce- ment learning bidding approach (GMARLB) for perform- ing multi-agent reinforcement learning. GMARLB inte- grates reinforcement learning, bidding and genetic algo- rithms. The general idea of our multi-agent systems is as follows: There are a number of individual agents in a team, each agent of the team has two modules: Q module and CQ module. Each agent can select actions to be performed at each step, which are done by the Q module. (...) While the CQ module determines at each step whether the agent should continue or relinquish control. Once an agent relinquishes its control, a new agent is selected by bidding algorithms. We applied GA-based GMARLB to the Backgammon game. The experimental results show GMARLB can achieve a su- perior level of performance in game-playing, outperforming PubEval, while the system uses zero built-in knowledge. (shrink)

This paper considers the consensus control problem of nonlinear spatial-temporal hyperbolic partial difference multiagent systems and parabolic partial difference multiagent systems with time delay. Based on the system’s own fixed topology and the method of generating the desired trajectory by introducing virtual leader, using the consensus tracking error between the agent and the virtual leader agent and neighbor agents in the last iteration, an iterative learning algorithm is proposed. The sufficient condition for the system consensus error to converge (...) along the iterative axis is given. When the iterative learning number k approaches infinity, the consensus error in the sense of the L 2 norm between all agents in the system will converge to zero. Furthermore, simulation results illustrate the effectiveness of the algorithm. (shrink)

Relational database management system is the most popular database system. It is important to maintain data security from information leakage and data corruption. RDBMS can be attacked by an outsider or an insider. It is difficult to detect an insider attack because its patterns are constantly changing and evolving. In this paper, we propose an adaptive database intrusion detection system that can be resistant to potential insider misuse using evolutionary reinforcement learning, which combines reinforcement learning and evolutionary learning. (...) The model consists of two neural networks, an evaluation network and an action network. The action network detects the intrusion, and the evaluation network provides feedback to the detection of the action network. Evolutionary learning is effective for dynamic patterns and atypical patterns, and reinforcement learning enables online learning. Experimental results show that the performance for detecting abnormal queries improves as the proposed model learns the intrusion adaptively using Transaction Processing performance Council-E scenario-based virtual query data. The proposed method achieves the highest performance at 94.86%, and we demonstrate the usefulness of the proposed method by performing 5-fold cross-validation. (shrink)

Sequential action makes up the bulk of human daily activity, and yet much remains unknown about how people learn such actions. In one motor learning paradigm, the serial reaction time (SRT) task, people are taught a consistent sequence of button presses by cueing them with the next target response. However, the SRT task only records keypress response times to a cued target, and thus it cannot reveal the full time‐course of motion, including predictive movements. This paper describes a mouse movement (...) trajectory SRT task in which the cursor must be moved to a cued location. We replicated keypress SRT results, but also found that predictive movement—before the next cue appears—increased during the experiment. Moreover, trajectory analyses revealed that people developed a centering strategy under uncertainty. In a second experiment, we made prediction explicit, no longer cueing targets. Thus, participants had to explore the response alternatives and learn via reinforcement, receiving rewards and penalties for correct and incorrect actions, respectively. Participants were not told whether the sequence of stimuli was deterministic, nor if it would repeat, nor how long it was. Given the difficulty of the task, it is unsurprising that some learners performed poorly. However, many learners performed remarkably well, and some acquired the full 10‐item sequence within 10 repetitions. Comparing the high‐ and low‐performers’ detailed results in this reinforcement learning (RL) task with the first experiment's cued trajectory SRT task, we found similarities between the two tasks, suggesting that the effects in Experiment 1 are due to predictive, rather than reactive processes. Finally, we found that two standard model‐free reinforcement learning models fit the high‐performing participants, while the four low‐performing participants provide better fit with a simple negative recency bias model. (shrink)

After generalizing the Archimedean property of real numbers in such a way as to make it adaptable to non-numeric structures, we demonstrate that the real numbers cannot be used to accurately measure non-Archimedean structures. We argue that, since an agent with Artificial General Intelligence (AGI) should have no problem engaging in tasks that inherently involve non-Archimedean rewards, and since traditional reinforcement learning rewards are real numbers, therefore traditional reinforcement learning probably will not lead to AGI. We indicate two (...) possible ways traditional reinforcement learning could be altered to remove this roadblock. (shrink)

Learning in natural environments is often characterized by a degree of inconsistency from an input. These inconsistencies occur, for example, when learning from more than one source, or when the presence of environmental noise distorts incoming information; as a result, the task faced by the learner becomes ambiguous. In this study, we investigate how learners handle such situations. We focus on the setting where a learner receives and processes a sequence of utterances to master associations between objects and their labels, (...) where the source is inconsistent by design: It uses both “correct” and “incorrect” object‐label pairings. We hypothesize that depending on the order of presentation, the result of the learning may be different. To this end, we consider two types of symbolic learning procedures: the Object‐Label (OL) and the Label‐Object (LO) process. In the OL process, the learner is first exposed to the object, and then the label. In the LO process, this order is reversed. We perform experiments with human subjects, and also construct a computational model that is based on a nonlinear stochastic reinforcement learning algorithm. It is observed experimentally that OL learners are generally better at processing inconsistent input compared to LO learners. We show that the patterns observed in the learning experiments can be reproduced in the simulations if the model includes (a) an ability to regularize the input (and also to do the opposite, i.e., undermatch) and (b) an ability to take account of implicit negative evidence (i.e., interactions among different objects/labels). The model suggests that while both types of learners utilize implicit negative evidence in a similar way, there is a difference in regularization patterns: OL learners regularize the input, whereas LO learners undermatch. As a result, OL learners are able to form a more consistent system of image‐utterance associations, despite the ambiguous learning task. (shrink)

Sequential action makes up the bulk of human daily activity, and yet much remains unknown about how people learn such actions. In one motor learning paradigm, the serial reaction time (SRT) task, people are taught a consistent sequence of button presses by cueing them with the next target response. However, the SRT task only records keypress response times to a cued target, and thus it cannot reveal the full time‐course of motion, including predictive movements. This paper describes a mouse movement (...) trajectory SRT task in which the cursor must be moved to a cued location. We replicated keypress SRT results, but also found that predictive movement—before the next cue appears—increased during the experiment. Moreover, trajectory analyses revealed that people developed a centering strategy under uncertainty. In a second experiment, we made prediction explicit, no longer cueing targets. Thus, participants had to explore the response alternatives and learn via reinforcement, receiving rewards and penalties for correct and incorrect actions, respectively. Participants were not told whether the sequence of stimuli was deterministic, nor if it would repeat, nor how long it was. Given the difficulty of the task, it is unsurprising that some learners performed poorly. However, many learners performed remarkably well, and some acquired the full 10‐item sequence within 10 repetitions. Comparing the high‐ and low‐performers’ detailed results in this reinforcement learning (RL) task with the first experiment's cued trajectory SRT task, we found similarities between the two tasks, suggesting that the effects in Experiment 1 are due to predictive, rather than reactive processes. Finally, we found that two standard model‐free reinforcement learning models fit the high‐performing participants, while the four low‐performing participants provide better fit with a simple negative recency bias model. (shrink)

Reinforcement Learning is a promising technique for creating agents that can be applied to real world problems. The most important features of RL are trial-and-error search and delayed reward. Thus, agents randomly act in the early learning stage. However, such random actions are impractical for real world problems. This paper presents a novel model of RL agents. A feature of our learning agent model is to integrate the Analytic Hierarchy Process into the standard RL agent model, which consists (...) of three modules: state recognition, learning, and action selecting modules. In our model, the AHP module is designed with {\\it primary knowledge} that humans intrinsically have in a process until a goal state is attained. This integration aims at increasing promising actions instead of completely random actions in the standard RL algorithms. Profit Sharing is adopted as a RL method for our model, since PS is known to be useful even in multi-agent environments. To evaluate our approach in a multi-agent environment, we test a PS RL method with our agent model on a pursuit problem in a grid world. Computational results show that our approach outperforms the standard PS in terms of learning speed in the earlier stages of learning. We also show that the learning performance of our approach is superior at least competitive to that of the standard one in the final stages of learning. (shrink)

Dual-systems frameworks propose that moral judgments are derived from both an immediate emotional response, and controlled/rational cognition. Recently Cushman (2013) proposed a new dual-system theory based on model-free and model-based reinforcement learning. Model-free learning attaches values to actions based on their history of reward and punishment, and explains some deontological, non-utilitarian judgments. Model-based learning involves the construction of a causal model of the world and allows for far-sighted planning; this form of learning fits well with utilitarian considerations that (...) seek to maximize certain kinds of outcomes. I present three concerns regarding the use of model-free reinforcement learning to explain deontological moral judgment. First, many actions that humans find aversive from model-free learning are not judged to be morally wrong. Moral judgment must require something in addition to model-free learning. Second, there is a dearth of evidence for central predictions of the reinforcement account—e.g., that people with different reinforcement histories will, all else equal, make different moral judgments. Finally, to account for the effect of intention within the framework requires certain assumptions which lack support. These challenges are reasonable foci for future empirical/theoretical work on the model-free/model-based framework. (shrink)

This work focuses on a multiagent system with time-varying interval uncertainty in the system matrix, where multiple agents interact through an undirected topology graph and only the bounding matrices on the uncertainty in the system matrix are known. A reduced-order interval observer, which is named the reduced-order neighborhood interval observer, is designed to estimate the relative state of each agent and those of its neighbors. It is shown that the reduced-order IO can guarantee the consensus of the uncertain (...) class='Hi'>multiagent system. Finally, simulation examples are proposed to verify the theoretical findings. (shrink)

Upshot: We acknowledge that our model can be implemented with different reinforcement learning algorithms. Subsystem formation has been successfully demonstrated on the basal level, and in order to show full subsystem formation in the communication system at least both intentional utterances and acceptance/rejection need to be implemented. The comments about intrinsic vs extrinsic rewards made clear that this distinction is not helpful in the context of the constructivist paradigm but rather needs to be replaced by a critical reflection on (...) whether one has truly created autopoietic agents or just an engineering system. (shrink)