An improved immune Q-learning algorithm

Currently many non-tractable considered problems have been solved satisfactorily through methods of approximate optimization called metaheuristic. These methods use non-deterministic approaches that find good solutions which, however, do not guarantee the determination of the global optimum. The success of a metaheuristic is conditioned by capacity to adequately alternate between exploration and exploitation of the solution space. A way to guide such algorithms while searching for better solutions is supplying them with more knowledge of the solution space (environment of the problem). This can to be made in terms of a mapping of such environment in states and actions using Reinforcement Learning. This paper proposes the use of a technique of Reinforcement Learning - Q-Learning Algorithm - for the constructive phase of GRASP and Reactive GRASP metaheuristic. The proposed methods will be applied to the symmetrical traveling salesman problem.

Contemporary behavioral learning theory provides a comprehensive description of how we and other animals learn, and places behavioral flexibility and automaticity at heart of adaptive behaviors. However, to our knowledge, the computations supporting the interactions between deliberative and habitual decision-making systems are still poorly understood. Previous functional magnetic resonance imaging (fMRI) results suggest that the dorsal striatum host complementary computations that may differentially support deliberative and habitual processes [1] in the form of a dynamical interplay rather than a serial recruitment of strategies. From the same instrumental task, we develop a dual-system computational model of the two systems that can predict both performance (i.e., participant choices) and modulations in reaction times during learning. The instrumental task is a trial-and-error learning task requiring participants to find the correct associations between color stimuli and finger responses. To model the habitual system, we use a simple Q-learning algorithm (QL) [2] whose properties are fast responses, but slow convergence. For the deliberative (i.e goal-directed) system, we propose a new Bayesian Working Memory (BWM) which searches for information in the history of previous trials and stops as soon as the uncertainty on the action to perform decreases below a certain threshold. Last, we also propose a model for QL and BWM coordination. Currently, most models of system selection tend to control action selection concurrently, using either the deliberative or habitual model according to uncertainty criteria [3,4]. Only one model has investigated the relation between working memory and reinforcement learning [5] without, however explicitly modeling the temporal aspect of memory manipulation. In our approach, we propose a model for QL and BWM coordination. QL and BWM are merged such that the expensive memory manipulation is under control of, among others, the level of convergence of the habitual learning. Consequently, we also predict specific reaction times for each model that can be compared with the evolution of reaction times in instrumental learning tasks. Models are optimized for each subject with the NSGA-2 multi-objective evolutionary algorithm. The first fitness function is the Bayesian Information Criterion for individual choices. The second fitness function is also a likelihood that maximizes the probability of performing reaction times similar to humans. We compare the ability of the new model to explain human behavior with the QL or BWM only, as well as with a combination of these models based on [4], which reveals that the proposed model is in general more accurate. To conclude, we suggest that a close combination of BWM and QL better explains both choices and reaction times for most participants.

In this paper, path planning and sorting of packages for Omnidirectional-Wheel conveyor are presented using Reinforcement Learning (RL). Q-learning, Double Q-learning, Deep Q-learning, and the Double Deep Q-learning algorithms are investigated. The RL algorithms enable the conveyor to self-learn the packages path and sort them without using conventional control or path planning theories. The RL algorithms are used for two different case studies on conveyors structures with different numbers of cells to compare and evaluate their performances in large- and small-scale sized structures. To explore the proposed methods response to external environment effects, two types of collisions between multiple packages were considered, the proposed RL algorithms showed their ability to resolve both types successfully. Comparative study between multiple RL algorithms for path planning showed that the Q-learning and Double Q-learning algorithms had outperformed their Deep learning versions for path planning in the two case studies. Furthermore, the proposed RL methods are compared experimentally to classic control and path planning theories using a hardware prototype for one of the presented case studies. The hardware experimental results showed that the proposed RL methods were as successful as the conventional methods in path planning and sorting in much less processing time. Two types of sorting scenarios (Type I and II) were tested for same package type and for multiple ones. For Type I sorting the Q-learning algorithm performed better than the Q-learning with weights approach, achieving better mean and minimum rewards while maximum rewards remain the same for both techniques. As for Type II sorting, only the Q-learning with weights approach was able to achieve it and converge in a reasonable time.

Q-learning solution for optimal consensus control of discrete-time multiagent systems using reinforcement learning

Contributors

At present, APT attack detection mainly focuses on the specific realization of detection technology, and there is little research on the timing strategy for active detection of APT attack. This paper focuses on the problem of APT dynamic detection with timing decision intelligence. We use the FlipIt game model and reinforcement learning method of Q-learning algorithm, consider the APT attack events as an exponentially distributed random process, and the defender using the attacker’s last move time (LM-defender) to learn his optimal timing strategy using Q-Learning algorithm. The defender's reinforcement learning mechanism for the timing of the APT attack action and outputs its timing selection strategy. Simulation experiments show that compared with the existing greedy algorithm, the timing strategy based on the Q-Learning algorithm does not require the attacker's prior information under the premise, it can achieve a higher average benefit than the greedy strategy. This result provides a theoretical reference for the design of timing strategy to defend APT attack intelligently and efficient.

The problem of path exploration and planning of agents in unknown environments is a popular application problem in the field of reinforcement learning. In this paper, we propose an improved reinforcement learning algorithm called the Q-Learning algorithm for adaptive exploration based on simulated annealing (AE-SAQL). We apply the algorithm to the agent path planning problem, improve the setting of the reward function and add the feedback information of the environment. By simulating the Metropolis criterion in the annealing algorithm and adding an adaptive adjustment mechanism, the agent fully explores the environment and makes full use of the environmental information, solving the exploration-utilization dilemma during the algorithm and finally enabling the agent to reach the target location safely. Compared with the standard Q-Learning algorithm and SARSA algorithm, AE-SAQL achieves better.

Reinforcement learning approach for resource allocation in humanitarian logistics

Mobile ad hoc networks (MANETs) consist of a set of nodes which can move freely and communicate with each other wirelessly. Due to the movement of nodes and unlike wired networks, the available routes used among the nodes for transmitting data packets are not stable. Hence, proposing real-time routing protocols for MANETs is regarded as one of the major challenges in this research domain. Algorithms compatible with the changes created in the network due to the nodes' movements are of high significance. For reducing data packet transmission time among nodes, not only should route shortness be considered but also route stability should be taken into consideration. Since available factors in different environments have specific behavior patterns especially in human environments, the parameters of link stability and route shortness were taken into consideration and the reinforcement learning was used to propose a method so as to make the best choice among the neighbors at any moment to transmit a packet to the destination. That is, the proposed method was aimed at predicting the behavior pattern of the nodes in relation to the target node through using reinforcement learning. The proposed method used Q-learning algorithm which has more homogeneity to estimate the value of actions. Simulation results in OPNET demonstrate the superiority of the proposed scheme over conventional MANET routing methods.

The article considers the solution of the traveling salesman problem using reinforcement learning. This task is to find the optimal route that passes through the specified cities at least once and then returns to the original city. The traveling salesman problem belongs to the class of NP-hard problems, and therefore finding an exact solution for sufficiently large data sets using classical mathematical methods turns out to be impossible in a reasonable time. The aim of the work is to increase the efficiency and speed of solving the traveling salesman problem with a large number of vertices by using the reinforcement learning method. A variant of solving the traveling salesman problem using the Q-learning algorithm is considered, which allows the agent to learn the optimal behavior strategy in the process of interacting with the environment, which is a set of cities and roads between them, which are modeled as a graph. Within the framework of this study, a software package was developed to solve the traveling salesman problem using reinforcement learning. A series of experiments related to the appearance of unforeseen conditions complicating the work of a traveling salesman, such as the appearance of "problem" zones or zones, the passage through which gives a reduction in time. The results of the experiments showed the capabilities of the developed algorithm for adaptation to changes and obstacles of the environment. The resulting algorithm shows itself to be more stable and reliable compared to the algorithms obtained by classical solution methods. The solution of this problem can be used in practice in a variety of real situations, including optimization of deliveries or network routing, assistance in the work of logistics departments of various transport, postal and courier services.

Techniques of optimization, known as metaheuristics, have achieved success in the resolution of many problems classified as NP-hard. These methods use non-deterministic approaches that find good solutions which, however, do not guarantee the determination of the global optimum. Beyond the inherent difficulties related to the complexity that characterizes the optimization problems, the metaheuristics still face the dilemma of the exploitation -exploration, which consists of choosing between a greedy search and a wider exploration of the solution space. A way to guide such algorithms during the search of better solutions is supplying them with more knowledge through the learning of the environment. This way, this work proposes the use of a technique of Reinforcement Learning -Q-Learning Algorithm -for the constructive phase of GRASP metaheuristic and to generate the initial population of a Genetic Algorithm. The proposed methods will be applied to the symmetrical traveling salesman problem.

Explaining what a reinforcement learning (RL) control agent learns play a crucial role in the safety critical control domain. Most of the approaches in the state-of-the-art focused on imitation learning methods that uncover the hidden reward function of a given control policy. However, these approaches do not uncover what the RL agent learns effectively from the agent-environment interaction. The policy learned by the RL agent depends in how good the state transition mapping is inferred from the data. When the state transition mapping is wrongly inferred implies that the RL agent is not learning properly. This can compromise the safety of the surrounding environment and the agent itself. In this paper, we aim to uncover the elements learned by data-driven RL control agents in a special class of discrete-time linear autonomous systems. Here, the approach aims to add a new explainable dimension to data-driven control approaches to increase their trust and safe deployment. We focus on the classical data-driven Q-learning algorithm and propose an explainable Q-learning (XQL) algorithm that can be further expanded to other data-driven RL control agents. Simulation experiments are conducted to observe the effectiveness of the proposed approach under different scenarios using several discrete-time models of autonomous platforms.

This paper introduces an approach to Reinforcement Learning Algorithm by introducing reduct concept of rough set methodology using a variation of Q-Learning algorithm. Unlike the conventional Q-Learning, the proposed algorithm calculates the reduct from look up table of previous episodes. In modified algorithm first action selection of an agent will based upon the reduct of previous episode. In Q-learning algorithm an agent makes action selections in an effort to maximize a return obtained from the environment. Agent will changes its policy for future actions based upon return. The problem considered in this paper is convergence of Q-value that takes more episodes in conventional Q-learning. The solution to this problem results from a new form Q-learning algorithm by calculating the reduct of previous episodes. The framework provided by a reduct based Q-Learning algorithm in order to simplify the learned moves and interested in eliminating conditional moves of an Agent. Effectiveness of proposed algorithm is simulated in a 10 x 10 Grid world deterministic environment and the result for the two forms of Q-Learning Algorithms is given.

Contributors

With the accelerated development of smart cities, the concept of a “smart industrial park” in which unmanned ground vehicles (UGVs) have wide application has entered the industrial field of vision. When faced with multiple tasks and heterogeneous tasks, the task execution efficiency of a single UGV is inefficient, thus the task planning research under multi-UGV cooperation has become more urgent. In this paper, under the anti-collision cooperation mechanism for multi-UGV path planning, an improved algorithm with optimized-weighted-speedy Q-learning (OWS Q-learning) is proposed. The slow convergence speed of the Q-learning algorithm is overcome to a certain extent by changing the update mode of the Q function. By improving the selection mode of learning rate and the selection strategy of action, the relationship between exploration and utilization is balanced, and the learning efficiency of multi-agent in complex environments is improved. The simulation experiments in static environment show that the designed anti-collision coordination mechanism effectively solves the coordination problem of multiple UGVs in the same scenario. In the same experimental scenario, compared with the Q-learning algorithm and other reinforcement learning algorithms, only the OWS Q-learning algorithm achieves the convergence effect, and the OWS Q-learning algorithm has the shortest collision-free path for UGVS and the least time to complete the planning. Compared with the Q-learning algorithm, the calculation time of the OWS Q-learning algorithm in the three experimental scenarios is improved by 53.93%, 67.21%, and 53.53%, respectively. This effectively improves the intelligent development of UGV in smart parks.