Multi-Agent Evolutionary Reinforcement Learning Based on Cooperative Games

Reinforcement learning (RL) has shown promising performance in tackling robotic manipulation tasks (RMTs), which require learning a prolonged sequence of manipulation actions to control robots efficiently. However, most RL algorithms often suffer from two problems when solving RMTs: inefficient exploration due to the extremely large action space and catastrophic forgetting due to the poor sampling efficiency. To alleviate these problems, this paper introduces an Evolutionary Reinforcement Learning algorithm with parameterized Action Primitives, called ERLAP, which combines the advantages of an evolutionary algorithm (EA) and hierarchical RL (HRL) to solve diverse RMTs. A library of heterogeneous action primitives is constructed in HRL to enhance the exploration efficiency of robots and dual populations with new evolutionary operators are run in EA to optimize these primitive sequences, which can diversify the distribution of replay buffer and avoid catastrophic forgetting. The experiments show that ERLAP outperforms four state-of-the-art RL algorithms in simulated RMTs with dense rewards and can effectively avoid catastrophic forgetting in a set of more challenging simulated RMTs with sparse rewards.

Designing novel biological sequences with desired properties is a significant challenge in biological science because of the extra large search space. The traditional design process usually involves multiple rounds of costly wet lab evaluations. To reduce the need for expensive wet lab experiments, machine learning methods are used to aid in designing biological sequences. However, the limited availability of biological sequences with known properties hinders the training of machine learning models, significantly restricting their applicability and performance. To fill this gap, we present ERLBioSeq, an Evolutionary Reinforcement Learning algorithm for BIOlogical SEQuence design. ERLBioSeq leverages the capability of reinforcement learning to learn without prior knowledge and the potential of evolutionary algorithms to enhance the exploration of reinforcement learning in the large search space of biological sequences. Additionally, to enhance the efficiency of biological sequence design, we developed a predictor for sequence screening in the biological sequence design process, which incorporates both the local and global sequence information. We evaluated the proposed method on three main types of biological sequence design tasks, including the design of DNA, RNA, and protein. The results demonstrate that the proposed method achieves significant improvement compared to the existing state-of-the-art methods.

Designing the fuzzy controllers by using evolutionary algorithms and reinforcement learning is an important subject to control the robots. In the present article, some methods to solve reinforcement fuzzy control problems are studied. All these methods have been established by combining Fuzzy-Q Learning with an optimization algorithm. These algorithms include the Ant colony, Bee Colony and Artificial Bee Colony optimization algorithms. Comparing these algorithms on solving Track Backer-Upper problem –a reinforcement fuzzy control problem– shows that Artificial Bee Colony Optimization algorithm has the best efficiency in combining with fuzzy- Q Learning.

Many real-world problems involve cooperation and/or competition among multiple agents. These problems often can be formulated as multi-agent problems. Recently, Reinforcement Learning (RL) has made significant progress on single-agent problems. However, multi-agent problems still cannot be easily solved by traditional RL algorithms. First, the multi-agent environment is considered as a non-stationary system. Second, most multi-agent environments only provide a shared team reward as feedback. As a result, agents may not be able to learn proper cooperative or competitive behaviors by traditional RL. Our algorithm adopts Evolution Strategies (ES) for optimizing policy which is used to control agents and a value decomposition method for estimating proper fitness for each policy. Evolutionary Algorithm is considered as a promising alternative for signal-agent problems. Owing to its simplicity, scalability, and efficiency on zeroth-order optimization, EAs can even outperform RLs on some tasks. In order to solve multi-agent problems by EA, a value decomposition method is used to decompose the team reward. Our method is parallel on multiple cores, which can speed up our algorithm significantly. We test our algorithm on two benchmarking environments, and the experiment results show that our algorithm is better than traditional RL and other representative gradient-free methods.

Computerized adaptive testing (CAT) aims to present items that statistically optimize the assessment process by considering the examinee's responses and estimated trait levels. Recent developments in reinforcement learning and deep neural networks provide CAT with the potential to select items that utilize more information across all the items on the remaining tests, rather than just focusing on the next several items to be selected. In this study, we reformulate CAT under the reinforcement learning framework and propose a new item selection strategy based on the deep Q-network (DQN) method. Through simulated and empirical studies, we demonstrate how to monitor the training process to obtain the optimal Q-networks, and we compare the accuracy of the DQN-based item selection strategy with that of five traditional strategies-maximum Fisher information, Fisher information weighted by likelihood, Kullback‒Leibler information weighted by likelihood, maximum posterior weighted information, and maximum expected information-on both simulated and real item banks and responses. We further investigate how sample size and the distribution of the trait levels of the examinees used in training affect DQN performance. The results show that DQN achieves lower RMSE and MAE values than traditional strategies under simulated and real banks and responses in most conditions. Suggestions for the use of DQN-based strategies are provided, as well as their code.

The developments in reinforcement learning provide a powerful and efficient learning framework for autonomous robotic systems. However, prior works rarely embed historical observations due to the exponentially increasing complexity, which may not perform well for large-scale long-horizon tasks that might require hundreds and thousands of steps to complete. In this paper, we propose Recurrent Imitation and Reinforcement Learning (RIRL) to address the challenges and enable robots for such tasks. The proposed RIRL incorporates a long short-term memory (LSTM) network to retain long-term memories, which could be an effective and efficient method to tackle the long dependency problem raised in long-horizon robotic tasks. To assess the performance of the RIRL, we test it with an optimized path planning problem for a robot to perform a Radiofrequency identification (RFID) inventory in dynamic and previously unknown environments. We experimentally validate RIRL’s feasibility and effectiveness in a visual game-based simulation platform, where the proposed RIRL model outperforms three baseline schemes with considerable gains.

Energy harvesting technology enables wireless sensor networks (WSNs) to be self-sustainable, for maintaining long-term key performance indicators, such as the data throughput and sensing coverage. Due to the highly dynamic and complex environment, energy sources (ES) cannot provide stable energy supply, which needs the efficient learning algorithm to enable system adaptations. This article reports on the development of reinforcement learning (RL) methodology to long-term data collection in self-sustainable WSNs. Specifically, we consider the WSN as a 2-D rectangular region, where a mobile sensor (MS) can harvest energy from ambient environments while transmitting the collected data to a fixed sink. Due to the changing environment and the mobility of the MS, the harvested energy by the MS at each slot presents spatiotemporal dynamics within the network, which severely affects the performance of data throughput from the MS to the sink. The MS's trajectory is investigated to maximize the long-term average MS-to-sink data throughput. Due to the unknown energy arrival information as well as the locations of ESs, we formulate the problem as a Markov decision process, which is then solved with model-free RL. In particular, the deep deterministic policy gradient (DDPG) is applied to tackle the continuous and deterministic movement space. Results show that the MS can learn and optimize the moving trajectory by intelligently tracking the aggregated received energy over slots. Finally, the MS can identify and move to the optimal location where the maximized long-term average MS-to-sink data throughput is achieved. Extensive numerical evaluations are conducted to investigate the impact of various system parameters on the network performance.

In this paper, two universal reinforcement learning methods are considered to solve the problem of maximum power point tracking for photovoltaics. Both methods exhibit fast achievement of the MPP under varying environmental conditions and are applicable in different PV systems. The only required knowledge of the PV system are the open-circuit voltage, the short-circuit current and the maximum power, all under STC, which are always provided by the manufacturer. Both methods are compared to a Fuzzy Logic Controller and the universality of the proposed methods is highlighted. After the implementation and the validation of proper performance of both methods, two evolutionary optimization algorithms (Big Bang—Big Crunch and Genetic Algorithm) are applied. The results demonstrate that both methods achieve higher energy production and in both methods the time for tracking the MPP is reduced, after the application of both evolutionary algorithms.

The integration of Evolutionary Algorithm (EA) and Reinforcement Learning (RL) has emerged as a promising approach for tackling some challenges in RL, such as sparse rewards, lack of exploration, and brittle convergence properties. However, existing methods often employ actor networks as individuals of EA, which may constrain their exploratory capabilities, as the entire actor population will stop evolution when the critic network in RL falls into local optimal. To alleviate this issue, this paper introduces a Two-stage Evolutionary Reinforcement Learning (TERL) framework that maintains a population containing both actor and critic networks. TERL divides the learning process into two stages. In the initial stage, individuals independently learn actor-critic networks, which are optimized alternatively by RL and Particle Swarm Optimization (PSO). This dual optimization fosters greater exploration, curbing susceptibility to local optima. Shared information from a common replay buffer and PSO algorithm substantially mitigates the computational load of training multiple agents. In the subsequent stage, TERL shifts to a refined exploitation phase. Here, only the best individual undergoes further refinement, while the rest individuals continue PSO-based optimization. This allocates more computational resources to the best individual for yielding superior performance. Empirical assessments, conducted across a range of continuous control problems, validate the efficacy of the proposed TERL paradigm.

Neurocontrol of a multi-effect batch distillation pilot plant based on evolutionary reinforcement learning

Online parameter controllers for evolutionary algorithms adjust values of parameters during the run. Recently, a new efficient parameter controller based on reinforcement learning was proposed by Karafotias et al. In this method parameter ranges are discretized into several intervals before the run. However, performing adaptive discretization during the run may increase efficiency of an evolutionary algorithm. Aleti et al. proposed another efficient controller with adaptive discretization. In this paper we propose a parameter controller based on reinforcement learning with adaptive discretization. The proposed controller is compared with the existing parameter adjusting methods on different configurations of an evolutionary algorithm. Results show that the new controller outperforms the other controllers on most of the considered test problems.

While teams can draw upon diverse perspectives of their members, the presence of contrasting views entail the risk that creative ideas are rejected during collaboration, without being considered for closer evaluation. Therein, organizations frequently employ rewards with the intention to motivating members and cultivating creativity in the team. Drawing on the Motivated Information Processing in Groups Model (MIP-G) we assess how varying reward structures influence the dissemination and integration of information during the collective idea generation process. In line with the MIP - G, we first posit that individual rewards foster proself motivation which in turn increases idea rejection, whereas team rewards emphasize prosocial motivation which decreases idea rejection. Further we postulate that mixed rewards combine proself and prosocial motivation. which is why we presume that these leads to greater idea rejection as compared to team rewards, but less rejection as compared to individual rewards. Based on an experimental study with 81 teams involving 293 participants, we demonstrate that individual rewards increase idea rejection. Contrary to our expectations, however, team rewards also significantly heighten idea rejection, whereas mixed rewards tend to reduce it. Thus, our research offers a new perspective on the motivated information processing in teams. Herein, it highlights that the combination of both proself, and prosocial motives is beneficial for the idea generation process in teams, whereas their separate consideration increases the tendency of team members to reject each other’s ideas.

Reinforcement learning (RL) has proven to be a powerful paradigm for deriving complex behaviors from simple reward signals in a wide range of environments. When applying RL to continuous control agents in simulated physics environments, the body is usually considered to be part of the environment. However, during evolution the physical body of biological organisms and their controlling brains are co-evolved, thus exploring a much larger space of actuator/controller configurations. Put differently, the intelligence does not reside only in the agent's mind, but also in the design of their body. We propose a method for uncovering strong agents, consisting of a good combination of a body and policy, based on combining RL with an evolutionary procedure. Given the resulting agent, we also propose an approach for identifying the body changes that contributed the most to the agent performance. We use the Shapley value from cooperative game theory to find the fair contribution of individual components, taking into account synergies between components. We evaluate our methods in an environment similar to the the recently proposed Robo-Sumo task, where agents in a software physics simulator compete in tipping over their opponent or pushing them out of the arena. Our results show that the proposed methods are indeed capable of generating strong agents, significantly outperforming baselines that focus on optimizing the agent policy alone. A video is available at: https://youtu.be/CHlecRim9PI

Scenario-based multi-period program optimization for capability-based planning using evolutionary algorithms

In multi-agent robotic systems, the overlap of actions selected by each agent results in poor cooperation, while, at the same time, conventional reinforcement learning entails a large computational cost because every agent must learn. In this paper, we propose a novel method to solve these problems. The agent uses a rule set to conduct its behavior. Our system consists of reinforcement learning with an Action Selection Priority Level (ASPL) module and a generalized rules module. Using the ASPL, the reinforcement learning module chooses a proper cooperative behavior while the generalized rule module can accelerate the learning process. By applying the proposed method to robot soccer, the learning process can be accelerated by reducing the search space. The results of simulation and real experiments indicate the effectiveness of the proposed method.