Generating intelligent agent behaviors in multi-agent game AI using deep reinforcement learning algorithm

Path planning refers to that the mobile robot can obtain the surrounding environment information and its own state information through the sensor carried by itself, which can avoid obstacles and move towards the target point. Deep reinforcement learning consists of two parts: reinforcement learning and deep learning, mainly used to deal with perception and decision-making problems, has become an important research branch in the field of artificial intelligence. This paper first introduces the basic knowledge of deep learning and reinforcement learning. Then, the research status of deep reinforcement learning algorithm based on value function and strategy gradient in path planning is described, and the application research of deep reinforcement learning in computer game, video game and autonomous navigation is described. Finally, I made a brief summary and outlook on the algorithms and applications of deep reinforcement learning.

Reinforcement Learning (RL) has become one of the most powerful methods in Artificial Intelligence (AI) which allows a system to learn as it is engaged in an interaction with its environment and to maximize its actions by performing a trial and error process. Although classical RL algorithms can give us a fundamental idea on RL, like the Q-Learning, its usefulness wanes in high-dimensional or continuous-state spaces. To overcome these shortcomings, Deep Reinforcement Learning (DRL) algorithms including Deep Q-Networks (DQN) build on neural networks to learn action-value functions, allowing horizontal scalability and more efficient learning. This paper reports on the experimental comparison of Q-Learning and DQN based on the environment CartPole-v1. Both of the algorithms were trained in the controlled environment and tested under various metrics that include cumulative rewards, success rate, convergence speed, safety (failure rate), and path efficiency. Findings indicate that Q-Learning moves towards moderate stability with slower convergence but DQN takes shorter learning time, is more reliable and performs better. Such results indicate the ground-breaking potential of the DRL in the progress of AI toward transcending classical RL restrictions. Future research topics encompass implementation of more challenging experimental settings, a test of other DRL algorithms (e.g., PPO, SAC), and generalization, scalability and safety in real-world environments.

Problem definition: The beer game is widely used in supply chain management classes to demonstrate the bullwhip effect and the importance of supply chain coordination. The game is a decentralized, multiagent, cooperative problem that can be modeled as a serial supply chain network in which agents choose order quantities while cooperatively attempting to minimize the network’s total cost, although each agent only observes local information. Academic/practical relevance: Under some conditions, a base-stock replenishment policy is optimal. However, in a decentralized supply chain in which some agents act irrationally, there is no known optimal policy for an agent wishing to act optimally. Methodology: We propose a deep reinforcement learning (RL) algorithm to play the beer game. Our algorithm makes no assumptions about costs or other settings. As with any deep RL algorithm, training is computationally intensive, but once trained, the algorithm executes in real time. We propose a transfer-learning approach so that training performed for one agent can be adapted quickly for other agents and settings. Results: When playing with teammates who follow a base-stock policy, our algorithm obtains near-optimal order quantities. More important, it performs significantly better than a base-stock policy when other agents use a more realistic model of human ordering behavior. We observe similar results using a real-world data set. Sensitivity analysis shows that a trained model is robust to changes in the cost coefficients. Finally, applying transfer learning reduces the training time by one order of magnitude. Managerial implications: This paper shows how artificial intelligence can be applied to inventory optimization. Our approach can be extended to other supply chain optimization problems, especially those in which supply chain partners act in irrational or unpredictable ways. Our RL agent has been integrated into a new online beer game, which has been played more than 17,000 times by more than 4,000 people.

Reinforcement learning, as a branch of machine learning, has been gradually applied in the control field. However, in the practical application of the algorithm, the hyperparametric approach to network settings for deep reinforcement learning still follows the empirical attempts of traditional machine learning (supervised learning and unsupervised learning). This method ignores part of the information generated by agents exploring the environment contained in the updating of the reinforcement learning value function, which will affect the performance of the convergence and cumulative return of reinforcement learning. The reinforcement learning algorithm based on dynamic parameter adjustment is a new method for setting learning rate parameters of deep reinforcement learning. Based on the traditional method of setting parameters for reinforcement learning, this method analyzes the advantages of different learning rates at different stages of reinforcement learning and dynamically adjusts the learning rates in combination with the temporal-difference (TD) error values to achieve the advantages of different learning rates in different stages to improve the rationality of the algorithm in practical application. At the same time, by combining the Robbins–Monro approximation algorithm and deep reinforcement learning algorithm, it is proved that the algorithm of dynamic regulation learning rate can theoretically meet the convergence requirements of the intelligent control algorithm. In the experiment, the effect of this method is analyzed through the continuous control scenario in the standard experimental environment of ”Car-on-The-Hill” of reinforcement learning, and it is verified that the new method can achieve better results than the traditional reinforcement learning in practical application. According to the model characteristics of the deep reinforcement learning, a more suitable setting method for the learning rate of the deep reinforcement learning network proposed. At the same time, the feasibility of the method has been proved both in theory and in the application. Therefore, the method of setting the learning rate parameter is worthy of further development and research.

Using Deep Reinforcement Learning (DRL) algorithm to deal with autonomous driving tasks usually have unsatisfied performance due to lack of robustness and means to escape local optimum. In this article, we designs a Survival-Oriented Reinforcement Learning (SORL) model that tackle these problems by setting survival rather than maximize total reward as first priority. In SORL model, we model autonomous driving task as Constrained Markov Decision Process (CMDP) and introduce Negative-Avoidance Function to learn from previous failure. The SORL model greatly speed up the training process and improve the robustness of normal Deep Reinforcement Learning algorithm.

Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system

Deep Reinforcement Learning (DRL) algorithm provides an effective and powerful way for the computer learning to make decisions according to what the computer sees, the most famous application of DRL is training the computer to play computer games, such as the computer Go and Atari Games. In this paper, we present that the DRL can used to improve the classical PID controller by training an adaptive PID controller with its parameters change according to the changing state, we name our new controller as DRPID, the deep reinforcement learning algorithm that we used to train the PID controller in this paper is deep deterministic policy gradient, which is very suitable to solve continue action control problem. We test our DRPID algorithm by training it to control an inverted pendulum in the OpenAI gym simulation environment, and we found DRPID controller works very well, outperforms the common PID controller with fixed parameters, by a great margin.

Much human and computational effort has aimed to improve how deep reinforcement learning (DRL) algorithms perform on benchmarks such as the Atari Learning Environment. Comparatively less effort has focused on understanding what has been learned by such methods, and investigating and comparing the representations learned by different families of DRL algorithms. Sources of friction include the onerous computational requirements, and general logistical and architectural complications for running DRL algorithms at scale. We lessen this friction, by (1) training several algorithms at scale and releasing trained models, (2) integrating with a previous DRL model release, and (3) releasing code that makes it easy for anyone to load, visualize, and analyze such models. This paper introduces the Atari Zoo framework, which contains models trained across benchmark Atari games, in an easy-to-use format, as well as code that implements common modes of analysis and connects such models to a popular neural network visualization library. Further, to demonstrate the potential of this dataset and software package, we show initial quantitative and qualitative comparisons between the performance and representations of several DRL algorithms, highlighting interesting and previously unknown distinctions between them.

The online 3D bin packing problem(3D-BPP) is widely used in the logistics industry and is of great practical significance for promoting the intelligent transformation of the industry. The heuristic algorithm relies too much on manual experience to formulate more perfect packing rules. In recent years, many scholars solve 3D-BPP via deep reinforcement learning(DRL) algorithms. However, they ignore many skills used in manual packing, one of the most important skill is workers put the item aside if the item is packed improperly. Inspired by this skill, we propose a DRL algorithm with a buffer zone. Firstly, we define the wasted space and the buffer zone. And then, we integrate them into the DRL algorithm framework. Importantly, we compare the bin utilization with di erent thresholds of wasted space and di erent buffer zone sizes. Experimental results show that our algorithm outperforms existing heuristic algorithms and DRL algorithms.

As vehicular technology advances, intelligent vehicles generate numerous computation‐intensive tasks, challenging the computational resources of both the vehicles and the Internet of Vehicles (IoV). Traditional IoV struggles with fixed network structures and limited scalability, unable to meet the growing computational demands and next‐generation mobile communication technologies. In congested areas, near‐end Mobile Edge Computing (MEC) resources are often overtaxed, while far‐end MEC servers are underused, resulting in poor service quality. A novel network framework utilizing sixth‐generation mobile communication (6G) and digital twin technologies, combined with task migration, promises to alleviate these inefficiencies. To address these challenges, a task migration and re‐offloading model based on task attribute classification is introduced, employing a hybrid deep reinforcement learning (DRL) algorithm—Dueling Double Q Network DDPG (QDPG). This algorithm merges the strengths of the Deep Deterministic Policy Gradient (DDPG) and the Dueling Double Deep Q‐Network (D3QN), effectively handling continuous and discrete action domains to optimize task migration and re‐offloading in IoV. The inclusion of the Mini Batch K‐Means algorithm enhances learning efficiency and optimization in the DRL algorithm. Experimental results show that QDPG significantly boosts task efficiency and computational performance, providing a robust solution for resource allocation in IoV.

The continuously changing nature of stock markets poses a non-trivial challenge to build automated trading agents that timely assist traders in their decision makings. Fixed sets of trading rules and offline pre-trained models are inefficient to adapt regarding the real-time fluctuations on the stock markets. Deep Reinforcement Learning (DRL) algorithms train autonomous agents that could potentially address such highly dynamic environments by integrating the generalization power of artificial neural networks with the online learning through experiences gained in the interactions with the environment. We investigate the capability of three DRL algorithms, namely Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG (TD3), and Soft Actor-Critic (SAC), for tackling the automated stock trading problem. We employ two datasets, from the US and Vietnam markets, with different characteristics and market trends, on which the DRL-trained agents are compared against a random trading agent. Experimental results indicate the potentials of these DRL algorithms and also exhibit their pitfalls when being applied in datasets where the majority of the stocks are not up-trending. We propose a simple, but effective, technique to assist the agents to minimize their losses.

With the vigorous development of computer-related technology, the “perception + decision” paradigm of the combination of deep learning and reinforcement learning has become a research hotspot. Nowadays, deep reinforcement learning algorithms have been successfully applied to the fields of games, industry and commerce. However, deep reinforcement learning algorithms often fall into the dilemma of “exploration” and “exploitation”, and the effect of these algorithms is easily affected by the quality of hyperparameters. In order to make up for the defects mentioned above, this paper introduces the particle swarm based reinforcement learning framework (PRL). Compared with the standard reinforcement learning algorithms, this framework greatly improves the exploration ability and obtains better scores in a series of gym experimental tests.KeywordsReinforcement learningParticle swarm optimizationTwin delayed deep deterministic policy gradients

In recent years, deep reinforcement learning (DRL) has made impressive achievements in many fields. However, existing DRL algorithms usually require a large amount of exploration to obtain a good action policy. In addition, in many complex situations, the reward function cannot be well designed to meet task requirements. These two problems will make it difficult for DRL to learn a good action policy within a relatively short period. The use of expert data can provide effective guidance and avoid unnecessary exploration. This study proposes a deep imitation reinforcement learning (DIRL) algorithm that uses a certain amount of expert demonstration data to speed up the training of DRL. In the proposed method, the learning agent imitates the expert's action policy by learning from demonstration data. After imitation learning, DRL is used to optimise the action policy in a self‐learning way. By experimental comparison on a video game called the Mario racing game, it is shown that the proposed DIRL algorithm with expert demonstration data can obtain much better performance than previous DRL algorithms without expert guidance.

This paper addresses a dynamic job shop scheduling problem (DJSSP) based on a real stone paper factory, incorporating parallel machines, order rejection, tardiness costs, machine deterioration, and inventory costs. Orders arrive at variable times, with some potentially rejected due to capacity constraints. Also, raw materials must be purchased at appropriate times and quantities. The model optimizes three objectives: profit maximization, inventory cost variance minimization, and machine workload variance minimization. A heuristic solution method based on a double-network deep reinforcement learning (DRL) algorithm, complemented by an artificial neural network for inventory management, was developed. Evaluation against public benchmarks and other DRL algorithms highlights the impact of inventory cost variance and workload balancing. Real-case evaluation demonstrated that multi-objective optimization reduced inventory cost variance by 9.8%, machine workload variance by 21.87%, and raw material holdings by 21.7%, while slightly decreasing profit (3.2%) and slightly increasing purchase frequency (4.7%). Sensitivity analysis highlighted the individual effects of each objective on profit, workload, and makespan, confirming the practical benefits of the proposed approach.

In recent years, deep reinforcement learning has combined the advantages of reinforcement learning and deep learning, and has made great progress in decision-making tasks. However, the training of deep reinforcement learning requires frequent interactions between the agent and the environment and repeated experiments. Adversaries have chances to poison the sample data collected by the agent by attacking the experimental environment in the training process, thereby bringing security risks and serious consequences to the training process of reinforcement learning. This work is committed to addressing the security risks in the field of deep reinforcement learning. However, this work improves the algorithm from the perspective of sample data filtering, and improves the security performance of deep reinforcement learning algorithm. There are two contributions in this work: one is to defend against adversarial attacks against deep reinforcement learning through cluster analysis and sample value evaluation; the other is to propose a deep reinforcement learning algorithm based on sample value evaluation on the basis of deterministic strategy gradient algorithm. The algorithm uses the clustering method to classify the sample pool, and measures the contribution value and security risk of the sample to the model training through the sample value evaluation. The classic game experiments show that the proposed algorithm is safe and effective. It reduces the threat of the agent falling into the adversarial sample attack and improves the training performance of deep reinforcement learning.