Advances in deep reinforcement learning enable better predictions of human behavior in time-continuous tasks

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

To solve problems involving sequential decision-making, Deep Reinforcement Learning (DRL) combines the advantages of Reinforcement Learning (RL) and Deep Learning (DL).These fusions are skilled at training agents for video games because they use neural networks to approximate nonlinear functions.The Deep Q Network (DQN), a key algorithm in this field, tends to overestimate Q-values despite its effectiveness.The Double Deep Q Network (Double DQN) was presented to address this.The Kera's framework in PyCharm is used in this study to examine the practical application and comparative analysis of DQN and Double DQN in the Flappy Bird game.To speed up its training, improvements were also made to the DQN model.The enhanced DQN performed better than the traditional DQN but less well than the Double DQN, according to the results.This study delves into the practical application and comparative analysis of DQN and Double DQN in the Flappy Bird game, utilizing the Kera's framework in PyCharm.Additionally, enhancements were made to the DQN model to expedite its training.Results affirmed that the enhanced DQN outperformed the conventional DQN but lagged the Double DQN.The training loss trajectory further substantiated Double DQN's superiority in mitigating overestimation issues.

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.

Reinforcement Learning (RL) is a successful and established Artificial Intelligence (AI) method, particularly with recent groundbreaking progress in Deep Reinforcement Learning (DRL). Reinforcement learning is very well suited for sequential decision-making tasks, wherein a learned agent learns an optimal policy after many interactions with an environment. The present paper examines the application of reinforcement learning for automating screening of literature in academic research, particularly in the fields of computer science and e-learning. Keyword filtering techniques, while predominantly applied, are found to be inflexible as well as unable to capture the dynamic nature of research themes. To overcome such constraints, we present a Deep Q-Network (DQN)-based reinforcement learning model that combines reinforcement learning with the Semantic Scholar API to enhance research paper classification based on dynamically acquired decision rules. The proposed reinforcement learning model was trained and tested with a dataset of 8,934 research papers, accessed by systematic searching. The agent filtered and picked 11 effective papers depending on improved selection criteria like publication date, keyword relevance, and scholarly topic provided. The model iteratively optimizes the decision-making process through reward-based learning and therefore maximizes categorization accuracy over time. Test experiments demonstrate utilization of RL-based suggested framework yields classification accuracy at 91.5%, recall at 86.3%, and precision at 89.7%. A comparison test demonstrates that the approach performs 12.5% better on recall and 9.8% better on accuracy compared to traditional keyword-filtering methods. The finding confirms the power of the model in minimizing false positives and false negatives for screening literature, hence proving the scalability and adaptability of reinforcement learning in managing high academic data. This work offers a scalable, cognitive approach to conducting systematic reviews of literature through the application of reinforcement learning to programmatically execute work in academic research. The work shows the promise of reinforcement learning to further enhance research methodology, make literature reviews more effective, and facilitate more knowledgeable decision-making in fast-changing scientific disciplines. Further research will be focused on incorporating hybrid AI models with multi-agent systems of reinforcement learning for responsiveness and classification enhancements.

In order to better understand state-of-the-art reinforcement learning agent, deep Q-network, a brief review of reinforcement learning and Q-learning are first described. Then recent advances of deep Q-network are presented, and double deep Q-network and dueling deep Q-network that go beyond deep Q-network are also given.

To decrease the impact of Partially Observable MDP on deep reinforcement learning performance of intersection signal control, A deep reinforcement learning is proposed in this paper with utilizing the real-time GPS data as well as learning the control of the traffic lights in single intersection. We integrate deep reinforcement learning network (DRQN) with recurrent neural network (RNN) and apply deep network, experience pool and greedy strategy in deep reinforcement learning strategy. It solves the problem of overestimation of target Q value and insufficient long-term experience learning in the standard reinforcement learning of traffic signal control. The comparison of performance was made between the proposed method and standard Deep Q-Network (DQN) on the partial observation of traffic situations. The experimental results show that both DQN and DRQN methods can adjust their traffic signal timing control strategies according to the specific traffic conditions as well as calculating a lower average delay time of vehicle than that of fixed-time control. Besides, the simulation effect of DRQN learning method is better than that of DQN learning method in different probe vehicle proportion environment.

While the recent advanced military operational concept requires an intelligent support of command and control, Reinforcement Learning (RL) has not been actively studied in the military domain. This study points out the limitations of RL for military applications from literature review and aims at improving the understanding of RL for military decision support under the limitations. Most of all, the black box characteristic of Deep RL makes the internal process difficult to understand in addition to complex simulation tools. A scalable weapon selection RL framework is built which can be solved either by a tabular form or a neural network form. The transition of the Deep Q-Network (DQN) solution to the tabular form makes it easier to compare the result to the Q-learning solution. Furthermore, rather than using one or two RL models selectively as before, RL models are divided as an actor and a critic, and systematically compared. A random agent, Q-learning and DQN agents as a critic, a Policy Gradient (PG) agent as an actor, Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) agents as an actor-critic approach are designed, trained, and tested. The performance results show that the trained DQN and PPO agents are the best decision supporter candidates for the weapon selection RL framework.

Reinforcement Learning (RL) has become one of the most powerful methods in Artificial Intelligence (AI) which allows a system to learn as it is engaged in an interaction with its environment and to maximize its actions by performing a trial and error process. Although classical RL algorithms can give us a fundamental idea on RL, like the Q-Learning, its usefulness wanes in high-dimensional or continuous-state spaces. To overcome these shortcomings, Deep Reinforcement Learning (DRL) algorithms including Deep Q-Networks (DQN) build on neural networks to learn action-value functions, allowing horizontal scalability and more efficient learning. This paper reports on the experimental comparison of Q-Learning and DQN based on the environment CartPole-v1. Both of the algorithms were trained in the controlled environment and tested under various metrics that include cumulative rewards, success rate, convergence speed, safety (failure rate), and path efficiency. Findings indicate that Q-Learning moves towards moderate stability with slower convergence but DQN takes shorter learning time, is more reliable and performs better. Such results indicate the ground-breaking potential of the DRL in the progress of AI toward transcending classical RL restrictions. Future research topics encompass implementation of more challenging experimental settings, a test of other DRL algorithms (e.g., PPO, SAC), and generalization, scalability and safety in real-world environments.

In this article, we consider a subclass of partially observable Markov decision process (POMDP) problems which we termed confounding POMDPs. In these types of POMDPs, temporal difference (TD)-based reinforcement learning (RL) algorithms struggle, as TD error cannot be easily derived from observations. We solve these types of problems using a new bio-inspired neural architecture that combines a modulated Hebbian network (MOHN) with deep Q-network (DQN), which we call modulated Hebbian plus Q-network architecture (MOHQA). The key idea is to use a Hebbian network with rarely correlated bio-inspired neural traces to bridge temporal delays between actions and rewards when confounding observations and sparse rewards result in inaccurate TD errors. In MOHQA, DQN learns low-level features and control, while the MOHN contributes to high-level decisions by associating rewards with past states and actions. Thus, the proposed architecture combines two modules with significantly different learning algorithms, a Hebbian associative network and a classical DQN pipeline, exploiting the advantages of both. Simulations on a set of POMDPs and on the Malmo environment show that the proposed algorithm improved DQN's results and even outperformed control tests with advantage-actor critic (A2C), quantile regression DQN with long short-term memory (QRDQN + LSTM), Monte Carlo policy gradient (REINFORCE), and aggregated memory for reinforcement learning (AMRL) algorithms on most difficult POMDPs with confounding stimuli and sparse rewards.

In order to make the robot obtain the optimal action directly from the original visual perception without any hand-crafted features and features matching, a novel end-to-end path planning method-mobile robot path planning using deep reinforcement learning is proposed. Firstly, a deep Q-network (DQN) is designed and trained to approximate the mobile robot state-action value function. Then, the Q value corresponding to each possible mobile robot action (i.e., turn left, turn right, forward) is determined by the well trained DQN, here, the input of the DQN is the original RGB image (image pixels) captured from the environment without any hand-crafted features and features matching; Finally, the current optimal mobile robot action is selected by the action selection strategy. Mobile robot reach to the goal point while avoiding obstacles ultimately. 30 times path planning experiments are conducted in the seekavoid_arena_01 environment on DeepMind Lab platform. The experimental results show that our deep reinforcement learning based robot path planning method is an effective end-to-end mobile robot path planning method.

With the rapid development of AI technology, personalized learning path planning is becoming an important research field. Dual system teaching models have great influence on the learning effect of students. Therefore, how to optimize the learning process of students through the personalized learning path is a great challenge. A Deep Reinforcement Learning (DRL) algorithm for personalized learning path planning is proposed. Based on deep Q-networks and reinforcement learning feedback mechanisms, the algorithm achieves adaptive optimization. The experimental results show that the algorithm based on the DRL is superior to the traditional method in learning performance, learning efficiency and satisfaction. Compared with traditional path planning algorithms based on rules and collaborative filtering, the proposed algorithm improves learning performance by 20%. Moreover, it has obvious advantages in learning motivation and efficiency. This paper provides a new way of thinking to intelligent education system, especially dual-system teaching model, and it is of great practical significance.

Machine learning offers advanced tools for efficient management of radio resources in modern wireless networks. In this study, we leverage a multi-agent deep reinforcement learning (DRL) approach, specifically the Parameterized Deep Q-Network (DQN), to address the challenging problem of power allocation and user association in massive multiple-input multiple-output (M-MIMO) communication networks. Our approach tackles a multi-objective optimization problem aiming to maximize network utility while meeting stringent quality of service requirements in M-MIMO networks. To address the non-convex and nonlinear nature of this problem, we introduce a novel multi-agent DQN framework. This framework defines a large action space, state space, and reward functions, enabling us to learn a near-optimal policy. Simulation results demonstrate the superiority of our Parameterized Deep DQN (PD-DQN) approach when compared to traditional DQN and RL methods. Specifically, we show that our approach outperforms traditional DQN methods in terms of convergence speed and final performance. Additionally, our approach shows 72.2% and 108.5% improvement over DQN methods and the RL method, respectively, in handling large-scale multi-agent problems in M-MIMO networks.

The usage of distributed Peer-to-Peer (P2P) networks has been growing steadily for a reasonable period. Various applications rely on the infrastructure of P2P networks, where nodes communicate to accomplish a task without the need for a central authority. One of the significant challenges in P2P networks is the ability of the network nodes to reach a consensus on a shared item; the challenge increases as time passes. Thus, this work proposes a new effective method for tweaking the Deep Reinforcement Learning (DRL) algorithm to train Deep Q Network (DQN) learning agents to reach a consensus among the P2P nodes. We propose various hierarchies of deep agents to address this crucial challenge in P2P networks. DRL is utilized to build and train agents; more precisely, DQN learning agents are constructed and trained. Two scenarios are proposed and evaluated. In the first scenario, one DQN agent is trained to find the consensus between the network nodes. In the second scenario, three hierarchies with different numbers of layers of agents are proposed and evaluated. In both scenarios, the P2P network used is a blockchain network. The best result was obtained using the third hierarchy of the second scenario; the overall accuracy of the model is 87.8%.

Reinforcement learning-based real-time optimization of friction stir welding parameters for copper–aluminium dissimilar interfaces

The transition to sustainable and intelligent energy systems has intensified the development of smart microgrids, which offer decentralized, resilient, and efficient power solutions. This review critically examines the integration of Artificial Intelligence (AI) and Deep Reinforcement Learning (DRL) into smart microgrid platforms, focusing on their role in optimizing sustainable energy management. Traditional energy management systems often struggle to adapt to the dynamic nature of modern energy demands, renewable energy intermittency, and grid complexity. AI-driven solutions, particularly DRL, provide adaptive, autonomous, and data-driven mechanisms for real-time decision-making and predictive control within microgrids. DRL, by learning optimal policies through interaction with the environment, is capable of handling multi-objective problems, including demand-response optimization, energy storage control, load forecasting, and distributed generation scheduling. This paper synthesizes recent advancements and applications of DRL algorithms such as Deep Q-Networks (DQN), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO) in smart microgrids. It also explores hybrid models that combine DRL with other AI techniques, such as fuzzy logic and neural networks, to improve performance under uncertainty and nonlinearity. Furthermore, the review evaluates benchmark testbeds, simulation tools, and real-time platforms used to implement and validate these intelligent systems. Challenges such as high computational costs, model generalizability, real-time implementation issues, and cybersecurity vulnerabilities are discussed. The study also highlights recent research trends emphasizing decentralized control, edge computing, and federated learning to enhance scalability and privacy in DRL-based microgrid applications. Future directions suggest the need for explainable AI (XAI), robust training environments, and standardization to facilitate the wider adoption of DRL in real-world microgrid systems. The integration of DRL into smart microgrids represents a transformative shift toward resilient, efficient, and sustainable energy ecosystems. This review offers a comprehensive understanding of how AI and DRL are revolutionizing energy management, providing a foundation for future research and practical deployment in smart energy infrastructures.