Modeling Low-risk Actions from Multivariate Time Series Data Using Distributional Reinforcement Learning

With the breakthrough of AlphaGo, deep reinforcement learning has become a recognized technique for solving sequential decision-making problems. Despite its reputation, data inefficiency caused by its trial and error learning mechanism makes deep reinforcement learning difficult to apply in a wide range of areas. Many methods have been developed for sample efficient deep reinforcement learning, such as environment modelling, experience transfer, and distributed modifications, among which distributed deep reinforcement learning has shown its potential in various applications, such as human-computer gaming and intelligent transportation. In this paper, we conclude the state of this exciting field, by comparing the classical distributed deep reinforcement learning methods and studying important components to achieve efficient distributed learning, covering single player single agent distributed deep reinforcement learning to the most complex multiple players multiple agents distributed deep reinforcement learning. Furthermore, we review recently released toolboxes that help to realize distributed deep reinforcement learning without many modifications of their non-distributed versions. By analysing their strengths and weaknesses, a multi-player multi-agent distributed deep reinforcement learning toolbox is developed and released, which is further validated on Wargame, a complex environment, showing the usability of the proposed toolbox for multiple players and multiple agents distributed deep reinforcement learning under complex games. Finally, we try to point out challenges and future trends, hoping that this brief review can provide a guide or a spark for researchers who are interested in distributed deep reinforcement learning.

To solve problems involving sequential decision-making, Deep Reinforcement Learning (DRL) combines the advantages of Reinforcement Learning (RL) and Deep Learning (DL).These fusions are skilled at training agents for video games because they use neural networks to approximate nonlinear functions.The Deep Q Network (DQN), a key algorithm in this field, tends to overestimate Q-values despite its effectiveness.The Double Deep Q Network (Double DQN) was presented to address this.The Kera's framework in PyCharm is used in this study to examine the practical application and comparative analysis of DQN and Double DQN in the Flappy Bird game.To speed up its training, improvements were also made to the DQN model.The enhanced DQN performed better than the traditional DQN but less well than the Double DQN, according to the results.This study delves into the practical application and comparative analysis of DQN and Double DQN in the Flappy Bird game, utilizing the Kera's framework in PyCharm.Additionally, enhancements were made to the DQN model to expedite its training.Results affirmed that the enhanced DQN outperformed the conventional DQN but lagged the Double DQN.The training loss trajectory further substantiated Double DQN's superiority in mitigating overestimation issues.

We present a novel statistical approach to incorporate uncertainty awareness in model-free distributional deep reinforcement learning for mission and safety-critical robotics. Deep learning predictions are influenced by uncertainties in the data, termed as aleatoric uncertainties, as well as uncertainties in the learning process and model structure, known as epistemic uncertainties. The proposed algorithm, called as Calibrated Evidential Quantile Regression in Deep-Q Networks (CEQR-DQN), addresses key challenges associated with separately estimating aleatoric and epistemic uncertainty in stochastic robotic environments. It combines deep evidential learning with quantile calibration based on the principles of conformal inference to provide explicit, sample-free computations of global uncertainty as opposed to local estimates based on simple variance. Thereby, the proposed approach overcomes limitations of traditional methods in computational and statistical efficiency and handling of out-of-distribution (OOD) observations. Tested on a suite of representative miniaturized Atari games (i.e., MinAtar), CEQR-DQN is shown to surpass similar existing frameworks in scores and learning speed. Its ability to rigorously evaluate uncertainties improves exploration strategies and can serve as a blueprint for other uncertainty-aware robotic algorithms.

Financial trading is one of the most popular problems for reinforcement learning in recent years. One of the important challenges is that investment is a multi-objective problem. That is, professional investors do not act solely on expected profi t but also carefully consider the potential risk of a given investment. To handle such a challenge, previous studies have explored various kinds of risk-sensitive rewards, for example, the Sharpe ratio as computed by a fi xed length of previous returns. This work proposes a new approach to deal with the profi t-to-risk tradeoff by applying distributional reinforcement learning to build a risk awareness policy instead of a simple risk-based reward function. Our new policy, termed C51-Sharpe, is to select the action based on the Sharpe ratio computed from the probability mass function of the return. This produces a signifi cantly higher Sharpe ratio and lower maximum drawdown without sacrifi cing profi t compared to the C51algorithm utilizing a purely profi t-based policy. Moreover, it can outperform other benchmarks, such as a Deep Q-Network (DQN) with a Sharpe ratio reward function. Besides the policy, we also studied the effect of using double networks and the choice of exploration strategies with our approach to identify the optimal training confi guration. We fi nd that the epsilon-greedy policy is the most suitable exploration for C51-Sharpe and that the use of double network has no signifi cant impact on performance. Our study provides statistical evidence of the effi ciency in risk-sensitive policy implemented by using distributional reinforcement algorithms along with an optimized training process.

In this paper, we propose a federated deep reinforcement learning framework to solve a multi-objective optimization problem, where we consider minimizing the expected long-term task completion delay and energy consumption of IoT devices. This is done by optimizing offloading decisions, computation resource allocation, and transmit power allocation. Since the formulated problem is a mixed-integer non-linear programming (MINLP), we first cast our problem as a multi-agent distributed deep reinforcement learning (DRL) problem and address it using double deep Q-network (DDQN), where the actions are offloading decisions. The immediate cost of each agent is calculated through solving either the transmit power optimization or local computation resource optimization, based on the selected offloading decisions (actions). Then, to enhance the learning speed of IoT devices (agents), we incorporate federated learning (FDL) at the end of each episode. FDL enhances the scalability of the proposed DRL framework, creates a context for cooperation between agents, and minimizes their privacy concerns. Our numerical results demonstrate the efficacy of our proposed federated DDQN framework in terms of learning speed compared to federated deep Q network (DQN) and non-federated DDQN algorithms. In addition, we investigate the impact of batch size, network layers, DDQN target network update frequency on the learning speed of the FDL.

This research was supported by a Marie Curie Intra-European Fellowship within the 6th European Community Framework Programme. The authors would like to thank Jan Sacha for an implementation of CRL in Java on which the experiments in this paper are based.

Despite recent advances in reinforcement learning (RL), its application in safety critical domains like autonomous vehicles is still challenging. Although punishing RL agents for risky situations can help to learn safe policies, it may also lead to highly conservative behavior. In this paper, we propose a distributional RL framework in order to learn adaptive policies that can tune their level of conservativity at run-time based on the desired comfort and utility. Using a proactive safety verification approach, the proposed framework can guarantee that actions generated from RL are fail-safe according to the worst-case assumptions. Concurrently, the policy is encouraged to minimize safety interference and generate more comfortable behavior. We trained and evaluated the proposed approach and baseline policies using a high level simulator with a variety of randomized scenarios including several corner cases which rarely happen in reality but are very crucial. In light of our experiments, the behavior of policies learned using distributional RL can be adaptive at run-time and robust to the environment uncertainty. Quantitatively, the learned distributional RL agent drives in average 8 seconds faster than the normal DQN policy and requires 83\% less safety interference compared to the rule-based policy with slightly increasing the average crossing time. We also study sensitivity of the learned policy in environments with higher perception noise and show that our algorithm learns policies that can still drive reliable when the perception noise is two times higher than the training configuration for automated merging and crossing at occluded intersections.

Deep reinforcement learning for artificial upwelling energy management

Distributional reinforcement learning (DRL) enhances the understanding of the effects of the randomness in the environment by letting agents learn the distribution of a random return, rather than its expected value as in standard reinforcement learning. Meanwhile, a challenge in DRL is that the policy evaluation typically relies on the representation of the return distribution, which needs to be carefully designed. In this paper, we address this challenge for the special class of DRL problems that rely on a discounted linear quadratic regulator (LQR), which we call <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">distributional LQR</i>. Specifically, we provide a closed-form expression for the distribution of the random return, which is applicable for all types of exogenous disturbance as long as it is independent and identically distributed (i.i.d.). We show that the variance of the random return is bounded if the fourth moment of the exogenous disturbance is bounded. Furthermore, we investigate the sensitivity of the return distribution to model perturbations. While the proposed exact return distribution consists of infinitely many random variables, we show that this distribution can be well approximated by a finite number of random variables. The associated approximation error can be analytically bounded under mild assumptions. When the model is unknown, we propose a model-free approach for estimating the return distribution, supported by sample complexity guarantees. Finally, we extend our approach to partially observable linear systems. Numerical experiments are provided to illustrate the theoretical results.

Analyzing massive amounts of data using complex machine learning models requires significant computational resources. The conventional approach to such problems involves centralizing training data and inference processes in the cloud, i.e., in data centers. However, with the proliferation of mobile devices and increasing application of the Internet-of-Things (IoT) paradigm, very large amounts of data are collected at the edges of wireless networks, and due to privacy constraints and limited communication resources, it is undesirable or impractical to upload this data from mobile devices to the cloud for centralized learning. This problem can be solved by distributed learning at the network edge, by which edge devices collaboratively train a shared learning model using real-time mobile data. The avoidance of raw-data uploading not only helps to preserve privacy but may also alleviate network-traffic congestion and minimize latency. With that said, distributed training still requires a substantial amount of information exchange between devices and edge servers over wireless links. In the process, wireless impairments such as noise, interference, and imperfect knowledge of channel states can significantly slow down distributed learning (e.g., convergence speed) and degrades its performance (e.g., learning accuracy). This makes it crucial to optimize wireless network performance so as to support the efficient deployment of distributed learning algorithms. On the other hand, distributed learning algorithms provide a powerful tool-set for solving complex problems in wireless communication and networking. One important framework, called federated learning (FL), enables users to collaboratively learn a shared model while helping to preserve local data privacy. The application of FL can endow edge devices with capabilities of user behavior prediction, user identification, and wireless environment analysis. As another example, distributed reinforcement learning is capable of leveraging distributed computation power and data to solve complex optimization and control problems that arise in various use cases, such as network control, user clustering, resource management, and interference alignment. To cover this paradigm of distributed learning over wireless networks, this two-part Special Issue features papers dealing with two main research challenges: a) optimization of wireless network performance for efficient implementation of distributed learning in wireless networks, and b) distributed learning for solving communication problems and optimizing network performance.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

Multi-echelon supply chain management faces unprecedented challenges from demand uncertainties, disruption risks, and complex interdependencies across network tiers. Traditional optimization approaches struggle to balance risk mitigation with operational efficiency in dynamic environments where distributional properties of returns significantly impact decision quality. This research proposes an adaptive risk-aware planning framework that leverages Distributional Reinforcement Learning (DRL) to optimize inventory positioning and order decisions across multi-echelon supply networks. Unlike conventional reinforcement learning methods that optimize expected returns, the proposed distributional approach explicitly models the full return distribution, enabling risk-sensitive policies that account for tail risks and variance. The framework incorporates a Quantile Regression Deep Q-Network architecture enhanced with attention mechanisms to capture temporal dependencies and inter-echelon coordination requirements. Experimental validation on benchmark multi-echelon supply chain scenarios demonstrates that the distributional approach achieves 18-24% reduction in total supply chain costs compared to traditional base-stock policies while maintaining service level constraints. The risk-aware policies exhibit superior robustness under demand volatility, reducing inventory variance by 31% and backorder occurrences by 42%. Furthermore, the learned policies demonstrate strong generalization capabilities across different demand distributions and network configurations. This research contributes to both theoretical understanding of risk-aware sequential decision-making in supply chains and provides practitioners with computationally tractable methods for adaptive multi-echelon planning under uncertainty.

As a step towards establishing reliable broadcast wireless local area networks (WLANs), this paper proposes acknowledgement (ACK)-less rate adaptation to alleviate reception failures at broadcast recipient stations (STAs) using distributional reinforcement learning (RL). The key point of this study is that the algorithms for learning the strategy of ACK-less rate adaptation are evaluated in terms of the broadcast performance, which is composed of the data rate of the broadcast access point (AP) and the reception success rate at the recipient STAs. ACK-less rate adaptation framework was realized using the received signal strength (RSS) of the uplink frames transmitted by the non-broadcast STAs to the non-broadcast APs, which correlated with the broadcast performance with a confounding effect from the deployment of the broadcast recipient STAs. However, this rate adaptation framework has the problem that it incurs the reception failures at a part of the broadcast recipient STAs, because deep Q-learning used in the previous framework cannot deal with the wide distribution of the broadcast performance. To address this challenge, this paper further discusses the rate adaptation using distributional RL, which approximates the entire distribution of the broadcast performance. The simulations confirmed the following: 1) Using the expected broadcast performance learned by deep Q-learning improved the performance in terms of the Pareto efficiency. 2) Learning the entire distribution of the broadcast performance enabled the broadcast AP to determine the tail of the distribution using risk measure, and to alleviate reception failures while implementing the rate adaptation in the same way as the method that learns only expected broadcast performance.

Humans have to respond to everyday tasks with goal-directed actions in complex and time-continuous environments. However, modeling human behavior in such environments has been challenging. Deep Q-networks (DQNs), an application of deep learning used in reinforcement learning (RL), enable the investigation of how humans transform high-dimensional, time-continuous visual stimuli into appropriate motor responses. While recent advances in DQNs have led to significant performance improvements, it has remained unclear whether these advancements translate into improved modeling of human behavior. Here, we recorded motor responses in human participants (N = 23) while playing three distinct arcade games. We used stimulus features generated by a DQN as predictors for human data by fitting the DQN’s response probabilities to human motor responses using a linear model. We hypothesized that advancements in RL models would lead to better prediction of human motor responses. Therefore, we used features from two recently developed DQN models (Ape-X and SEED) and a third baseline DQN to compare prediction accuracy. Compared to the baseline DQN, Ape-X and SEED involved additional structures, such as dueling and double Q-learning, and a long short-term memory, which considerably improved their performances when playing arcade games. Since the experimental tasks were time-continuous, we also analyzed the effect of temporal resolution on prediction accuracy by smoothing the model and human data to varying degrees. We found that all three models predict human behavior significantly above chance level. SEED, the most complex model, outperformed the others in prediction accuracy of human behavior across all three games. These results suggest that advances in deep RL can improve our capability to model human behavior in complex, time-continuous experimental tasks at a fine-grained temporal scale, thereby opening an interesting avenue for future research that complements the conventional experimental approach, characterized by its trial structure and use of low-dimensional stimuli.

Categorical Distributional Reinforcement Learning (CDRL) has demonstrated superior sample efficiency in learning complex tasks compared to conventional Reinforcement Learning (RL) approaches. However, the practical application of CDRL is encumbered by challenging projection steps, detailed parameter tuning, and domain knowledge. This paper addresses these challenges by introducing a pioneering Continuous Distributional Model-Free RL algorithm tailored for continuous action spaces. The proposed algorithm simplifies the implementation of distributional RL, adopting an actor-critic architecture wherein the critic outputs a continuous probability distribution. Additionally, we propose an ensemble of multiple critics fused through a Kalman fusion mechanism to mitigate overestimation bias. Through a series of experiments, we validate that our proposed method provides a sample-efficient solution for executing complex continuous-control tasks.