A Reinforcement Learning Method for Multiasset Roadway Improvement Scheduling Considering Traffic Impacts

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

With the rapid development of electric vehicles (EVs) and renewable energy, the charging scheduling of EVs with renewable energy becomes a hot issue. Nevertheless, as both the renewable energy and charging demand of EVs are hard to predict, it is difficult to get the optimal charging strategy of the EVs charging problem with renewable energy. In this paper, we consider a multi-EVs charging problem with a discontinuous charging process, where the charging decision for each EV is constrained. We model the problem as Markov decision processes (MDPs) and propose a novel Deep Reinforcement Learning (DRL) method called Constrained Double Deep Q-learning Network (CDDQN) to solve the MDP problem with large state space and decision constraints. Compared with the most of existing DRL methods, the CDDQN method embeds an action constraint model to a double deep Q-learning network (DDQN), which decreases the error of Q-value estimation and improves the accuracy of the charging policy by generating more effective training data. We conduct experiments on a simulation study and compare the proposed method with other DRL methods and experience charging policies. The quantitative results show that the proposed method achieves superior performance on getting the charging policy and outperforms the other methods in getting the minimum charging cost obviously.

To train a mobile robot to navigate using end-to-end approach which maps sensors data into actions, we can use deep reinforcement learning (DRL) method by providing training environments with proper reward functions. Although some studies have shown the success of DRL in navigation task for mobile robots, the method needs appropriate hyperparameter settings such as the environment's timestep size and the robot's velocity range to produce a good navigation policy. The previous existing DRL framework has proposed the use of odometry sensor to generate dynamic timestep size in the environment to solve the mismatch problem between the timestep size and the robot's velocity. However, the framework lacks a procedure for checking terminal conditions which may occur during action executions resulting inconsistency in the environment and poor navigation policies. In the case of navigation task, terminal conditions may happen when the robot achieves the navigation goal position or collides with obstacles while performing an action in one timestep. To cope with this problem, we propose a state transition checking method in the DRL environment which is specific for navigation task that leverages odometry and laser sensor to ensure that the environment follows Markov Decision Process with dynamic timestep size. We also introduce a velocity increment scheduling to stabilize the mobile robot during training. Our experiment results show that state transition checking along with the velocity increment scheduling are able to make the robot navigate faster with higher success rate compared to other existing DRL frameworks.

Adaptive bitrate (ABR) streaming algorithms play an important role in ensuring a high Quality of Experience (QoE) for the consumer. However, a lot of ABR algorithms tend to be too ad hoc. In response, methods based on a Markov Decision Process (MDP) offer more intelligent models. In particular, Reinforcement Learning (RL) methods typically do so via QoE metrics. However, RL methods are plagued by high complexity and long convergence times due to their model-free nature. This paper proposes qMDP, which is an RL method with an MDP partially modeled by an M/D/1/K queue. Our study shows that qMDP results in higher QoE and faster convergence compared to a QoE-only model-free version.

Deep reinforcement learning based energy management for a hybrid electric vehicle

In this paper, an application of Markov Decision Processes (MDP) for modeling selected marketing process is presented. The process is converted into MDP model, where states of the MDP are determined by a configuration of state vector. Elements of the state vector represent most important attributes of the customer in the modeled process. Movement between the states is determined by actions of the customer. In constructed MDP model, individual states with assigned initial reward values then represent consequences of action chosen by the customer resulting in either incresing or reducing the revenue following moving into these states. Value iteration method is then used for computation of the expected final rewards for each state. Based on provided realistic data, customer behavior is analyzed and the best course of action is proposed. Model suitability for future predictions of desired action outcome rate is discussed as well.

The application of stochastic dynamic programming, or Markov Decision Processes (MDP) to the modelling of behavior is discussed. Two examples are described: the hunting behavior of Serengeti lions, and the oviposition behavior of parasitic wasps. The MDP approach appears to explain the published data for these examples much more satisfactorily than does classical optimal foraging theory. The scope for applying MDP models to resource management problems is discussed briefly.

Background: One of the problems of the modern lawmakers in different countries is that they try to regulate an object before they study the nature of its origin, which, logically, entails many errors regarding its definition in the legal framework. The absence of unified definitions and clear classification of virtual assets as tools for implementing the methods of financial and management accounting of property according to their fundamental and unique features makes it nearly impossible to determine the features of virtual assets important for legal regulation and, therefore, to enshrine them in laws and establish a proper legal framework. The paper is dedicated to solving a relevant and cross-discipline scientific and applied task of developing a comprehensive multilevel classification of virtual assets. Unlike the few existing classifications that focus on parts of the virtual asset phenomenon and selective methods of its implementation, the paper proposes an all-encompassing comparison of all known types of virtual assets, which confirms the comprehensiveness of the classification proposed in this paper. Purpose: To develop and substantiate a comprehensive and multilevel classification of known types of virtual assets, which allows solving the cross-discipline scientific and applied task of systematizing virtual assets for future development of a single approach to regulating relations, the objects of which are different types of virtual assets. Materials and Methods: In order to study the nature of virtual assets and develop a comprehensive classification, a set of scientific research methods has been used: analysis, including cause and effect analysis, synthesis, comparison, generalization, systematization and interpretation of results and induction. Results: The author describes a triune nature of virtual assets: technological, economic and legal, information and applied. This classification of virtual assets will allow determining promising tools for accounting of property and rights. Unlike other known approaches to differentiating virtual assets, where crypto-assets (or cryptocurrencies) were unjustified “leaders”, the author has distinguished the group of tokenized assets for the first time. This particular group, due to its direct relation to property, allows performing accounting as well as reaccounting of property and rights in modern digital accounting systems – decentralized information platforms based on the distributed ledger technology (blockchain), whereas this accounting cannot be performed using crypto-assets due to absence of direct relation to property. Out of virtual assets, the author distinguishes a digital asset and analyzes the semantic features of the term “digital asset”. The digital asset is based on a unique information resource as the original asset and on the property of derivativeness from the real asset, which greatly differentiates it from other types of virtual assets. All of that allows considering it as an effective tool for implementing the methods of financial and management accounting of property. Thus, owners of digital assets can use the new way of accounting of their property and personal non-property rights. Based on the properties of a digital asset, the author distinguishes other types of virtual assets: polyasset and monoasset, with the relevant examples. The author provides the characteristics of their features and structural components while comparing them to the features of digital assets and giving clear and well-known financial and legal analogies regarding the implementation of mutual obligations between parties to a traditional deal. The paper also contains the first systematization of seven properties and parameters of a tokenized asset and, therefore, description of properties of three variations of a tokenized asset: monoasset, polyasset and digital asset. This allowed presenting the varieties of virtual assets as a three-level classification based on the complexity of the nature of virtual assets. The author’s classification distinguishes seven types of virtual assets and contains their description. Conclusions: Overall, the proposed approach to classification allows giving a scientific answer to the question of how to compare the multitude of known virtual assets and how to relate them to the legal framework of a state. These developments will be useful for legislators in basically every country, financial, tax and banking state bodies, as well as private companies when keeping books and performing accounting of virtual assets in their business activity.

With the increasing complexity of power system structures and the increasing penetration of renewable energy, the number of possible power system operation modes increases dramatically. It is difficult to make manual power flow adjustments to establish an initial convergent power flow that is suitable for operation mode analysis. At present, problems of low efficiency and long time consumption are encountered in the formulation of operation modes, resulting in a very limited number of generated operation modes. In this paper, we propose an intelligent power flow adjustment and generation model based on a deep network and reinforcement learning. First, a discriminator is trained to judge the power flow convergence, and the output of this discriminator is used to construct a value function. Then, the reinforcement learning method is adopted to learn a strategy for power flow convergence adjustment. Finally, a large number of convergent power flow samples are generated using the learned adjustment strategy. Compared with the traditional flow adjustment method, the proposed method has significant advantages that the learning of the power flow adjustment strategy does not depend on the parameters of the power system model. Therefore, this strategy can be automatically learned without manual intervention, which allows a large number of different operation modes to be efficiently formulated. The verification results of a case study show that the proposed method can independently learn a power flow adjustment strategy and generate various convergent power flows.

The centralized traffic grid signal control by the reinforcement learning method is challenging due to the difficulties of searching policy in the large state and action space. In order to solve these problems, a deep reinforcement learning (DRL) method via the action and state decomposition mechanism is proposed. We apply long short‐term memory to construct the agent which decomposes the high‐dimensional state and action space into sub‐spaces and makes decisions incrementally. This is a significant difference between our method and other methods. Through the specifically designed structure of the agent, the difficulty of searching policies can be mitigated, and our method can effectively control the traffic lights in a grid with hundreds of intersections. Experiments on synthetic data and real‐world data show that our method has better performance than traditional control methods and state‐of‐the‐art DRL‐based methods with an improvement of 21% on the queue length on synthetic data and the best travel time with an improvement of 9.35% on real‐world data.

In order to build a low-carbon and environmentally friendly energy system, a large number of distributed generation connected to the grid, which has brought great challenges to the stable operation of the distribution network. The intermittency and volatility of distributed generation lead to repeated voltage violations in power system. At the same time, the frequent switching of transformer taps and capacitor switches greatly increases the economic cost. This paper introduces the method of deep reinforcement learning to transform the Volt-Var optimize problem into a Markov decision process with continuous action space. The soft actor-critic algorithm is used to train the agent to control the static var compensator and the smart inverter to optimize the reactive power of the distribution network. Comparative studies on IEEE 12-node and 33-node test systems demonstrate the performance of the proposed solution.

An equilibrium approach for compensating public–private partnership concessionaires for reduced tolls during roadway maintenance

AQROM: A quality of service aware routing optimization mechanism based on asynchronous advantage actor-critic in software-defined networks

In this paper, we propose a reinforcement learning based region of interest (ROI) bit allocation method for gaming video coding in Versatile Video Coding (VVC). Most current ROI-based bit allocation methods rely on bit budgets based on frame-level empirical weight allocation. The restricted bit budgets influence the efficiency of ROI-based bit allocation and the stability of video quality. To address this issue, the bit allocation process of frame and ROI are combined and formulated as a Markov decision process (MDP). A deep reinforcement learning (RL) method is adopted to solve this problem and obtain the appropriate bits of frame and ROI. Our target is to improve the quality of ROI and reduce the frame-level quality fluctuation, whilst satisfying the bit budgets constraint. The RL-based ROI bit allocation method is implemented in the latest video coding standard and verified for gaming video coding. The experimental results demonstrate that the proposed method achieves a better quality of ROI while reducing the quality fluctuation compared to the reference methods.

Abstract: Online resource management in cloud computing must respond to rapidly changing demand and system conditions, often exceeding the adaptability of static or periodically recomputed optimisation plans. This study formulates cloud resource management as a Markov decision process (MDP) and investigates deep reinforcement learning (DRL) as an approximate dynamic programming (ADP) approach for deriving adaptive control policies. The system state encompasses workload, capacity utilisation, and performance indicators, while actions govern virtual machine placement, scaling, and server power-state transitions. Reward functions combine energy and operating costs with penalties associated with service-level agreement violations. DRL algorithms, including deep Q-networks, advantage actor–critic, and proximal policy optimization, are implemented in PyTorch and trained through interaction with a CloudSim-based simulation environment calibrated to realistic load and power profiles. The learned policies are evaluated against deterministic and scenario-based stochastic optimisation baselines using metrics such as average cost, energy consumption, SLA violations, and robustness to non-stationary demand. Results show that DRL policies can approach or surpass the performance of stochastic optimisation solutions in highly dynamic scenarios, while maintaining feasible control actions. The study demonstrates how DRL can be positioned within an OR/ADP framework and highlights conditions under which learning-based methods offer significant advantages for online cloud control. Keywords: deep reinforcement learning, approximate dynamic programming, cloud computing, Markov decision processes, online resource management, energy-aware control, PyTorch, simulation