Deep Reinforcement Learning-Based Adaptive Scheduling for Wireless Time-Sensitive Networking.

In this paper, we address the difficulty of solving large-scale multi-dimensional knapsack instances (MKP), presenting a novel deep reinforcement learning (DRL) framework. In this DRL framework, we train different agents compatible with a discrete action space for sequential decision-making while still satisfying any resource constraint of the MKP. This novel framework incorporates the decision variable values in the 2D DRL where the agent is responsible for assigning a value of 1 or 0 to each of the variables. To the best of our knowledge, this is the first DRL model of its kind in which a 2D environment is formulated, and an element of the DRL solution matrix represents an item of the MKP. Our framework is configured to solve MKP instances of different dimensions and distributions. We propose a K-means approach to obtain an initial feasible solution that is used to train the DRL agent. We train four different agents in our framework and present the results comparing each of them with the CPLEX commercial solver. The results show that our agents can learn and generalize over instances with different sizes and distributions. Our DRL framework shows that it can solve medium-sized instances at least 45 times faster in CPU solution time and at least 10 times faster for large instances, with a maximum solution gap of 0.28% compared to the performance of CPLEX. Furthermore, at least 95% of the items are predicted in line with the CPLEX solution. Computations with DRL also provide a better optimality gap with respect to state-of-the-art approaches.

Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach

With the rapid development of smart grid and renewable energy, demand response mechanism has become a key means to balance microgrid supply and demand and improve energy efficiency. As the traditional power demand response algorithm is difficult to deal with the uncertainty of power demand and the adverse effects caused by customers, this paper proposes a power demand response algorithm based on deep reinforcement learning. Firstly, we study and design a price -based layered electricity energy demand response model. Then, the dynamic optimal pricing decision of power trading market is described as Markov decision process, and the learning mechanism under the framework of deep reinforcement learning is expounded. Finally, a solution algorithm based on deep reinforcement learning is designed. The simulation results show that the proposed algorithm can adjust the electricity price adaptively according to the load demand, which can reduce the cost of users by about 17% and reduce the peak load demand.

To address the static limitations of current career assessment methods and the lack of a long-term perspective in planning strategies, this study constructs an intelligent analysis framework named Deep learning-driven Value-added Career Analytics Model (DV-CAM). Integrating Deep Learning (DL) with value-added assessment, the framework aims to realize dynamic and accurate evaluation of individual professional competencies and generate personalized career planning paths that maximize the benefits of long-term development. Specifically, DV-CAM first leverages Bidirectional Encoder Representations from Transformers (BERT) and Long Short-Term Memory (LSTM) to extract and dynamically update individual competency profiles from multi-source time-series data. Second, it designs a value-added evaluation index system covering the absolute level of competency, growth slope, growth curvature and stability to quantitatively diagnose development potential. Finally, under the framework of Deep Reinforcement Learning (DRL), this study takes competency status, value-added potential and job requirements as joint state inputs, and solves for the optimal sequence of career development actions by optimizing long-term cumulative rewards. Experiments based on the public Occupational Information Network (O*NET) dataset show that DV-CAM achieves a mean squared error of 0.0312 and a mean absolute error of 0.1289 for competency assessment on the test set. In planning tasks, its long-term cumulative reward and target job matching degree improvement rate reach 185.3% and 42.7% respectively, both outperforming various benchmark models. The results demonstrate that DV-CAM can diagnose competency status more accurately, and effectively guide individuals to achieve long-term career development goals through its forward-looking planning strategies. By combining the concept of value-added assessment with advanced deep learning technologies, this study provides a data-driven, dynamically perceivable and long-term oriented new paradigm for the fields of career planning and competency assessment.

Effective ways to optimise real-time pump scheduling to maximise energy efficiency are being sought to meet the challenges in the energy market. However, the considerable number of evaluations of popular optimisation methods based on metaheuristics cause significant delays for real-time pump scheduling, and the simplification of traditional deterministic methods may introduce bias towards the optimal solutions. To address these limitations, an exploration-enhanced deep reinforcement learning (DRL) framework is proposed to address real-time pump scheduling problems in water distribution systems. The experimental results indicate that E-PPO can learn suboptimal scheduling policies for various demand distributions and can control the application time to 0.42 s by transferring the online computation-intensive optimisation task offline. Furthermore, a form of penalty of the tank level was found that can reduce energy costs by up to 11.14% without sacrificing the water level in the long term. Following the DRL framework, the proposed method makes it possible to schedule pumps in a more agile way as a timely response to changing water demand while still controlling the energy cost and level of tanks.

Manually designing an effective and efficient controller for complex mechanics, such as bio-inspired robots or underactuated mechanical system, typically are very difficult. It requires precise motion planning and dynamic control. Reinforcement learning or genetic algorithm based learning methods suffers from representing the high dimensional models. The combination of deep learning and reinforcement learning provide a feasible way to handle such difficulties. However, priori-less searching sometimes tends to be low efficient and usually finds the “mechanic” solution instead of the “natural” one. In this paper, the traditional nonlinear control concept is integrated into the deep reinforcement learning (DRL) framework. The whole process is implemented on the brachiation control problem of a two link bio-primate robot. Deep Deterministic Policy Gradient (DDPG) is used to search for the optimal control policy. The searching process is realized by interacting with the dynamic model instead of real robot. The energy based planning and control concept is adopted, which utilize the fact that when the shoulder joint angle is fixed, energy of the whole system keeps constant. By regulating the angle and energy, the robot can be restricted on a particular trajectory. The energy concept is encoded within the reward function and trained in the Gym environment. For varying targets point-to-point control, the network structure is also modified to accept the target coordinates. Effectiveness of the proposed methods are verified by simulation and experimental results.

A multi-objective deep reinforcement learning framework

This paper proposes a deep reinforcement learning (DRL) framework that learns a reusable policy to find the optimal placement of decoupling capacitors (decaps) on power distribution network (PDN) with an arbitrary probing port. The proposed DRL framework trains a policy parameterized by pointer network, which is a sequence-to-sequence neural network, based on REINFORCE algorithm. The policy finds the positional combination of a pre-defined number of decaps that best suppresses self-impedance of a given probing port on PDN with randomly assigned keep-out regions. Verification was done by allocating 20 decaps on ten randomly generated test sets with an arbitrary probing port and randomly selected keep-out regions. Performance of the policy generated by the proposed DRL framework was evaluated based on the magnitude of probing port self-impedance suppression followed by decap placement over 434 frequencies between 100MHz and 20GHz. The policy generated by the proposed framework achieves greater impedance suppression with fewer samples in comparison to random search heuristic method.

5th-Generation (5G) and Time-Sensitive Networking (TSN) are regarded as competitive new technologies for future industrial networks; 5G-TSN collaboration transmission has drawn more attention because it can provide a guarantee of low-latency, ultra-reliable and deterministic transmission for time-critical automation applications. However, the methodologies of resource scheduling mechanisms in 5G and Time-Sensitive Networking (TSN) are quite different, which may lead to an inefficient Quality of Service (QoS) guarantee for deterministic transmission across 5G and TSN. Therefore, an efficient 5G-TSN joint scheduling algorithm based on Deep Deterministic Policy Gradient (DDPG) is proposed and analyzed in this article. The proposed algorithm takes both 5G radio channel information and the Gate Control List (GCL) state in the TSN domain into consideration, aiming to provide a latency guarantee for time-triggered applications across 5G and TSN as well as a throughput guarantee for video applications in 5G systems. The simulation results compare the latency and throughput performance of the proposed joint scheduling algorithm with several traditional 5G scheduling algorithms; meanwhile, several GCL setting methods are given to verify the impacts on latency and throughput performance within the proposed algorithm. The simulation results demonstrate that the proposed DDPG-based joint scheduling algorithm can significantly enhance the multi-application-carrying capability of 5G-TSN collaboration architecture.

We present a hierarchical deep reinforcement learning (DRL) framework with prominent sampling efficiency and sim-to-real transfer ability for fast and safe navigation: the low-level DRL policy enables the robot to move towards the target position and keep a safe distance to obstacles simultaneously; the high-level DRL policy is supplemented to further enhance the navigation safety. We select a waypoint located on the path from the robot to the ultimate goal as the sub-goal to reduce the state space and avoid sparse reward. Moreover, the path is generated based on either a local or a global map, which can significantly improve the sampling efficiency, safety, and generalization ability of the proposed DRL framework. Additionally, a target-directed representation for the action space can be derived based on the sub-goal to improve the motion efficiency and reduce the action space. In order to demonstrate the eminent sampling efficiency, motion performance, obstacle avoidance, and generalization ability of the proposed framework, we implement sufficient comparisons with the non-learning navigation methods and DRL-based baselines, with videos, data, code, and other supplemental material shown on our website<xref ref-type="fn" rid="fn1">¹</xref><fn id="fn1"><label>¹</label> https://github.com/zw199502/RL <inline-formula><tex-math notation="LaTeX">$\_$</tex-math></inline-formula> navigation </fn>.

Non-terrestrial network (NTN) communication is included in the 3GPP standard because of its excellent features such as resistance to ground physical attacks and wide coverage. Since the available power and storage resources are limited on high throughput satellites (HTSs), optimizing the resource allocation can greatly improve the performance of the HTS based communication system. Notably, weather and other factors make the channel status between the satellite and the terrestrial base stations constantly change. In this paper, we first exploit the model-free feature of reinforcement learning to formulate the aforementioned dynamic and unpredictable channel conditions into the power allocation problem in HTS systems. Due to the complexity of environment state and power allocation, a deep neural network is introduced to replace the Q table, and a deep reinforcement learning framework is built. Furthermore, a power allocation algorithm based on a deep reinforcement learning framework is presented. Finally, the simulation results show that the proposed algorithm is superior to existing algorithms in terms of the long-term system throughput performance, and has better adaptability to dynamic channel environment, thereby improving network performance.

With The advent of the Internet of Things (IoT) and its use cases there is a necessity for improved latency which has led to edgecomputing technologies. IoT applications need a cloud environment and appropriate scheduling based on the underlying requirements of a given workload. Due to the mobility nature of IoT devices and resource constraints and resource heterogeneity, IoT application tasks need more efficient scheduling which is a challenging problem. The existing conventional and deep learning scheduling techniques have limitations such as lack of adaptability, issues with synchronous nature and inability to deal with temporal patterns in the workloads. To address these issues, we proposed a learning-based framework known as the Deep Reinforcement Learning Framework (DRLF). This is designed in such a way that it exploits Deep Reinforcement Learning (DRL) with underlying mechanisms and enhanced deep network architecture based on Recurrent Neural Network (RNN). We also proposed an algorithm named Reinforcement Learning Dynamic Scheduling (RLbDS) which exploits different hyperparameters and DRL-based decision-making for efficient scheduling. Real-time traces of edge-cloud infrastructure are used for empirical study. We implemented our framework by defining new classes for CloudSim and iFogSim simulation frameworks. Our empirical study has revealed that RLbDS out performs many existing scheduling methods.

Infrastructure‐to‐infrastructure (I2I) communication enables the exchange of traffic data between intersections, which brings a new challenge to urban traffic control. This paper proposes a novel deep reinforcement learning (DRL) framework for urban traffic signal control within the vehicle‐to‐everything (V2X) ecosystem. The framework incorporates a joint‐state representation integrating traffic data from both I2I and vehicle‐to‐infrastructure (V2I) communications, while considering communication range effects. The reward function is designed to optimize both local intersection conditions and global network performance, facilitating adaptive signal coordination. A simulation case study in Huzhou, China, evaluates the proposed model against conventional pretimed, actuated, nonjoint‐state DRL, and joint‐state reinforcement learning (RL) models. Results demonstrate superior performance of the joint‐state DRL model in episode rewards, average speed, and average time loss, particularly during peak traffic periods. To address data limitations, Monte Carlo cross‐validation (MCCV) is employed, further validating the model’s robustness. Results show consistent performance advantages in average speed and time loss across peak and off‐peak periods, with slight variations compared to joint‐state RL models in certain intervals. The impacts of the communication range are also discussed with the proposed model. Pearson correlation analysis reveals a strong positive correlation between the communication range and convergence time across all traffic periods. Meanwhile, correlations between the communication range and reward, average speed, and average time loss vary by traffic period. Findings highlight the transformative potential of integrating DRL with V2X communication technologies for enhancing traffic signal control in complex urban environments. The proposed model offers a flexible, adaptive approach to traffic management, optimizing flow while maintaining safety standards, with implications for future smart city developments.

In coupled power distribution and transportation (CPT) system, a joint scheduling framework for mobile energy storage systems (MESSs) and Volt/VAR control (VVC) ensures reliable power distribution grid operations while supporting electric vehicle loads at electric vehicle charging stations (EVCSs). However, conventional model-based optimization methods for MESS scheduling and VVC may yield suboptimal solutions and greater computation times because of MESS operation and VVC in uncertain environment of CPT systems. To resolve this issue, this study proposes a model-free deep reinforcement learning (DRL) framework. In this framework, smart inverters of MESSs and solar photovoltaic (PV) systems cooperate to minimize the real power loss and mitigate the violations of both MESSs’ state of charge (SOC) and voltage in the power distribution network, while MESSs travel via the transportation network to satisfy EV loads at EVCSs. A MESS routing algorithm based on Dijkstra’s algorithm is developed to determine the optimal destinations of the MESSs. In addition, two safety modules are developed to ensure that neither SOC nor voltage violations occur by adjusting real and/or reactive power of MESSs and PV systems during the training process. The developed MESS routing algorithm and safety modules are integrated into the proposed DRL framework, wherein the DRL agent performs the desired MESS scheduling and VVC through safe exploration during the training procedure. The proposed approach is tested in coupled IEEE 33-bus power distribution and 15-node transportation systems and coupled IEEE 57-bus power distribution and 42-node transportation systems. Numerical examples demonstrate the advantages of the proposed approach in terms of training convergence, real power loss, and SOC/voltage violation.

Robot navigation has transitioned from avoiding static obstacles to adopting socially aware navigation strategies for coexisting with humans. Consequently, socially aware navigation in dynamic, human-centric environments has gained prominence in the field of robotics. One of the methods for socially aware navigation, the reinforcement learning technique, has fostered its advancement. However, defining appropriate reward functions, particularly in congested environments, holds a significant challenge. These reward functions, crucial for guiding robot actions, necessitate intricate human-crafted design due to their complex nature and inability to be set automatically. The multitude of manually designed reward functions contains issues such as hyperparameter redundancy, imbalance, and inadequate representation of unique object characteristics. To address these challenges, we introduce a transformable Gaussian reward function (TGRF). The TGRF possesses two main features. First, it reduces the burden of tuning by utilizing a small number of hyperparameters that function independently. Second, it enables the application of various reward functions through its transformability. Consequently, it exhibits high performance and accelerated learning rates within the deep reinforcement learning (DRL) framework. We also validated the performance of TGRF through simulations and experiments.