Event-based reinforcement learning algorithm for dynamic resource allocation in smart grid

In this letter, we investigate the adaptive resource allocation with Lyapunov optimization for time-varying wireless networks. The transmit power minimization is characterized by a stochastic optimization model. A dynamic resource allocation (DRA) algorithm is developed to accommodate the wireless network dynamics, i.e., time-varying wireless channels and as well the queuing dynamics. We investigate the tracking error between the DRA algorithm output and the target optimal resource allocation solution. Based on these results, we further develop an adaptive-compensation resource allocation (ACRA) algorithm, which iterates only once when the network state changes for saving the huge iteration overheads. Finally, we determine a sufficient condition that the ACRA algorithm asymptotically tracks the moving equilibrium point with no tracking errors. Simulation results validate the theoretical analysis of our proposed scheme.

In this paper, we investigate a fifth generation cellular vehicle-to-everything based platoon system and propose low-complexity dynamic manager selection and resource allocation algorithms in the finite block length regime, to meet the ultra reliable and low latency communication requirements of intra-platoon safety-related data transmission. A closed-form expression for the platoon manager index is derived, based on which a low-complexity dynamic platoon manager selection algorithm is proposed, enabling a significant communication performance enhancement over the conventional head vehicle platoon manager algorithm. This is also the first work to investigate the impact of finite block length on platoon communication. The optimal finite block length coding rate under automatic repeat request retransmission is obtained. A subchannel multiplexing criterion is proposed and closed-form expressions for the optimal transmission powers of the platoon manager and its subchannel multiplexing partner are derived. Based on the theoretical findings, a joint resource allocation and coding rate optimization algorithm is proposed, achieving a near-optimal performance in terms of groupcast latency with a significant complexity reduction over exhaustive search.

Cloud computing is emerging as an increasingly popular computing paradigm, allowing dynamic scaling of resources available to users as needed. This requires a highly accurate demand prediction and a resource allocation methodology. The existing methodologies for dynamic resource allocation do not provide effective performance isolation between the VM and Artificial Demand Analysis machines since it gets affected by interferences. To overcome these issues, this paper proposes a conceptual model and an effective algorithm to achieve dynamic resource allocation by migrating tasks or requests in VMs. At first, task demands from the multiple users go to the feature extraction process. In feature extraction, features of the user's tasks and cloud server are extracted. Next both features are reduced by using Modified PCA algorithm to reduce the dynamic resource allocation processing time. Finally, both the features are combined and resource allocation is performed using Hybrid Particle Swarm Optimization and Modified Genetic Algorithm (HPSO-MGA). Then the optimized task has been scheduled to particular VM for allocating the resources. The experimental result of the proposed resource allocation methodology indicates better performance when compared with the existing methods Firefly and Krill herd Load Balancing (LB). For 100 VMs the reliability of HPSO-MGA is 0.87 but the exiting krill herd LB and IDSA gives 0.78 and 0.85, which is lower than the proposed one.

In green transportation systems, Vehicular Edge Computing (VEC) is playing a critical role by enabling real-time responses to compute-intensive tasks. Furthermore, incentivizing diverse stakeholders in VEC to share resources and ensuring equitable resource allocation are pivotal to achieving sustainability. In existing studies, a key limitation is that resource price is often simplified to a static decision factor, thus failing to function as a dynamic indicator that reflects real-time load state of the RSUs and the essential balance between the total utility of RSUs and vehicle user experience (QoE). At the same time, the issue is more challenging when facing the load balancing within multi-RSU. To resolve this issue, the paper builds a differentiated resource price model and designs a pricing strategy that combines vehicle QoE and RSUs cost, in order to maximize the total utility of RSUs. Specifically, the paper propose an Incentive-Driven Collaborative Resource Management (IDCRM) algorithm, which integrates the Soft Actor-Critic (SAC) based Computing Resource Allocation and Pricing (SCRAP) algorithm for dynamic resource allocation and pricing by individual RSU, and RSUs cooperative offloading algorithm to address multi-RSU load balancing. The experimental results demonstrate that our proposed IDCRM effectively maximizes the provider’s revenue while ensuring vehicle QoE and significantly alleviates the resource bottleneck of a single RSU through a collaborative mechanism, achieving efficient load balancing across multi-RSU.

The diverse load profile formation and utility preferences of multitype electricity users challenge real-time pricing (RTP) and welfare equilibrium. This paper designs an RTP strategy for smart grids. On the demand side, it constructs utility functions reflecting user characteristics and uses multi-agents for different user interests. Considering industrial users, small-scale microgrids, distributed generation, and battery energy storage systems are included. Based on supply and demand interest, a distributed online multi-agent reinforcement learning (RL) algorithm is proposed. A bi-level stochastic model in the Markov decision process framework optimizes the RTP strategy. Through information exchange, an adaptive pricing scheme balances interest and achieves optimal strategies. Simulation results confirm the effectiveness of the proposed method and algorithm in peak shaving and valley filling. Three load fluctuation scenarios are compared, showing the algorithm's adaptability. The findings reveal the potential of the RL-based bi-level pricing model in resource allocation and user benefits in smart grids. Innovations in user modeling, model construction, and algorithm application have theoretical and practical significance in the electricity market research.

Cloud computing’s simulation and modeling capabilities are crucial for big data analysis in smart grid power; they are the key to finding practical insights, making the grid resilient, and improving energy management. Due to issues with data scalability and real-time analytics, advanced methods are required to extract useful information from the massive, ever-changing datasets produced by smart grids. This research proposed a Dynamic Resource Cloud-based Processing Analytics (DRC-PA), which integrates cloud-based processing and analytics with dynamic resource allocation algorithms. Computational resources must be able to adjust the changing grid circumstances, and DRC-PA ensures that big data analysis can scale as well. The DRC-PA method has several potential uses, including power grid optimization, anomaly detection, demand response, and predictive maintenance. Hence the proposed technique enables smart grids to proactively adjust to changing conditions, boosting resilience and sustainability in the energy ecosystem. A thorough simulation analysis is carried out using realistic circumstances within smart grids to confirm the usefulness of the DRC-PA approach. The methodology is described in the intangible, showing how DRC-PA is more efficient than traditional methods because it is more accurate, scalable, and responsive in real-time. In addition to resolving existing issues, the suggested method changes the face of contemporary energy systems by paving the way for innovations in grid optimization, decision assistance, and energy management.

Resource allocation is an important part of non-orthogonal multiple access technology. How to allocate resources reasonably to achieve high service quality has become an important research topic. The existing resource allocation algorithms have the disadvantages of low system performance improvement, inability to achieve dynamic resource allocation, and high complexity. In view of the limitations of the existing resource allocation methods, this paper proposes to minimize the power consumption of the base station by obtaining the minimum power required by the terminal to reach the target rate under certain fairness constraints, and at the same time obtain the channel state through the base station. The information calculates the actual rate reached by the terminal in real time. If the rate is lower than a certain value, the resources are reallocated to achieve dynamic resource allocation. Finally, the simulation shows that the performance of the proposed algorithm is improved when the terminal is fixed. When the terminal moves, the system performance is greatly improved and the base station energy consumption is reduced.

In this paper we study a minimalist decentralized algorithm for resource allocation in a simplified Grid-like environment. We consider a system consisting of large number of heterogenous reinforcement learning agents that share common resources for their computational needs. There is no communication between the agents: the only information that agents receive is the (expected) completion time of a job it submitted to a particular resource and which serves as a reinforcement signal for the agent. The results of our experiments suggest that reinforcement learning can be used to improve the quality of resource allocation in large scale heterogenous system.

Future wireless communication systems are expected to provide a broad range of multimedia services that have a significant traffic asymmetry between uplink and downlink. The code division multiple access‐time division duplex (CDMA‐TDD) system is a promising solution to cope with the problem of traffic asymmetry. However, the TDD system is subject to inter‐cell interference compared to frequency division duplex (FDD) system. Since both uplink and downlink share the same frequency in TDD, uplink and downlink may interfere each other especially when neighboring cells require different rates of traffic load. Thus, the resource allocation among cells is an important issue in TDD. In this paper, the resource allocation in the CDMA‐TDD is formulated as a mixed integer programming (MIP) problem. A dynamic resource allocation algorithm (DRAA) is provided that effectively solves the traffic asymmetry problem. The MIP problem is also solved by a well‐known branch and bound procedure. Both the crossed slot and non‐crossed slot allocation are examined and compared to the DRAA. Computational result shows that proposed DRAA gives a good performance as the traffic asymmetry increases between the uplink and downlink. Copyright © 2003 John Wiley & Sons, Ltd.

This paper proposes a reinforcement learning (RL) algorithm for the security problem of state estimation of cyber-physical system (CPS) under denial-of-service (DoS) attacks. The security of CPS will inevitably decline when faced with malicious cyber attacks. In order to analyze the impact of cyber attacks on CPS performance, a Kalman filter, as an adaptive state estimation technology, is combined with an RL method to evaluate the issue of system security, where estimation performance is adopted as an evaluation criterion. Then, the transition of estimation error covariance under a DoS attack is described as a Markov decision process, and the RL algorithm could be applied to resolve the optimal countermeasures. Meanwhile, the interactive combat between defender and attacker could be regarded as a two-player zero-sum game, where the Nash equilibrium policy exists but needs to be solved. Considering the energy constraints, the action selection of both sides will be restricted by setting certain cost functions. The proposed RL approach is designed from three different perspectives, including the defender, the attacker and the interactive game of two opposite sides. In addition, the framework of Q-learning and state–action–reward–state–action (SARSA) methods are investigated separately in this paper to analyze the influence of different RL algorithms. The results show that both algorithms obtain the corresponding optimal policy and the Nash equilibrium policy of the zero-sum interactive game. Through comparative analysis of two algorithms, it is verified that the differences between Q-Learning and SARSA could be applied effectively into the secure state estimation in CPS.

This paper considers a joint optimal design of admission control and resource allocation for multimedia services delivery in high‐speed railway (HSR) wireless networks. A stochastic network optimization problem is formulated which aims at maximizing the system utility while stabilizing all transmission queues under the average power constraint. By introducing virtual queues, the original problem is equivalently transformed into a queue stability problem, which can be naturally decomposed into three separate subproblems: utility maximization, admission control, and resource allocation. A threshold‐based admission control strategy is proposed for the admission control subproblem. And a distributed resource allocation scheme is developed for the mixed‐integer resource allocation subproblem with guaranteed global optimality. Then a dynamic admission control and resource allocation algorithm is proposed, which is suitable for distributed implementation. Finally, the performance of the proposed algorithm is evaluated by theoretical analysis and numerical simulations under realistic conditions of HSR wireless networks.

P2P power trading based on reinforcement learning for nanogrid clusters

Nowadays, Device-to-Device (D2D) communication in LTE system has been proposed as a possible candidate for vehicle-to-vehicle(V2V) communication. As V2V communications share the same licensed frequency spectrum pool with typical UEs communications in one LTE cell, efficient radio resource allocation is a challenging issue. In this paper, we proposed a traffic flow based dynamic resource allocation algorithm for vehicular communication networks, which could make radio resource to be utilized in a more efficient way with a flexible node-grouping algorithm. The simulation results show, compared with random allocation and greedy allocation algorithms, the proposed algorithm can significantly reduce running time and significantly improve spectrum utilization efficiency.

Sensor fusion for visual tracking is attractive since the integration of multiple sensors and/or features with different characteristics has potential to improve tracking performance. However, there exist several critical limitations to sensor fusion techniques: (1) the measurement cost increases typically as many times as the number of sensors, (2) it is not straightforward to quantify the confidence of each source and give each sensor a proper weight for state estimation, and (3) there is no principled algorithm for dynamic resource allocation to achieve better performance. We describe a method to combine information from multiple sensors and estimate the current tracker state by using a mixture of sequential Bayesian filters (e.g., particle filter)—one filter for each sensor, where each filter makes a different level of contribution to estimate the combined posterior in a reliable manner. In this framework, multiple sensors interact to determine an appropriate sensor for each particle dynamically; each particle is allocated to only one of the sensors for measurement and a different number of particles may be assigned to each sensor as a result. The level of the contribution of each sensor changes dynamically based on its prior information and relative measurement confidence. We apply this technique to visual tracking problems with multiple cameras or multiple features, and demonstrate its effectiveness through tracking results in real videos.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.