Combined DR Pricing and Voltage Control Using Reinforcement Learning Based Multi-Agents and Load Forecasting

Q-learning solution for optimal consensus control of discrete-time multiagent systems using reinforcement learning

The peak-shaving efficiency analysis of natural gas time-of-use pricing for residential consumers: Evidence from multi-agent simulation

Short-term load forecasting is viewed as one promising technology for demand prediction under the most critical inputs for the promising arrangement of power plant units. Thus, it is imperative to present new incentive methods to motivate such power system operations for electricity management. This paper proposes an approach for short-term electric load forecasting using long short-term memory networks and an improved sine cosine algorithm called MetaREC. First, using long short-term memory networks for a special kind of recurrent neural network, the dispatching commands have the characteristics of storing and transmitting both long-term and short-term memories. Next, four important parameters are determined using the sine cosine algorithm base on a logistic chaos operator and multilevel modulation factor to overcome the inaccuracy of long short-term memory networks prediction, in terms of the manual selection of parameter values. Moreover, the performance of the MetaREC method outperforms others with regard to convergence accuracy and convergence speed on a variety of test functions. Finally, our analysis is extended to the scenario of the MetaREC_long short-term memory with back propagation neural network, long short-term memory networks with default parameters, long short-term memory networks with the conventional sine-cosine algorithm, and long short-term memory networks with whale optimization for power load forecasting on a real electric load dataset. Simulation results demonstrate that the multiple forecasts with MetaREC_long short-term memory can effectively incentivize the high accuracy and stability for short-term power load forecasting.

This paper surveys a new method to reduce the infected cells and free virus particles (virions) via a nonlinear HIV model. Three scenarios are considered for control performance evaluation. At first, the system and initial conditions are considered known completely. In the second case, the initial conditions are taken randomly. In the third scenario, in addition to uncertainty in initial condition, an additive noise is taken into account. The optimal control method is used to design an effective drug-schedule to reduce the number of infected cells and free virions with and without uncertainty. By using the Q-learning algorithm, which is the most applicable algorithm in reinforcement learning, the drug delivery rate is obtained off-line. Since Q-learning is a model-free algorithm, it is expected that the performance of the control in the presence of uncertainty does not change significantly. Simulation results confirm that the proposed control method has a good performance and high functionality in controlling the free virions for both certain and uncertain HIV models.

In recent years, with the advance of Artificial Intelligence, automatic music composition has been demonstrated. However, there are many music genres and music instruments. For a same piece of music, different music instruments would produce different effects. Invented some 2500 years ago, Guzheng is one of the oldest music instruments in China and the world. It has distinct timbres and patterns that cannot be duplicated by other music instruments. Therefore, it is interesting to see whether AI can compose Guzheng music or alike. In this paper we present a method that can automatically compose and play Guzheng music. First, we collect a number of existing Guzheng music pieces and convert them into Music Instrument Digital Interface format. Second, we use these data to train a Long Short-Term Memory (LSTM) network and use the trained network to generate new Guzheng music pieces. Next, we use the Reinforcement Learning to optimize the LSTM network by adding special Guzheng playing techniques. Comparing to the existing AI methods, such as LSTM and Generative Adversary Network, our new method is more effective in capturing the characteristics of Guzheng music. According to the evaluations from skilled Guzheng players and general audiences, our Guzheng music is very close to the real Guzheng music. The presented method can also be used to automatically compose the music of other Chinese music instruments.

In this paper we show the application of multi-agent modeling and simulation with distributed reinforcement learning to one of the major problems in power system operations, i.e. voltage control. In this research some agents in the power network work together to provide a desirable voltage profile, using a combination of multi-agent system (MAS) technology and some of the reinforcement learning approaches. In this schema, individual agents who are assigned to voltage controller devices in the power system learn from their experiences to control the system voltage, and also cooperate and communicate with each other to satisfy the whole team goals. A detailed evaluation of methods for controlling voltage in power systems, including multi-agent coordination and distributed reinforcement learning (DRL), demonstrates that this framework yields effective plans, good agent coordination, and successful implementation. In the proposed approach, agent development and communication simulation have been carried out in the Java Agent Development (JADE) framework.

This paper proposes an intelligent power load forecasting method based on reinforcement learning, aiming to address the limitations of traditional forecasting techniques in coping with the integration of high-proportion renewable energy and complex electricity demand. The study first identifies key influencing factors such as temperature and humidity through Spearman correlation analysis and then constructs an innovative model architecture that integrates the Long Short-Term Memory (LSTM) network and Deep Deterministic Policy Gradient (DDPG). The model utilizes the LSTM network to extract load time-series features, achieves dynamic optimization through the DDPG’s reinforcement learning mechanism, and introduces an adaptive reward function and curriculum learning strategy to enhance model performance. Experiments on the power dataset of Tempe Campus at Arizona State University show that the proposed method significantly outperforms traditional LSTM and DDPG models in terms of indicators such as root mean square error (32.4 kW), mean absolute percentage error (2.7%), and R2 (0.963), especially demonstrating stronger adaptability in complex scenarios such as load mutations and holidays. The research findings provide a new technical solution for high-precision load forecasting in smart grid environments.

P2P power trading based on reinforcement learning for nanogrid clusters

In multi-agent robotic systems, the overlap of actions selected by each agent results in poor cooperation, while, at the same time, conventional reinforcement learning entails a large computational cost because every agent must learn. In this paper, we propose a novel method to solve these problems. The agent uses a rule set to conduct its behavior. Our system consists of reinforcement learning with an Action Selection Priority Level (ASPL) module and a generalized rules module. Using the ASPL, the reinforcement learning module chooses a proper cooperative behavior while the generalized rule module can accelerate the learning process. By applying the proposed method to robot soccer, the learning process can be accelerated by reducing the search space. The results of simulation and real experiments indicate the effectiveness of the proposed method.

In the deregulated energy market, the accuracy of load forecasting has a significant effect on the planning and operational decision making of utility companies. Electric load is a random non-stationary process influenced by a number of factors which make it difficult to model. To achieve better forecasting accuracy, a wide variety of models have been proposed. These models are based on different mathematical methods and offer different features. This paper presents a new two-stage approach for short-term electrical load forecasting based on least-squares support vector machines. With the aim of improving forecasting accuracy, one more feature was added to the model feature set, the next day average load demand. As this feature is unknown for one day ahead, in the first stage, forecasting of the next day average load demand is done and then used in the model in the second stage for next day hourly load forecasting. The effectiveness of the presented model is shown on the real data of the ISO New England electricity market. The obtained results confirm the validity advantage of the proposed approach.

The accurate prediction of power system load is extremely important for the operation of the power market and the safe operation of the power grid. In order to improve the accuracy of short-term load forecasting of power systems, a combination model based on long short term memory network (LSTM) and light gradient boosting machine (LightGBM) is proposed. The experiment first decomposes historical load data by EMD, uses historical weather data and load data decomposed by EMD to establish LSTM prediction model and LightGBM prediction model respectively, and then these two predicted values are linearly combined to obtain the final predicted value. The electrical load data of the 2016 Electrician Mathematical Contest in Modeling is used as an example to verify. The experimental results show that the LSTM-LightGBM combined model has higher forecasting accuracy and application prospects for power load forecasting than traditional load forecasting methods and standard LSTM and LightGBM load forecasting methods.

In this paper, path planning and sorting of packages for Omnidirectional-Wheel conveyor are presented using Reinforcement Learning (RL). Q-learning, Double Q-learning, Deep Q-learning, and the Double Deep Q-learning algorithms are investigated. The RL algorithms enable the conveyor to self-learn the packages path and sort them without using conventional control or path planning theories. The RL algorithms are used for two different case studies on conveyors structures with different numbers of cells to compare and evaluate their performances in large- and small-scale sized structures. To explore the proposed methods response to external environment effects, two types of collisions between multiple packages were considered, the proposed RL algorithms showed their ability to resolve both types successfully. Comparative study between multiple RL algorithms for path planning showed that the Q-learning and Double Q-learning algorithms had outperformed their Deep learning versions for path planning in the two case studies. Furthermore, the proposed RL methods are compared experimentally to classic control and path planning theories using a hardware prototype for one of the presented case studies. The hardware experimental results showed that the proposed RL methods were as successful as the conventional methods in path planning and sorting in much less processing time. Two types of sorting scenarios (Type I and II) were tested for same package type and for multiple ones. For Type I sorting the Q-learning algorithm performed better than the Q-learning with weights approach, achieving better mean and minimum rewards while maximum rewards remain the same for both techniques. As for Type II sorting, only the Q-learning with weights approach was able to achieve it and converge in a reasonable time.

Currently many non-tractable considered problems have been solved satisfactorily through methods of approximate optimization called metaheuristic. These methods use non-deterministic approaches that find good solutions which, however, do not guarantee the determination of the global optimum. The success of a metaheuristic is conditioned by capacity to adequately alternate between exploration and exploitation of the solution space. A way to guide such algorithms while searching for better solutions is supplying them with more knowledge of the solution space (environment of the problem). This can to be made in terms of a mapping of such environment in states and actions using Reinforcement Learning. This paper proposes the use of a technique of Reinforcement Learning - Q-Learning Algorithm - for the constructive phase of GRASP and Reactive GRASP metaheuristic. The proposed methods will be applied to the symmetrical traveling salesman problem.

The load of power grids in high load density areas fluctuates violently, and traditional prediction models are difficult to capture dynamic changes, resulting in insufficient prediction accuracy and scheduling efficiency. This paper constructs a joint optimization model that integrates the improved bidirectional long short-term memory network (BiLSTM) and deep reinforcement learning (DRL), taking advantage of bidirectional learning and dynamic optimization to improve load forecasting accuracy and dynamic scheduling capabilities. Multi-source heterogeneous load data is integrated through data cleaning and standardization and normalization technology to enhance the consistency of input data. For the BiLSTM model, the optimized network structurethe and attention mechanism is introduced to improve the ability to capture key features of historical loads, and multi-layer perceptrons are combind to enhance nonlinear feature extraction. The dynamic feedback mechanism of DRL is further used to adjust the load scheduling strategy in real-time to achieve collaborative iteration of prediction and optimization. The experimental results show that the improved model has superior performance in short-term to long-term load forecasting, with a mean absolute error (MAE) of 0.05-0.12, a mean square error (MSE) of 0.005-0.025, and a determination coefficient () of 0.97-0.99, which is more accurate than the traditional model. In the 100% high load scenario, the scheduling time is reduced by 6 minutes after joint training, and the energy loss rate is reduced by 1.4%, significantly optimizing the operation efficiency of the power grid. This study provides a high-precision, low-latency solution for the intelligent scheduling of power grids in high-load density areas.

The internet of robotic things (IoRT) is an emerging technology that combines user equipment (UE) by allowing communications among each other and data transmission with existing communications and network protocols. However, current IoRT network topologies and resources are insufficient to handle this massive data flow and meet the quality of service (QoS) requirements due to the rapid increment of connected UEs. Hence, the most crucial challenge is radio resource management by controlling the emitting power of the antenna called power allocation (PA), considering the interfering multiple access channel (IMAC). In this paper, we propose a data-driven and model-free twin delayed deep deterministic policy gradient (TD3) algorithm which controls the continuous power level of the PA. TD3 is a modified algorithm of deep deterministic policy gradient (DDPG) that consists of six networks: two actors (one for model and the other for target) and four critics (two for models and two for targets) networks. Results show that the proposed TD3 algorithm outperforms the model-based methods such as fractional programming (FP) and weighted MMSE (WMMSE) as well as model-free algorithms, for example, deep Q network (DQN) and DDPG on sum-rate performance with good generalization power.