System Frequency Control Method Driven by Deep Reinforcement Learning and Customer Satisfaction for Thermostatically Controlled Load

Aiming at the frequency instability and the increase of control cost caused by the uncertainty of the island microgrid, this paper proposes a multi-objective control strategy for the island microgrid based on the Soft Actor Critic (SAC) algorithm. Firstly, considering the influence of the output distribution of micro gas turbines on the control cost, a multi-objective control model that coordinates control cost and frequency stability is constructed. Secondly, in order to cope with the sparse reward task of multi-objective control, this paper takes the frequency deviation and control cost as the state set, takes the unit output as the action set, and designs the corresponding reward function, thus completing the process design of the multi-objective control strategy. Finally, the calculation example shows that the SAC controller can effectively reduce the control cost on the premise of ensuring the frequency regulation requirements of the microgrid, and the comprehensive control effect is far superior to traditional control methods such as PI control and fuzzy control.

Great attention has been attracted to the growing energy consumption of the subway system. It is significant to investigate the energy-efficient train control methodology which is regarded as an effective way to cut down the energy cost of the subway system. In this paper, a soft actor-critic-based method is proposed to determine the optimal driving strategy. Firstly, the inverse problem of the energy-efficient train control problem, i.e., minimizing the trip time with constant traction energy, is presented, and it is converted to a finite Markov decision process which can be solved with deep reinforcement learning (DRL) algorithms. Secondly, the soft actor-critic (SAC) algorithm is proposed to determine the optimal driving strategy. Finally, the effectiveness of the proposed method is verified via the case studies. It is also illustrated that the SAC-based method has a good performance in robustness and stability. Moreover, compared with other DRL algorithms such as the deep Q network (DQN), the SAC-based method saves about 66% training steps on average to achieve the optimum, which reduces the computational resources.

With the continuous progress of UAV technology and the rapid development of mobile edge computing (MEC), the UAV-assisted MEC system has shown great application potential in special fields such as disaster rescue and emergency response. However, traditional deep reinforcement learning (DRL) decision-making methods suffer from limitations such as difficulty in balancing multiple objectives and training convergence when making mixed action space decisions for UAV path planning and task offloading. This article innovatively proposes a hybrid decision framework based on the improved Dynamic Adaptive Genetic Optimization Algorithm (DAGOA) and soft actor–critic with hierarchical action decomposition, an uncertainty-quantified critic ensemble, and adaptive entropy temperature, where DAGOA performs an effective search and optimization in discrete action space, while SAC can perform fine control and adjustment in continuous action space. By combining the above algorithms, the joint optimization of drone path planning and task offloading can be achieved, improving the overall performance of the system. The experimental results show that the framework offers significant advantages in improving system performance, reducing energy consumption, and enhancing task completion efficiency. When the system adopts a hybrid decision framework, the reward score increases by a maximum of 153.53% compared to pure deep reinforcement learning algorithms for decision-making. Moreover, it can achieve an average improvement of 61.09% on the basis of various reinforcement learning algorithms such as proposed SAC, proximal policy optimization (PPO), deep deterministic policy gradient (DDPG), and twin delayed deep deterministic policy gradient (TD3).

By addressing the drawbacks of static optimization techniques, this research seeks to improve the dynamic energy management of thermoelectric generators (TEGs). Finding the best deep reinforcement learning (DRL) algorithm to maximize energy distribution, prolong battery life, and boost system efficiency in the face of the variable conditions present in waste heat recovery systems is the goal. The TEG system was modeled using the Markov decision process and implemented in a computer-simulated environment. Three advanced DRL algorithms were used: soft actor-critic (SAC), proximal policy optimization (PPO), and deep Q-networks (DQN); which were trained to act as intelligent controllers. The performance of each algorithm was systematically evaluated and compared using key metrics, including average cumulative reward, battery health, system efficiency, and a novel metric termed the energy fulfillment rate, which measures the ability to meet demand while storing surplus energy. The comparative analysis revealed a critical trade-off between maximizing performance and ensuring hardware longevity. The SAC algorithm demonstrated the best overall performance, achieving the highest average reward (− 7.03) and energy fulfillment rate (22.84%). However, the A2C, DDPG, and PPO algorithms all achieved a perfect average battery health of 100.00%, highlighting their superior capability for preserving system longevity, albeit with slightly lower rewards. The DQN algorithm consistently showed the least effective performance across all metrics, particularly in maintaining battery health (60.73%). The SAC algorithm is the most suitable of the methods tested for dynamically managing TEG systems. Its underlying principle of maximization of entropy enables a better exploration of control strategies, leading to a better balance between immediate energy dispatch and long-term storage goals. The findings confirm the significant potential of DRL to create efficient and adaptive controllers for renewable energy applications, although further validation of physical hardware is required to confirm real-world viability.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-025-27210-7.

There is widespread belief that firms should pursue superiority in both customer satisfaction and productivity. However, there is reason to believe these two goals are not always compatible. If a firm improves productivity by “downsizing,” it may achieve an increase in productivity in the short-term, but future profitability may be threatened if customer satisfaction is highly dependent on the efforts of personnel. If so, there are potential tradeoffs between customer satisfaction and productivity for industries as diverse as airlines, banking, education, hotels, and restaurants. Managers in these types of service industries, as well as goods industries in which the service component is increasing, need to understand whether or not this is the case. For example, if efforts to improve productivity can actually harm customer satisfaction—and vice-versa—the downsizing of U.S. and European companies should be viewed with concern. It follows that developing a better understanding of how customer satisfaction and productivity relate to one another is of substantial and growing importance, especially in light of expected continued growth in services throughout the world economy. The objective of this paper is to investigate whether there are conditions under which there are tradeoffs between customer satisfaction and productivity. A review of the literature reveals two conflicting viewpoints. One school of thought argues that customer satisfaction and productivity are compatible, as improvements in customer satisfaction can decrease the time andeffort devoted to handling returns, rework, warranties, and complaint management, while at the same time lowering the cost of making future transactions. The second argues that increasing customer satisfaction should increase costs, as doing so often requires efforts to improve product attributes or overall product design. A conceptual framework useful in resolving these contradictory viewpoints is developed. The framework serves, in turn, as a basis for developing a theoretical model relating customer satisfaction and productivity. The model predicts that customer satisfaction and productivity are less likely to be compatible when: 1) customer satisfaction is relatively more dependent on customization—the degree to which the firm's offering is customized to meet heterogeneous customers' needs—as opposed to standardization—the degree to which the firm's offering is reliable, standardized, and free from deficiencies; and 2) when it is difficult (costly) to provide high levels of both customization and standardization simultaneously. To move forward from the model's propositions to the development of testable hypotheses, we argue that services are more likely than goods to have the preceding characteristics. Hence, tradeoffs between customer satisfaction and productivity should be more prevalent for services than for goods. Although this classification is not precise—many services are standardizable and many goods have a service component—it has the advantage of allowing an initial test of the propositions. The empirical work employs a database matching customer-based measures of firm performance with traditional measures of business performance, such as productivity and Return on Investment (ROI). The central feature of this database is the set of customer satisfaction indices provided by the Swedish Customer Satisfaction Barometer (SCSB). The SCSB provides a uniform set of comparable customer-based firm performance measures and offers a unique opportunity to test the study's hypotheses. The findings indicate that the association between changes in customer satisfaction and changes in productivity is positive for goods, but negative for services. In addition, while both customer satisfaction and productivity are positively associated with ROI for goods and services, the interaction between the two is positive for goods but significantly less so for services. Taken together, the findings suggest support for the contention that tradeoffs are more likely for services. Hence, simultaneous attempts to increase both customer satisfaction and productivity are likely to be more challenging in such industries. Of course, this does not imply that such firms should not seek improvements in both productivity and customer satisfaction. For example, appropriate applications of information technology may improve both customer satisfaction and productivity simultaneously. The findings should provide motivation for future research concerning the nature of customer satisfaction and productivity, as well as appropriate strategy and tactics for each one. It is worth emphasizing that this is an issue that is not only important today, but certainly will become even more important in the future. As the growth of services continues and world markets become increasingly competitive, the importance of customer satisfaction will also increase. To compete in such a world, firms must strike the right balance between their efforts to compete efficiently and their efforts to compete effectively.

Due to the excessive degree of freedom of the space flexible manipulator, we can hardly obtain its accurate dynamic model for its motion planning. In this work, we formulate the precise motion control of the free-floating space piecewise constant curvature (FSPCC) continuum manipulator (i.e. space flexible manipulator) as a sparse reward problem in reinforcement learning, and use the Soft Actor-Critic (SAC) algorithm along with the Random Network Distillation (RND) method to train the optimal policies. Firstly, we use the RND method to jointly train a predictive network and a fixed network. The discrepancy between the output values of the two networks is served as an internal reward for the environment. Secondly, the SAC algorithm aims to maximize the expected return and the entropy of the policy. Policies with high entropy will successfully complete the task while acting as randomly as possible. Finally, the internal rewards tend to incentivize the agent to explore more widely for faster convergence of the algorithm. We applied this method to the FSPCC continuum manipulator simulation model and the results demonstrate that the SAC algorithm together with RND method can control the FSPCC continuum manipulator to catch the target quickly, even in the presence of sparse reward.

The aggregate gradation of asphalt mixture is one of the most important factors affecting the service life of asphalt pavement. In order to study the gradation of asphalt mixture with the best comprehensive performance, this study puts forward a new method of mineral aggregate gradation optimization based on the fuzzy analytic hierarchy process and comprehensive evaluation method, aiming at the multi-level and multi-index evaluation of the road performance of asphalt mixture. First, the orthogonal test design method is applied to design nine target gradations with coarse aggregate (>4.75 mm) and fine aggregate (<4.75 mm) of AC-20 (asphalt concrete) mixture serving as two factors and the upper, middle, and lower positions of the gradation curve as three levels, and then the road performance test research is carried out. Second, a comprehensive model for evaluation of road performance of mineral aggregate gradation is established. A fuzzy complementary judgment matrix is constructed, and the index weights of each level and the hierarchical total ranking weight are calculated. Then, the membership function is introduced into the comprehensive evaluation model for the road performance of mineral aggregate gradation, and the membership values of each index of asphalt mixture road performance are obtained. Finally, the fuzzy comprehensive evaluation method is used to find out the comprehensive evaluation value of the road performance of the nine graded asphalt mixtures, and the mineral aggregate gradation is optimized. The research results show that the 1# gradation of the asphalt mixture has the highest comprehensive road performance evaluation value, and the combination of the fuzzy hierarchical analysis process and comprehensive evaluation method can more objectively and comprehensively evaluate the comprehensive road performance of the asphalt mixture and provide a useful reference for the optimal selection of asphalt mixture mineral gradation.

The present study applies a Deep Reinforcement Learning (DRL) algorithm to Active Flow Control (AFC) of a two-dimensional flow around a confined square cylinder. Specifically, the Soft Actor-Critic (SAC) algorithm is employed to modulate the flow of a pair of synthetic jets placed on the upper and lower surfaces of the confined squared cylinder in flow configurations characterized by Re of 100, 200, 300, and 400. The investigation starts with an analysis of the baseline flow in the absence of active control. It is observed that at Re = 100 and Re = 200, the vortex shedding exhibits mono-frequency characteristics. Conversely, at Re = 300 and Re = 400, the vortex shedding is dominated by multiple frequencies, which is indicative of more complex flow features. With the application of the SAC algorithm, we demonstrate the capability of DRL-based control in effectively suppressing vortex shedding, while significantly diminishing drag and fluctuations in lift. Quantitatively, the data-driven active control strategy results in a drag reduction of approximately 14.4%, 26.4%, 38.9%, and 47.0% for Re = 100, 200, 300, and 400, respectively. To understand the underlying control mechanism, we also present detailed flow field comparisons, which showcase the adaptability of DRL in devising distinct control strategies tailored to the dynamic conditions at varying Re. These findings substantiate the ability of DRL to control chaotic, multi-frequency dominated vortex shedding phenomena, underscoring the robustness of DRL in complex AFC problems.

Owing to the unique advantages of low cost and controllability, reconfigurable intelligent surface (RIS) is a promising candidate to address the blockage issue in millimeter wave (mmWave) communication systems, consequently has captured widespread attention in recent years. However, the joint active beamforming and passive beamforming design is an arduous task due to the high computational complexity and the dynamic changes of wireless environment. In this paper, we consider a RIS-assisted multi-user multiple-input single-output (MU-MISO) mmWave system and aim to develop a deep reinforcement learning (DRL) based algorithm to jointly design active hybrid beamformer at the base station (BS) side and passive beamformer at the RIS side. By employing an advanced soft actor-critic (SAC) algorithm, we propose a maximum entropy based DRL algorithm, which can explore more stochastic policies than deterministic policy, to design active analog precoder and passive beamformer simultaneously. Then, the digital precoder is determined by minimum mean square error (MMSE) method. The experimental results demonstrate that our proposed SAC algorithm can achieve better performance compared with conventional optimization algorithm and DRL algorithm.

The large-scale grid connection of photovoltaic power generation results in the power system frequency fluctuation and frequency regulation capacity decline, which endangers the dynamic security of power grid frequency. In order to reduce the frequency deviation of power grid and tap the potential of photovoltaic power generation participating in frequency regulation, a control method of photovoltaic power participating in frequency regulation of power grid is proposed in this paper. The maximum allowable power of photovoltaic power is predicted through system identification, and the load shedding rate is determined based on this, and the output power is corrected according to the system frequency deviation to participate in frequency regulation of power grid. The model based on MATLAB Simulation results shows the effectiveness of the proposed method.

Energy management strategy for electro-hydraulic hybrid electric vehicles considering optimal mode switching: A soft actor-critic approach trained on a multi-modal driving cycle

With the development of artificial intelligence, intelligent communication jamming decision making is an important research direction of cognitive electronic warfare. In this paper, we consider a complex intelligent jamming decision scenario in which both communication parties choose to adjust physical layer parameters to avoid jamming in a non-cooperative scenario and the jammer achieves accurate jamming by interacting with the environment. However, when the situation becomes complex and large in number, traditional reinforcement learning suffers from the problems of failure to converge and a high number of interactions, which are fatal and unrealistic in a real warfare environment. To solve this problem, we propose a deep reinforcement learning based and maximum-entropy-based soft actor-critic (SAC) algorithm. In the proposed algorithm, we add an improved Wolpertinger architecture to the original SAC algorithm in order to reduce the number of interactions and improve the accuracy of the algorithm. The results show that the proposed algorithm shows excellent performance in various scenarios of jamming and achieves accurate, fast, and continuous jamming for both sides of the communication.

Shifting towards a more data-driven culture is a key antecedent of business success in today's digital era. Previous research has paid attention to exploring the influence of marketing analytics on business performance, and rarely examined how the use of marketing analytics influences customer agility and satisfaction. According to the dynamic capabilities view and using previous studies, we developed a conceptual framework to explore the effect of marketing analytics use on customer agility and customer satisfaction. In Study 1, we utilised cross-sectional data collected from 468 managers from various industries. In study 2, we employed longitudinal data in a three-wave longitudinal utilising a cross-lagged panel model. Study 1 indicated that data acquisition and tool acquisition are key drivers in adopting marketing analytics. Marketing analytics use has stronger effect on customer agility when market turbulence is high. They also revealed that the influence of customer agility on customer satisfaction in such conditions is stronger. Study 2 revealed that marketing analytics use at time point T1 has a significant and positive influence on customer agility at time point T2, while customer agility at time point T2 has a significant and positive influence on customer satisfaction at time point T3. These findings indicate strong temporal effects between marketing analytics use, customer agility, and customer satisfaction. The findings suggest that researchers should look beyond direct effects of marketing analytics use and shift their attention on how a marketing analytics can be leveraged to enable and support dynamic capabilities and customer satisfaction.

In the domain of continuous control, deep reinforcement learning (DRL) demonstrates promising results. However, the dependence of DRL on deep neural networks (DNNs) results in the demand for extensive data and increased computational cost. To address this issue, a novel hybrid actor-critic reinforcement learning (RL) framework is introduced. The proposed framework integrates the broad learning system (BLS) with DNN, aiming to merge the strengths of both distinct architectural paradigms. Specifically, the critic network employs BLS for rapid value estimation via ridge regression, while the actor network retains the DNN structure to optimize policy gradients. This hybrid design is generalizable and can enhance existing actor-critic algorithms. To demonstrate its versatility, the proposed framework is integrated into three widely used actor-critic algorithms-deep deterministic policy gradient (DDPG), soft actor-critic (SAC), and twin delayed DDPG (TD3), resulting in BLS-augmented variants. The experimental results reveal that all BLS-enhanced versions surpass their original counterparts in terms of training efficiency and accuracy. These improvements highlight the suitability of the proposed framework for real-time control scenarios, where computational efficiency and rapid adaptation are critical.

This research was designed to examine the effect of using social media technology on improving participation and customer satisfaction in the banking industry due to the mediating role of value perceived by customers of the studied banks. The present study was an applied and descriptive-surveying research in terms of objective and methodology, respectively. The research statistical sample consisted of 235 senior managers of state and private banks in the banking industry of Iran. A questionnaire was employed to collect the research data, and its validity and reliability were con and Cronbach's alpha indices. The structural equation modelling method was used in the SmartPLS software to test the research hypotheses once the data was gathered. The results of testing the hypotheses demonstrated that the use of social media technology has a positive impact on improving participation intention, satisfaction, and value perceived by customers of the studied banks, which directly predicts 0.486, 0.551, and 0.568 of changes related to the participation intention, satisfaction, and perceived value, respectively. The results analysis also revealed that social media technology has a positive impact on participation intention and customer satisfaction through the value perceived by customers. Hence, the use of social media technology through the mediating variable indirectly predicts 0.517 and 0.449 of changes related to participation intention and customer satisfaction, respectively. Accordingly, one may acknowledge that the studied banks can move towards sustainable improvement of customer participation and satisfaction by accurately utilizing social media technology such as social networks aimed at boosting the continuous relationship with their customers as well as improving the capabilities of customer relationship management by related technologies.