Human-Machine Coadaptation Based on Reinforcement Learning with Policy Gradients

SummaryOne desired aspect of microservice architecture is the ability to self‐adapt its own architecture and behavior in response to changes in the operational environment. To achieve the desired high levels of self‐adaptability, this research implements distributed microservice architecture model running a swarm cluster, as informed by the Monitor, Analyze, Plan, and Execute over a shared Knowledge (MAPE‐K) model. The proposed architecture employs multiadaptation agents supported by a centralized controller, which can observe the environment and execute a suitable adaptation action. The adaptation planning is managed by a deep recurrent Q‐learning network (DRQN). It is argued that such integration between DRQN and Markov decision process (MDP) agents in a MAPE‐K model offers distributed microservice architecture with self‐adaptability and high levels of availability and scalability. Integrating DRQN into the adaptation process improves the effectiveness of the adaptation and reduces any adaptation risks, including resource overprovisioning and thrashing. The performance of DRQN is evaluated against deep Q‐learning and policy gradient algorithms, including (1) a deep Q‐learning network (DQN), (2) a dueling DQN (DDQN), (3) a policy gradient neural network, and (4) deep deterministic policy gradient. The DRQN implementation in this paper manages to outperform the aforementioned algorithms in terms of total reward, less adaptation time, lower error rates, plus faster convergence and training time. We strongly believe that DRQN is more suitable for driving the adaptation in distributed services‐oriented architecture and offers better performance than other dynamic decision‐making algorithms.

Lexicographic multi-objective problems, which consist of multiple conflicting subtasks with explicit priorities, are common in real-world applications. Despite the advantages of Reinforcement Learning (RL) in single tasks, extending conventional RL methods to prioritized multiple objectives remains challenging. In particular, traditional Safe RL and Multi-Objective RL (MORL) methods have difficulty enforcing priority orderings efficiently. Therefore, Lexicographic Multi-Objective RL (LMORL) methods have been developed to address these challenges. However, existing LMORL methods either rely on heuristic threshold tuning with prior knowledge or are restricted to discrete domains. To overcome these limitations, we propose Lexicographically Projected Policy Gradient RL (LPPG-RL), a novel LMORL framework which leverages sequential gradient projections to identify feasible policy update directions, thereby enabling LPPG-RL broadly compatible with all policy gradient algorithms in continuous spaces. LPPG-RL reformulates the projection step as an optimization problem, and utilizes Dykstra's projection rather than generic solvers to deliver great speedups, especially for small- to medium-scale instances. In addition, LPPG-RL introduces Subproblem Exploration (SE) to prevent gradient vanishing, accelerate convergence and enhance stability. We provide theoretical guarantees for convergence and establish a lower bound on policy improvement. Finally, through extensive experiments in a 2D navigation environment, we demonstrate the effectiveness of LPPG-RL, showing that it outperforms existing state-of-the-art continuous LMORL methods.

Piezoelectric inkjet printing has important applications in high-precision manufacturing, but precise control of droplet quality faces challenges such as the nonlinear coupling between drive waveform parameters and droplet characteristics, as well as environmental disturbances. To address these issues, this paper proposes a closed-loop droplet control method based on a curriculum deep deterministic policy gradient (DDPG) algorithm, with the standard deviation of droplet volume and velocity as the main optimization objectives. To tackle the complex mapping relationships and boost training efficiency, we first built a high-fidelity simulation environment using a four-layer deep residual network trained on real-world industrial ink droplet data. Following that, we designed a dynamic weight-adaptive composite reward mechanism and integrated a curriculum learning strategy with the Deep Deterministic Policy Gradient (DDPG) algorithm. Relying on our self-developed droplet observation platform, we achieve closed-loop control of the standard deviation of droplet volume and velocity. Experimental results show that this method can precisely control the droplet volume within ±0.1 pl of the target value, while keeping the velocity standard deviation below 40 mm/s. The adjustment process is stable without severe oscillations, and the method demonstrates strong robustness against environmental disturbances such as ink supply pressure. This provides an efficient and practical solution for the automated control of droplet quality in piezoelectric inkjet printing.

The design and implementation of a deep reinforcement learning and quantum finance theory-inspired portfolio investment management system

In today's logistics and delivery landscape, mobile robot delivery systems have attracted considerable attention due to their efficiency and adaptability. Nevertheless, current robotic delivery solutions encounter various obstacles in complex and dynamically changing environments. Traditional algorithms, for instance, struggle with processing high-dimensional and unstructured data, resulting in inefficient adaptation to real-time environmental changes, which compromises accuracy and efficiency in path planning and task execution. Moreover, the lack of robust perception and decision-making mechanisms limits the robots' ability to handle intricate scenarios and fluctuating delivery demands. To tackle these challenges, this paper proposes an optimization approach for mobile robot delivery systems that leverages deep learning. The study initially integrates a spatial attention mechanism within the model, enabling the robot to focus on critical environmental regions and dynamically adjust attention points, thus enhancing obstacle recognition and avoidance in complex settings, which improves navigation accuracy and path planning. Furthermore, the Deep Deterministic Policy Gradient (DDPG) algorithm is utilized to optimize policies, supporting efficient learning in high-dimensional continuous spaces and empowering robots to acquire effective delivery strategies in challenging environments. Finally, an end-to-end optimization approach allows the system to convert sensor inputs directly into control commands, reducing intermediate complexity and minimizing error accumulation, thereby streamlining the system’s structure. Experimental results confirm that the proposed method substantially boosts delivery system performance, excelling in key metrics like path planning accuracy, task efficiency, and system robustness. The successful integration of the spatial attention mechanism with the deep policy gradient algorithm demonstrates a valuable new approach for advancing future robot delivery system optimizations.

Anti-rolling devices are widely used in modern shipboard components. In particular, ship anti-rolling control systems are developed to achieve a wide range of ship speeds and efficient anti-rolling capabilities. However, factors that are challenging to solve accurately, such as strong nonlinearities, a complex working environment, and hydrodynamic system parameters, limit the investigation of the rolling motion of ships at sea. Moreover, current anti-rolling control systems still face several challenges, such as poor nonlinear adaptability and manual parameter adjustment. In this regard, this study developed a dynamic model for a ship anti-rolling system. In addition, based on deep reinforcement learning (DRL), an efficient anti-rolling controller was developed using a deep deterministic policy gradient (DDPG) algorithm. Finally, the developed system was applied to a ship anti-rolling device based on the Magnus effect. The advantages of reinforcement learning adaptive control enable controlling an anti-rolling system under various wave angles, ship speeds, and wavelengths. The results revealed that the anti-rolling efficiency of the intelligent ship anti-rolling control method using the DDPG algorithm surpassed 95% and had fast convergence. This study lays the foundation for developing a DRL anti-rolling controller for full-scale ships.

Hybrid deep reinforcement learning based eco-driving for low-level connected and automated vehicles along signalized corridors

Thermal processes with prolonged and variable delays pose considerable difficulties due to unpredictable system dynamics and external disturbances, often resulting in diminished control effectiveness. This work presents a hybrid control strategy that synthesizes deep reinforcement learning (DRL) strategies with nonlinear model predictive control (NMPC) to improve the robust control performance of a thermal process with a long time delay. In this approach, NMPC cost functions are formulated as learning functions to achieve control objectives in terms of thermal tracking and disturbance rejection, while an actor–critic (AC) reinforcement learning agent dynamically adjusts control actions through an adaptive policy based on the exploration and exploitation of real-time data about the thermal process. Unlike conventional NMPC approaches, the proposed framework removes the need for predefined terminal cost tuning and strict constraint formulations during the control execution at runtime, which are typically required to ensure robust stability. To assess performance, a comparative study was conducted evaluating NMPC against AC-based controllers built upon policy gradient algorithms such as the deep deterministic policy gradient (DDPG) and the twin delayed deep deterministic policy gradient (TD3). The proposed method was experimentally validated using a temperature control laboratory (TCLab) testbed featuring long and varying delays. Results demonstrate that while the NMPC–AC hybrid approach maintains tracking control performance comparable to NMPC, the proposed technique acquires adaptability while tracking and further strengthens robustness in the presence of uncertainties and disturbances under dynamic system conditions. These findings highlight the benefits of integrating DRL with NMPC to enhance reliability in thermal process control and optimize resource efficiency in thermal applications.

Traffic flow optimization in urban environments remains one of the key challenges of modern research, even despite the significant volume of scientific works devoted to this topic. Despite the achievements, this problem still does not have a universal solution that would work effectively in real-world scenarios. One of the main difficulties is the processing of a large array of input data, in particular, traffic data, which constantly comes from sensors installed throughout the urban road network. Traditionally, due to the scale of the task, researchers have focused on the development of systems with localized agents. Such agents usually manage traffic at individual intersections, while their coordination is carried out within the framework of multi-stream agent systems. However, modern approaches take into account the volume and complexity of input data through the use of deep learning methods. In particular, the use of the deep deterministic policy gradient (DDPG) algorithm is proposed, on the basis of which large input data can be processed. As part of the experimental study, a simple intersection model was tested to verify the effectiveness of the approach. The DDPG algorithm performed better in the simple model compared to Q-learning. DDPG provided a reward in the range of 4-4.3 points, while the reward of Q-learning was in the range of 2-4 points. To evaluate the performance of the DDPG approach compared to Q-learning and random timings, the main criterion is the average reward per episode. DDPG and Q-learning achieve similar reward levels, but DDPG shows stable convergence (0.04-0.21 points), while Q-learning remains unstable (0.04-0.43 points). The study of intra-episode performance shows that DDPG achieves improvements mainly closer to the end of the episode. Overall, this algorithm has proven successful for this scenario, and the results obtained can serve as a basis for further improvements and applications in more complex traffic scenarios.

Multi-agent reinforcement learning excels at addressing group intelligent decision-making problems involving sequential decision-making. In particular, in complex, high-dimensional state and action spaces, it imposes higher demands on the reliability, stability, and adaptability of decision algorithms. The reinforcement learning algorithm based on the multi-agent deep strategy gradient incorporates a function approximation method using discriminant networks. However, this can lead to estimation errors when agents evaluate action values, thereby reducing model reliability and stability and resulting in challenging convergence. With the increasing complexity of the environment, there is a decline in the quality of experience collected by the experience playback pool, resulting in low efficiency of the sampling stage and difficulties in algorithm convergence. To address these challenges, we propose an innovative approach called the empirical clustering layer-based multi-agent dual dueling policy gradient (ECL-MAD3PG) algorithm. Experimental results demonstrate that our ECL-MAD3PG algorithm outperforms other methods in various complex environments, demonstrating a remarkable 9.1% improvement in mission completion compared to MADDPG within the context of complex UAV cooperative combat scenarios.

The linear quadratic regulator (LQR) problem has reemerged as an important theoretical benchmark for reinforcement learning-based control of complex dynamical systems with continuous state and action spaces. In contrast with nearly all recent work in this area, we consider multiplicative noise models, which are increasingly relevant because they explicitly incorporate inherent uncertainty and variation in the system dynamics and thereby improve robustness properties of the controller. Robustness is a critical and poorly understood issue in reinforcement learning; existing methods which do not account for uncertainty can converge to fragile policies or fail to converge at all. Additionally, intentional injection of multiplicative noise into learning algorithms can enhance robustness of policies, as observed in <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ad hoc</i> work on domain randomization. Although policy gradient algorithms require optimization of a nonconvex cost function, we show that the multiplicative noise LQR cost has a special property called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">gradient domination</i> , which is exploited to prove global convergence of policy gradient algorithms to the globally optimum control policy with polynomial dependence on problem parameters. Results are provided both in the model-known and model-unknown settings where samples of system trajectories are used to estimate policy gradients

The research on autonomous driving based on deep reinforcement learning algorithms is a research hotspot. Traditional autonomous driving requires human involvement, and the autonomous driving algorithms based on supervised learning must be trained in advance using human experience. To deal with autonomous driving problems, this paper proposes an improved end-to-end deep deterministic policy gradient (DDPG) algorithm based on the convolutional block attention mechanism, and it is called multi-input attention prioritized deep deterministic policy gradient algorithm (MAPDDPG). Both the actor network and the critic network of the model have the same structure with symmetry. Meanwhile, the attention mechanism is introduced to help the vehicles focus on useful environmental information. The experiments are conducted in the open racing car simulator (TORCS)and the results of five experiment runs on the test tracks are averaged to obtain the final result. Compared with the state-of-the-art algorithm, the maximum reward increases from 62,207 to 116,347, and the average speed increases from 135 km/h to 193 km/h, while the number of success episodes to complete a circle increases from 96 to 147. Also, the variance of the distance from the vehicle to the center of the road is compared, and the result indicates that the variance of the DDPG is 0.6 m while that of the MAPDDPG is only 0.2 m. The above results indicate that the proposed MAPDDPG achieves excellent performance.

As an efficient mode of public transportation, urban rail transit promotes sustainable development by reducing energy consumption and improving service quality. Numerous studies have been conducted on this subject. However, many existing methods struggle to adapt to environmental changes due to the complexity of actual line conditions. Therefore, based on deep deterministic policy gradient (DDPG) and genetic algorithm (GA), a new optimization algorithm GA-DDPG is proposed in this paper. This algorithm utilizes DDPG for continuous action control of trains and employs GA to optimize the noise generation hyperparameter of DDPG, thereby enhancing the learning and adaptive capabilities of the algorithm. Firstly, a reference system is established based on expert knowledge to ensure safe train operation and guide the training of the algorithm model. Then, under the intervention of reference system, the GA-DDPG is used to optimize and adjust the train operation control strategy to achieve a balance in punctuality, energy efficiency, and comfort. Finally, simulation experiment shows that GA-DDPG achieves at least 5.2% energy savings and 51.2% passenger comfort compared to DDPG. More experiments further demonstrate that GA-DDPG possesses better adaptability and robustness than DDPG when the environment changes.

Deep Deterministic Policy Gradient (DDPG) algorithm has been successful for state-of-the-art performance in high-dimensional continuous control tasks. However, due to the complexity and randomness of the environment, DDPG tends to suffer from inefficient exploration and unstable training. In this work, we propose Self-Adaptive Double Bootstrapped DDPG (SOUP), an algorithm that extends DDPG to bootstrapped actor-critic architecture. SOUP improves the efficiency of exploration by multiple actor heads capturing more potential actions and multiple critic heads evaluating more reasonable Q-values collaboratively. The crux of double bootstrapped architecture is to tackle the fluctuations in performance, caused by multiple heads of spotty capacity varying throughout training. To alleviate the instability, a self-adaptive confidence mechanism is introduced to dynamically adjust the weights of bootstrapped heads and enhance the ensemble performance effectively and efficiently. We demonstrate that SOUP achieves faster learning by at least 45% while improving cumulative reward and stability substantially in comparison to vanilla DDPG on OpenAI Gym's MuJoCo environments.

This paper presents the analyses of a deep reinforcement learning, DRL approach for maximum power point tracking (MPPT) of a photovoltaic PV boost converter system. The DRL controller used is the Deep Deterministic Policy Gradient (DDPG) algorithm, which is designed to address the limitations of conventional perturb and observe, P&amp;O methods with PID control, such as reduced efficiency under non – uniform irradiance or load variations. The DDPG framework offers an adaptive and computationally efficient alternative by learning the nonlinear mapping between PV states and duty cycle control. Thus, a comparative study of the DDPG with the conventional approach is performed under non – uniform irradiance and varying load resistance. Parameters of power and voltage were evaluated as well as the oscillatory performance for both controllers. The outcome of the results shows that the DDPG controller can achieve higher power extraction with significantly reduced under higher load impedances The conventional approach, however, achieves a better performance under uniform irradiation and low resistance conditions. This is due to the absence of multiple peaks that allows the conventional approach to rapidly converge to the MPP, while the exploratory nature DDPG introduces minor bias due to its exploratory learning process. These findings highlight the potential of DDPG to enhance the efficiency and adaptability of MPPT, particularly relevant in standalone DC microgrid applications.