Reinforcement Learning for Server-Aware Offloading in Multi-Tier Multi-Instance Computing Architecture

In this paper, we, for the first time, proposed the Reinforcement Learning (RL) based interconnection design for 3D X-Point array structure considering crosstalk and IR drop. We applied the Markov Decision Process (MDP) to correspond to finding the optimal interconnection design problem to RL problem. We defined interconnection state to the vector, design to the action and the number of bits, crosstalk and IR drop are considered as the reward. The Proximal Policy Optimization (PPO) and Long Short-Term Memory (LSTM) are used to RL algorithms. The proposed interconnection design model is well trained and shows convergence of reward score in 16×16, 32×32 and 64×64 cases. We verified that the trained model finds out optimal interconnection design considering both memory size and signal integrity issues.

Tracking controllers enable robotic systems to accurately follow planned reference trajectories. In particular, reinforcement learning (RL) has shown promise in the synthesis of controllers for systems with complex dynamics and modest online compute budgets. However, the poor sample efficiency of RL and the challenges of reward design make training slow and sometimes unstable, especially for high-dimensional systems. In this work, we leverage the inherent Lie group symmetries of robotic systems with a floating base to mitigate these challenges when learning tracking controllers. We model a general tracking problem as a Markov decision process (MDP) that captures the evolution of both the physical and reference states. Next, we prove that symmetry in the underlying dynamics and running costs leads to an MDP homomorphism, a mapping that allows a policy trained on a lower-dimensional “quotient” MDP to be lifted to an optimal tracking controller for the original system. We compare this symmetry-informed approach to an unstructured baseline, using Proximal Policy Optimization (PPO) to learn tracking controllers for three systems: the Particle (a forced point mass), the Astrobee (a fully-actuated space robot), and the Quadrotor (an underactuated system). Results show that a symmetry-aware approach both accelerates training and reduces tracking error at convergence.

Traditional firewall systems use static rule sets, making them unsuitable for growing cyber threats and network circumstances. In this research, we automate firewall rule development and optimization using Reinforcement Learning (RL) to increase network security and reduce human setup. This paper introduces FireRL, a system that uses RL to help make smart decisions about firewall rules by treating it like a Markov Decision Process (MDP), enabling an RL agent to learn optimal firewall policies from simulated network traffic. This proposed method utilizes the Proximal Policy Optimization (PPO) Actor-Critic algorithm to balance throughput, latency, and threat mitigation via stable, on-policy policy gradient updates. Experiments are performed in benign and dangerous traffic situations. Firewalls outperform static firewalls because the PPO agent quickly reacts to new threats and reduces false positives by directly optimizing the rule setting policy. Resistance to new attack vectors demonstrates the system's flexibility and resilience. This research concludes with a self-optimizing firewall approach that greatly lowers expert-led settings. FireRL's proactive and scalable RL-based defense is ideal for current cybersecurity.

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Automated market makers (AMMs) have revolutionized decentralized finance (DeFi), enabling trustless asset exchange through algorithmic liquidity pools. One notable AMM is Uniswap, which improved capital efficiency by introducing concentrated liquidity, allowing Liquidity Providers (LPs) to allocate capital within specific price ranges. However, this flexibility requires active management to maximize fee generation while effectively addressing impermanent loss (IL) and gascosts. We formulate concentrated liquidity management (CLM) as a stochastic control problem and propose deep reinforcement learning (RL) strategies, specifically Proximal Policy Optimization (PPO) and Deep Q-Network (DQN), to optimize liquidity positioning. Unlike heuristics, RL agents adapt dynamically to market conditions. We adopt a two-stage synthetic-to-historical evaluation strategy, involving RL model training on synthetic data and evaluation on historical data. Results show that training on synthetic data and testing historic data allows RL-based strategies to perform well across regimes, capturing more fees while keeping net performance ahead of a passive buy-and-hold strategy. These findings highlight the robustness and practical viability of such strategies for CLM.

One popular method for optimizing systems, referred to as ANN-PSO, uses an artificial neural network (ANN) to approximate the system and an optimization method like particle swarm optimization (PSO) to select inputs. However, with reinforcement learning developments, it is important to compare ANN-PSO to newer algorithms, like Proximal Policy Optimization (PPO). To investigate ANN-PSO’s and PPO’s performance and applicability, we compare their methodologies, apply them on steady-state economic optimization of a chemical process, and compare their results to a conventional first principles modeling with nonlinear programming (FP-NLP). Our results show that ANN-PSO and PPO achieve profits nearly as high as FP-NLP, but PPO achieves slightly higher profits compared to ANN-PSO. We also find PPO has the fastest computational times, 10 and 10,000 times faster than FP-NLP and ANN-PSO, respectively. However, PPO requires more training data than ANN-PSO to converge to an optimal policy. This case study suggests PPO has better performance as it achieves higher profits and faster online computational times. ANN-PSO shows better applicability with its capability to train on historical operational data and higher training efficiency.

Proximal Policy Optimization (PPO), a popular on-policy deep reinforcement learning method, employs a stochastic policy for exploration. In this paper, we propose a colored noise-based stochastic policy variant of PPO. Previous research highlighted the importance of temporal correlation in action noise for effective exploration in off-policy reinforcement learning. Building on this, we investigate whether correlated noise can also enhance exploration in on-policy methods like PPO. We discovered that correlated noise for action selection improves learning performance and outperforms the currently popular uncorrelated white noise approach in on-policy methods. Unlike off-policy learning, where pink noise was found to be highly effective, we found that a colored noise, intermediate between white and pink, performed best for on-policy learning in PPO. We examined the impact of varying the amount of data collected for each update by modifying the number of parallel simulation environments for data collection and observed that with a larger number of parallel environments, more strongly correlated noise is beneficial. Due to the significant impact and ease of implementation, we recommend switching to correlated noise as the default noise source in PPO.

The rapid advancement of artificial intelligence (AI) has accelerated the semiconductor industry, driving chip performance requirements to unprecedented levels. However, conventional system-on-chip (SoC) designs are increasingly constrained by the physical and economic limits of monolithic scaling, where continued transistor miniaturization yields diminishing returns and rising fabrication costs. To address these challenges, the industry has shifted toward chiplet-based heterogeneous integration and advanced packaging technologies. By interconnecting multiple specialized chiplets through high-density, high-bandwidth interfaces, this modular approach improves scalability, reusability, and cost efficiency while enabling faster time-to-market. However, chiplet-based systems introduce significant design complexity, particularly in signal integrity (SI) and power integrity (PI), where highly constrained interconnects and power delivery networks must meet stringent electrical requirements. Traditional design methodologies rely heavily on expert-driven manual tuning and extensive full-wave electromagnetic simulations using electronic design automation (EDA) tools, resulting in long design cycles that can span weeks or months. To overcome these limitations, this thesis investigates the application of AI, particularly deep reinforcement learning (RL), to automate and optimize chiplet-based system design with a focus on SI and PI challenges in advanced packaging technologies. The research initially explores a supervised learning approach using convolutional neural networks (CNNs) to classify signal impairments from eye diagrams. While effective, this approach reveals a critical dependence on large labeled datasets, which are scarce in SI/PI domains, motivating a shift toward RL for autonomous design space exploration without labeled data. The first major contribution proposes an RL framework for automated power distribution network (PDN) design in a four-chiplet system. Two RL algorithms, Dueling Double Deep Q-Network (DDDQN) and Proximal Policy Optimization (PPO), are evaluated under different reward strategies. Results show that dense analytical rewards based on power domain squareness and bump proximity enable the RL agent to achieve lower self-impedance than human-designed PDNs, ensuring power integrity across 28 power domains. The complete PDN layout, which typically requires three months of manual effort, is generated in under one hour using a single GPU. The second contribution focuses on signal integrity optimization for chiplet interconnects in CoWoS redistribution layers (RDLs). An RL agent is trained using a reward function that balances eye width, voltage transfer function (VTF) crosstalk, and interconnect density. The agent explores design variations across data rates from 4 to 32 GT/s using a simulation environment integrated with commercial EDA tools. Experimental results demonstrate up to 24.67% improvement in eye width and over 22 dB reduction in VTF crosstalk while satisfying UCIe electrical specifications. The training process also generates labeled datasets for future supervised learning. The third contribution introduces an RL-based routing framework for UCIe bridge interconnects in advanced System-in-Package (SiP). Using Proximal Policy Optimization, the framework learns to generate routing layouts that comply with constraints such as ground-signal-ground structures, layer assignment, and congestion avoidance. It supports all UCIe module configurations and produces design rule check-compliant solutions within 100 training episodes, reducing design time from four weeks to less than one hour while improving signal distribution and reducing via usage. In summary, this thesis presents a unified AI-driven methodology for addressing SI and PI challenges in chiplet-based systems. The proposed RL frameworks significantly outperform traditional design approaches in both efficiency and quality, demonstrating the potential of deep RL to enable scalable and intelligent design automation for advanced packaging and chiplet-based architectures.

Optimization of high-speed fixed-wing UAV penetration strategy based on deep reinforcement learning

Currently, helicopter towing operations mainly rely on towing systems and multi-person coordinated control, resulting in low work efficiency and posing a threat to life safety. With the development of artificial intelligence and computer science, automating helicopter towing systems is key to achieving safe and efficient helicopter operations. This paper proposes a helicopter tracking control method based on the Proximal Policy Optimization (PPO) algorithm for the helicopter towing trajectory tracking problem. This paper first improves the PPO algorithm network structure and sets the hyperparameters of the algorithm. Then, a deep learning environment is constructed based on the Markov decision process, and the state space and action space are reasonably set. A reward function is designed based on lateral deviation, longitudinal deviation, and heading deviation. Finally, the PPO algorithm is used to train the trajectory tracking control strategy, and the optimal tracking control strategy is obtained and verified by simulation. The simulation results show that the helicopter tracking control strategy based on the PPO algorithm has a fast convergence speed, and the tracking effect verifies the feasibility and generalization of the algorithm.

This paper proposes a reinforcement learning (RL) based path following strategy for underactuated airships with magnitude and rate saturation. The Markov decision process (MDP) model for the control problem is established. Then an error bounded line-of-sight (LOS) guidance law is investigated to restrain the state space. Subsequently, a proximal policy optimization (PPO) algorithm is employed to approximate the optimal action policy through trial and error. Since the optimal action policy is generated from the action space, the magnitude and rate saturation can be avoided. The simulation results, involving circular, general, broken-line, and anti-wind path following tasks, demonstrate that the proposed control scheme can transfer to new tasks without adaptation, and possesses satisfying real-time performance and robustness.

The optimal execution of large financial orders is a fundamental problem in modern algorithmic trading, necessitating a delicate balance between market impact costs and market volatility risk. Traditional approaches, such as the Almgren-Chriss model, rely on closed-form solutions derived from rigid assumptions regarding market dynamics, which often fail to capture the stochastic and non-linear nature of liquidity in high-frequency environments. This paper presents a novel Reinforcement Learning (RL) framework designed to derive optimal execution strategies under strict slippage and impact constraints. We formulate the execution problem as a Markov Decision Process (MDP) where the agent learns to schedule trade volumes dynamically in response to real-time Limit Order Book (LOB) imbalances and price volatility. Unlike standard approaches that treat market impact as a deterministic penalty, our framework incorporates a stochastic impact model that accounts for transient liquidity resilience. We employ a Proximal Policy Optimization (PPO) algorithm, augmented with a Lagrangian relaxation technique to strictly enforce execution constraints. Our extensive simulation results using historical tick data demonstrate that the proposed RL agent significantly outperforms classic benchmarks such as Time-Weighted Average Price (TWAP) and Volume-Weighted Average Price (VWAP) in reducing Implementation Shortfall (IS), particularly in regimes of high volatility and low liquidity.

The task allocation decision-making architecture for Single Pilot Operation (SPO) represents a critical development direction in aviation to address soaring operational costs and the global pilot shortage. This paper systematically reviews the synergistic enabling mechanisms of Reinforcement Learning (RL) and Deep Learning (DL) within this architecture. DL serves as the perceptual foundation, processing visual information via CNNs and optimizing human-machine interaction through Transformers to achieve efficient multimodal data comprehension and situational awareness. RL functions as the decision core, leveraging methods such as multi-agent Proximal Policy Optimization (PPO) and Deep Q-Network (DQN) to model complex task allocation problems as Markov Decision Processes, enabling dynamic resource scheduling and multi-constraint optimization. Through deep integration in a “perception-decision-optimization” closed-loop, this dual approach significantly enhances the SPO system’s responsiveness, robustness, and safety in high-real-time, high-uncertainty environments. This collaborative mechanism provides critical theoretical foundations and technical pathways for developing next-generation intelligent aviation systems compliant with airworthiness standards and enabling efficient human-machine collaboration.

Rapid expansion of the Internet of Things (IoT) and mobile applications has increased the demand for low latency and efficient computational resource management. Although offering high processing power, cloud computing suffers from network congestion and latency issues, making edge computing a viable alternative. Task offloading in cloud-edge environments is crucial for optimizing resource allocation, reducing delays, and enhancing system performance. For intelligent job offloading in an Edge-Cloud architecture, this study suggests a hybrid reinforcement learning model based on Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO). The model is trained on the datasets simulated in EdgeCloudSim simulator and consisting of tasks, network conditions, computational capabilities of servers, and energy efficiency constraints. The results of the experiment shows that the suggested model considerably reduces task execution time, lowers energy usage, and boosts system efficiency. This work provides a robust deep reinforcement learning-based solution for optimizing task offloading in future edge-cloud computing infrastructures.

This book chapter delves into the application of Proximal Policy Optimization (PPO) for achieving stable decision-making in non-stationary environments. PPO, a widely recognized reinforcement learning algorithm, was examined through the lens of its stability mechanisms, including clipping, adaptive learning rates, entropy regularization, and discount factors. The chapter explores how these elements contribute to maintaining reliable performance in dynamic settings where the environment was continuously evolving. The integration of experience replay with PPO was discussed, emphasizing its role in enhancing stability and sample efficiency. Challenges such as balancing exploration and exploitation, optimizing hyperparameters, and managing computational costs are highlighted. Potential future directions for improving PPO’s adaptability and robustness in non-stationary environments are proposed, including dynamic discounting and novel replay strategies. This chapter provides critical insights into leveraging PPO for real-world applications in uncertain and rapidly changing conditions, making it a valuable resource for researchers and practitioners in reinforcement learning.