DRISHTI-AI: Drone-based RL System for Intelligent Human Tracking and Identification

Proximal Policy Optimization (PPO), a popular on-policy deep reinforcement learning method, employs a stochastic policy for exploration. In this paper, we propose a colored noise-based stochastic policy variant of PPO. Previous research highlighted the importance of temporal correlation in action noise for effective exploration in off-policy reinforcement learning. Building on this, we investigate whether correlated noise can also enhance exploration in on-policy methods like PPO. We discovered that correlated noise for action selection improves learning performance and outperforms the currently popular uncorrelated white noise approach in on-policy methods. Unlike off-policy learning, where pink noise was found to be highly effective, we found that a colored noise, intermediate between white and pink, performed best for on-policy learning in PPO. We examined the impact of varying the amount of data collected for each update by modifying the number of parallel simulation environments for data collection and observed that with a larger number of parallel environments, more strongly correlated noise is beneficial. Due to the significant impact and ease of implementation, we recommend switching to correlated noise as the default noise source in PPO.

The rapid advancement of artificial intelligence (AI) has accelerated the semiconductor industry, driving chip performance requirements to unprecedented levels. However, conventional system-on-chip (SoC) designs are increasingly constrained by the physical and economic limits of monolithic scaling, where continued transistor miniaturization yields diminishing returns and rising fabrication costs. To address these challenges, the industry has shifted toward chiplet-based heterogeneous integration and advanced packaging technologies. By interconnecting multiple specialized chiplets through high-density, high-bandwidth interfaces, this modular approach improves scalability, reusability, and cost efficiency while enabling faster time-to-market. However, chiplet-based systems introduce significant design complexity, particularly in signal integrity (SI) and power integrity (PI), where highly constrained interconnects and power delivery networks must meet stringent electrical requirements. Traditional design methodologies rely heavily on expert-driven manual tuning and extensive full-wave electromagnetic simulations using electronic design automation (EDA) tools, resulting in long design cycles that can span weeks or months. To overcome these limitations, this thesis investigates the application of AI, particularly deep reinforcement learning (RL), to automate and optimize chiplet-based system design with a focus on SI and PI challenges in advanced packaging technologies. The research initially explores a supervised learning approach using convolutional neural networks (CNNs) to classify signal impairments from eye diagrams. While effective, this approach reveals a critical dependence on large labeled datasets, which are scarce in SI/PI domains, motivating a shift toward RL for autonomous design space exploration without labeled data. The first major contribution proposes an RL framework for automated power distribution network (PDN) design in a four-chiplet system. Two RL algorithms, Dueling Double Deep Q-Network (DDDQN) and Proximal Policy Optimization (PPO), are evaluated under different reward strategies. Results show that dense analytical rewards based on power domain squareness and bump proximity enable the RL agent to achieve lower self-impedance than human-designed PDNs, ensuring power integrity across 28 power domains. The complete PDN layout, which typically requires three months of manual effort, is generated in under one hour using a single GPU. The second contribution focuses on signal integrity optimization for chiplet interconnects in CoWoS redistribution layers (RDLs). An RL agent is trained using a reward function that balances eye width, voltage transfer function (VTF) crosstalk, and interconnect density. The agent explores design variations across data rates from 4 to 32 GT/s using a simulation environment integrated with commercial EDA tools. Experimental results demonstrate up to 24.67% improvement in eye width and over 22 dB reduction in VTF crosstalk while satisfying UCIe electrical specifications. The training process also generates labeled datasets for future supervised learning. The third contribution introduces an RL-based routing framework for UCIe bridge interconnects in advanced System-in-Package (SiP). Using Proximal Policy Optimization, the framework learns to generate routing layouts that comply with constraints such as ground-signal-ground structures, layer assignment, and congestion avoidance. It supports all UCIe module configurations and produces design rule check-compliant solutions within 100 training episodes, reducing design time from four weeks to less than one hour while improving signal distribution and reducing via usage. In summary, this thesis presents a unified AI-driven methodology for addressing SI and PI challenges in chiplet-based systems. The proposed RL frameworks significantly outperform traditional design approaches in both efficiency and quality, demonstrating the potential of deep RL to enable scalable and intelligent design automation for advanced packaging and chiplet-based architectures.

The adoption of novel technologies such as Unmanned Aerial Vehicles (UAVs) in Search and Rescue (SAR) operations remains limited. As a result, their full potential is not yet realized. Although UAVs have been deployed on an ad hoc basis, typically under manual control by dedicated operators, assisted and fully autonomous configurations remain largely unexplored. In this study, three SAR frameworks are systematically evaluated within a unified benchmarking framework: conventional ground missions, UAV-assisted missions, and fully autonomous UAV operations. As the key performance indicator, the target localization time was quantified and used as the means of comparison amongst frameworks. The conventional and assisted frameworks were experimentally tested through physical hardware in a controlled outdoor setting, wherein simulated callouts occurred via rescue teams. The autonomous swarm framework was simulated in the form of a multi-agent Reinforcement Learning (RL) method via the use of the Proximal Policy Optimization (PPO) algorithm. This enabled the optimization of the decentralized cooperative actions that could occur for efficient exploration of a partially observed three-dimensional environment. Our results demonstrated that the autonomous swarm significantly outperformed the conventional and assisted approaches in terms of speed and coverage. Finally, a detailed depiction of the framework’s integration into an operational system is provided.

Aiming at the problem of multi-objective path planning (MOPP) and dynamic obstacle avoidance of manipulator in dynamic environment, this paper proposes a solution based on hierarchical reinforcement learning (HRL) framework. Traditional path planning methods have some problems in dynamic scenes, such as poor real-time performance, difficult to balance multi-objective conflicts and insufficient adaptability to environmental changes. Therefore, this paper designs a two-tier architecture including global path planning layer and local obstacle avoidance layer, which is trained by Proximal policy optimization (PPO) and Soft Actor-Critic (SAC) algorithms respectively, and achieves collaborative optimization through an efficient information exchange mechanism between layers. At the same time, a multi-objective reward function based on dynamic weight adjustment strategy is introduced. Fuzzy logic is used to adaptively balance the relationship among path length, obstacle avoidance safety and energy consumption according to environmental complexity. Combined with Long Short-Term Memory (LSTM), the trajectory of obstacles is predicted, and the potential field method is further introduced to modify the obstacle avoidance reward, which improves the real-time response ability and robustness of the algorithm in dynamic environment. The experimental results show that the HRL-SAC-PPO method proposed in this paper shows superior performance in both static and dynamic scenarios. In the static scene, the success rate of this method reaches 100%, the average path length is shortened to 2.13m, no collision occurs, and the energy consumption is reduced to 1.12 kJ, which shows a good multi-objective optimization effect. In the dynamic scene, the trajectory error of obstacles predicted by LSTM is only 4.2%, and the safe distance between the robot arm and obstacles is improved by 35%, which significantly enhances the reliability of obstacle avoidance. In addition, the average decision delay of this method is only 11.3ms, and the peak delay is 23ms, which is much lower than that of the contrast algorithm, showing stronger real-time response ability. The ablation experiment further verified the key role of LSTM trajectory prediction, dynamic weight adjustment and layered structure on the overall performance. In the welding task verification of the real UR5 manipulator, the success rate of the system in dynamic environment is 95.3%, the average path length is 3.41m, and the maximum joint acceleration is only 0.87 radian/s², which is far below the safety threshold, indicating that the algorithm has good stability and obstacle avoidance ability in practical application. The comprehensive performance comparison shows that this method performs well in different industrial scenarios, and has stronger environmental adaptability and comprehensive path planning ability.

Magnetic levitation (Maglev) technology is widely adopted for transportation, photolithography, clean room, and material-handling applications, owing to its vibration- and friction-free, untethered motion. The magnetic movers are actuated using the stator coil array, which functions as mobile maglev robots (MLRs) with precision position and orientation motion control. Therefore, this article studies the MLRs for intelligent manufacturing, such as material handling and collaboration with manufacturing components, by integrating them with deep reinforcement learning (DRL) policy controllers. Linear DRL policies are developed and compared between Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO) for the horizontal motion control, which takes 455 μs to compute the actuating forces for real-time operations. During policy training, random magnitude horizontal step responses are ticked every 500 ms to explore and exploit the dynamic capacity of the MLRs. The PPO policies are finalized based on the discovered deterministic nature of the maglev environment. Furthermore, the symmetry of the MLR configuration is utilized for policy training, with a threshold boundary included to enable MLRs to translate smoothly across the horizontal operating range. The trained policies exhibit aggressive oscillations and persistent offsets relative to references, respectively. To alleviate the fitting problem, distance-based soft state machines are introduced to integrate the DRL policy with the basic controller. The state machine parameters are empirically tuned using various location step responses. This feature enables the dynamic responses of the DRL policy to be combined with the precision responses at references. The soft state machines are also trained as auxiliary policy actions and evaluated using full operating-range trajectory tracking and long-stroke step responses alongside the policy and the basic controller. The DRL policy controllers outperform the basic controllers, with improved rise and settling times in step responses, while showing enhanced performance in tracking trajectories. Finally, the policy robustness is empirically evaluated.

Epidemics of infectious diseases are an important threat to public health and global economies. Yet, the development of prevention strategies remains a challenging process. For this reason, we investigate a deep reinforcement learning approach to automatically learn prevention strategies in an epidemiological model, in the context of pandemic influenza. To this end, we construct a new epidemiological meta-population model, with 379 patches, that balances between model complexity and computational efficiency such that the use of reinforcement learning techniques becomes attainable. First, we set up a ground truth such that we can evaluate the performance of the ""Proximal Policy Optimization"" algorithm to learn in a single district of this epidemiological model. Next, we consider a larger scale problem, by conducting an experiment where we aim to learn a joint policy to control the districts in a community of 11 tightly coupled districts, for which no ground truth can be established. This experiment shows that deep reinforcement learning can be used to learn mitigation policies in complex epidemiological models with a large state space. Moreover, through this experiment, we demonstrate that there can be an advantage to consider collaboration between districts when designing prevention strategies.

This paper proposes an adaptive level-generation framework that integrates Proximal Policy Optimization (PPO) with the Token-based One-shot Arbitrary Dimension GAN (TOAD-GAN) to achieve dynamic difficulty adjustment in platformer games. The PPO agent maps player-behavior metrics to difficulty parameters that condition TOAD-GAN through a post-generation imputation module. The approach avoids retraining the generator by modifying enemy density, power-up distribution, and time limits directly on the tokenized output. Experiments show that PPO trained with a difficulty-margin simulator converges substantially better than a random simulator baseline (average reward −136.7 vs. −2260.8). Structural fidelity is preserved, with TPKL-Divergence increasing only modestly (0.47 → 0.57 after difficulty imputation). A rule-based playability repair module further improves solvability from 65% to 80% across 100 generated levels. Results demonstrate that RL-driven conditioning can effectively guide token-based generators toward producing adaptive and playable content.

Optimizing the fuel cycle cost through the optimization of nuclear reactor core loading patterns (LPs) involves multiple objectives and constraints, leading to a vast number of candidate solutions that cannot be explicitly solved. To advance the state of the art in core reload patterns, we have developed methods based on deep Reinforcement Learning (RL) for both single- and multi-objective optimization. Our previous research laid the groundwork for these approaches and demonstrated their ability to discover high-quality patterns within a reasonable time frame. On the other hand, Stochastic Optimization (SO) approaches are commonly used in the literature, but there is no rigorous explanation that shows which approach is better in which scenario. In this paper, we demonstrate the advantage of our RL-based approach, specifically using Proximal Policy Optimization (PPO) against the most commonly used SO-based methods: Genetic Algorithm, Parallel Simulated Annealing with mixing of states, and Tabu Search, as well as an ensemble-based method, i.e. the Prioritized replay Evolutionary and Swarm Algorithm. We found that the LP scenarios derived in this paper are amenable to a global search to identify promising research directions rapidly but then need to transition into a local search to exploit these directions efficiently and prevent getting stuck in local optima. PPO adapts its search capability via a policy with learnable weights, allowing it to function as both a global search method and a local search method. Subsequently, we compared all algorithms against PPO in long runs, which exacerbated the differences seen in the shorter cases. Overall, the work demonstrates the statistical superiority of PPO compared to the other considered algorithms.

This study aims to systematically compare the performance of two deep reinforcement learning algorithms – Proximal Policy Optimization (PPO) and Deep Q-Network (DQN) – across different game environments. To achieve this, eight distinct test environments from the OpenAI Gymnasium library (CartPole-v1, FrozenLake-v1, LunarLander-v3, Taxi-v3, MountainCar-v0, Blackjack-v1, CliffWalking-v0, and Acrobot-v1) were utilized. Each environment was trained over 1,000,000 timesteps. For each algorithm, key performance metrics such as average reward, training time, standard deviation, success rate, and the highest and lowest reward values were calculated and visualized through graphs. Additionally, the strengths and weaknesses of the algorithms in different environments were analyzed. The results indicate that PPO performs more consistently and effectively in tasks requiring continuous actions, whereas DQN achieves faster and more reliable outcomes in deterministic environments with discrete action spaces. This study provides meaningful insights by comparing the performance of PPO and DQN under identical conditions, while most prior research has examined these algorithms separately.

A bio-inspired neuro-adaptive deep reinforcement learning approach for real-time solar tracking system to enhance photovoltaic efficiency

Despite the fact that deep reinforcement learning (RL) has surpassed human-level performances in various tasks, it still has several fundamental challenges. First, most RL methods require intensive data from the exploration of the environment to achieve satisfactory performance. Second, the use of neural networks in RL renders it hard to interpret the internals of the system in a way that humans can understand. To address these two challenges, we propose a framework that enables an RL agent to reason over its exploration process and distill high-level knowledge for effectively guiding its future explorations. Specifically, we propose a novel RL algorithm that learns high-level knowledge in the form of a finite reward automaton by using the L* learning algorithm. We prove that in episodic RL, a finite reward automaton can express any non-Markovian bounded reward functions with finitely many reward values and approximate any non-Markovian bounded reward function (with infinitely many reward values) with arbitrary precision. We also provide a lower bound for the episode length such that the proposed RL approach almost surely converges to an optimal policy in the limit. We test this approach on two RL environments with non-Markovian reward functions, choosing a variety of tasks with increasing complexity for each environment. We compare our algorithm with the state-of-the-art RL algorithms for non-Markovian reward functions, such as Joint Inference of Reward machines and Policies for RL (JIRP), Learning Reward Machine (LRM), and Proximal Policy Optimization (PPO2). Our results show that our algorithm converges to an optimal policy faster than other baseline methods.

One popular method for optimizing systems, referred to as ANN-PSO, uses an artificial neural network (ANN) to approximate the system and an optimization method like particle swarm optimization (PSO) to select inputs. However, with reinforcement learning developments, it is important to compare ANN-PSO to newer algorithms, like Proximal Policy Optimization (PPO). To investigate ANN-PSO’s and PPO’s performance and applicability, we compare their methodologies, apply them on steady-state economic optimization of a chemical process, and compare their results to a conventional first principles modeling with nonlinear programming (FP-NLP). Our results show that ANN-PSO and PPO achieve profits nearly as high as FP-NLP, but PPO achieves slightly higher profits compared to ANN-PSO. We also find PPO has the fastest computational times, 10 and 10,000 times faster than FP-NLP and ANN-PSO, respectively. However, PPO requires more training data than ANN-PSO to converge to an optimal policy. This case study suggests PPO has better performance as it achieves higher profits and faster online computational times. ANN-PSO shows better applicability with its capability to train on historical operational data and higher training efficiency.

This paper aims to introduce a solution to Sudan's inadequate electricity supply; focusing on current unconnected rural areas and the high cost of connecting these areas to Sudan's national grid. Microgrids were introduced as a viable option to create small scale distributed grids that depend solely on renewable energy to generate sufficient electricity to satisfy their loads. The paper also aims to enhance the usability of Microgrids by introducing a Machine learning technique to their secondary control that uses energy trading to ensure that all loads in islanded Microgrids are secured. The algorithm uses Reinforcement learning as control for the trading procedure. Data was extracted from a Matlab simulation and was then used to enhance the design of the Reinforcement learning environment. A generic environment for microgrids was designed and implemented which be further used in Reinforcement learning smart grids applications. A set of trading rules were implemented so that the Reinforcement Learning agents can use them across three Microgrids. The agent sees the three micro grids as one primary and two acting as trading game players. Two deep Reinforcement learning algorithms were explored as a solution; the first was an on-policy algorithm; Proximal Policy Optimization (PPO), and the second was an off-policy algorithm; Deep Deterministic Policy Gradients (DDPG). The results of applying both algorithms at three villages in Northern Kordufan State, Hamza ELsheikh, Tannah, and Um Bader were then compared. The algorithms achieved grid equilibrium without any grid loss and achieved profit from the trading process, reducing the time of return for the initial cost of the microgrid.

The unmanned aerial vehicle (UAV) has been applied in unmanned air combat because of its flexibility and practicality. The short-range air combat situation is rapidly changing, and the UAV has to make the autonomous maneuver decision as quickly as possible. In this paper, a type of short-range air combat maneuver decision method based on deep reinforcement learning is proposed. Firstly, the combat environment, including UAV motion model and the position and velocity relationships, is described. On this basic, the combat process is established. Secondly, some improved points based on proximal policy optimization (PPO) are proposed to enhance the maneuver decision-making ability. The gate recurrent unit (GRU) can help PPO make decisions with continuous timestep data. The actor network's input is the observation of UAV, however, the input of the critic network, named state, includes the blood values which cannot be observed directly. In addition, the action space with 15 basic actions and well-designed reward function are proposed to combine the air combat environment and PPO. In particular, the reward function is divided into dense reward, event reward and end-game reward to ensure the training feasibility. The training process is composed of three phases to shorten the training time. Finally, the designed maneuver decision method is verified through the ablation study and confrontment tests. The results show that the UAV with the proposed maneuver decision method can obtain an effective action policy to make a more flexible decision in air combat.

This paper presents a wireless distributed pyroelectric sensor system for tracking and identifying multiple humans based on their body heat radiation. This study aims to make pyroelectric sensors a low-cost alternative to infrared video sensors in thermal gait biometric applications. In this system, the sensor field of view (FOV) is specifically modulated with Fresnel lens arrays for functionality of tracking or identification, and the sensor deployment is chosen to facilitate the process of data-object-association. An Expectation-Maximization-Bayesian tracking scheme is proposed and implemented among slave, master, and host modules of a prototype system. Information fusion schemes are developed to improve the system identification performance for both individuals and multiple subjects. The fusion of thermal gait biometric information measured by multiple nodes is tested at four levels: sample, feature, score, and decision. Experimentally, the prototype system is able to simultaneously track two individuals in both follow-up and crossover scenarios with average tracking errors less than 0.5 m. The experimental results also demonstrate system's potential to be a reliable biometric system for the verification/identification of a small group of human subjects. The developed wireless distributed infrared sensor system can run as a standalone prisoner/patient monitoring system under any illumination conditions, as well as a complement for conventional video and audio human tracking and identification systems.