Policy Optimization in Automated Point Merge Trajectory Planning: An Artificial Intelligence-based Approach

An Agent-Based Approach to Aircraft Conflict Resolution with Constraints

This paper describes a simulation tool developed to aid the analysis of air traffic management domains and potential air traffic control strategies. The simulation represents aircraft and automated air traffic control support tool functionality based on a computational model of aircraft flight management system trajectories. It simulates the effects of environmental disturbances and air traffic controller intervention on air traffic in fast time. The paper outlines a methodology that leverages these capabilities to help with tool design and evaluation, and discusses the results of preliminary simulations aimed toward using the proposed methodology for characterizing the effectiveness of particular strategy-tool combinations for supporting more efficient terminal-area air traffic management. This research is supported by NASA's Vehicle Systems Program Quiet Aircraft Technology project and Airspace Systems Program Advanced Air Transportation Technology project.

This paper/chapter deals with the key drivers for adopting and developing an Occupational Health and Safety System (OHS) with a special focus on air traffic management and traffic controller’s workplace. A such system includes regulation and legal compliance procedures, actions and monitoring for ensuring workplace safety, incentives and motivation for the air traffic controller and associate personnel health and wellbeing. By a systemic approach, the key characteristics of OHS towards air traffic management are presented, highlighting the key aspects for implementing a quality management system in air traffic control, which is the cornerstone of airport operation efficiency and productivity on one hand; and the nature of job and the intensive working environment is well recognised. Based on air traffic providers functional analysis the key occupational aspects for air traffic control are taken into consideration, providing the benefits for implementing quality management systems (QMS) and OHS is real business. Conventional wisdom is to highlight the importance for establishing and incorporating a modern custom-made OHS system in accordance with the requirements addressed by OHSAS 18001 to develop and implement a QMS for air traffic services. Contribution of this paper is to highlight the key priorities for managers and decision makers in field of air traffic services providers, depicting ways and recommendation for adopting an efficient path for implementing OHS in a QMS environment.

Reinforcement learning (RL), a promising framework for data-driven decision making in an uncertain environment, has successfully been applied in many real-world operation and control problems. However, the application of RL in a large-scale decentralized multi-agent environment remains a challenging problem due to the partial observability and limited communications between agents. In this paper, we develop a model-based kernel RL approach and a model-free deep RL approach for learning a decentralized, shared policy among homogeneous agents. By leveraging the strengths of both these methods, we further propose a novel deep ensemble multi-agent reinforcement learning (MARL) method that efficiently learns to arbitrate between the decisions of the local kernel-based RL model and the wider-reaching deep RL model. We validate the proposed deep ensemble method on a highly challenging real-world air traffic control problem, where the goal is to provide effective guidance to aircraft to avoid air traffic congestion, conflicting situations, and to improve arrival timeliness, by dynamically recommending adjustments of aircraft speeds in real-time. Extensive empirical results from an open-source air traffic management simulation model, developed by Eurocontrol and built on a real-world data set including thousands of aircrafts, demonstrate that our proposed deep ensemble MARL method significantly outperforms three state-of-the-art benchmark approaches.

As air traffic volumes continue to grow, understanding and managing traffic flows within Terminal Maneuvering Areas (TMAs) has become essential to meet increasing capacity demands.TMAs, the airspaces surrounding airports managed by Air Traffic Controller (ATC) for landings and takeoffs, often face challenges due to limited capacity, leading to performance degradation and significant environmental footprint.While existing Key Performance Indicators (KPIs) effectively assess the en route phase of flight journeys, there remains a gap in the metrics specifically designed for evaluating TMAs.The purpose of this thesis is to develop a comprehensive evaluation framework tailored for TMAs to assess performance, identify inefficiencies, and highlight areas for improvement.First, the existing general metrics were modified for TMA-specific evaluation, proved effective in detecting inconsistencies in air traffic control and identifying optimization opportunities.New TMA-specific performance indicators were also developed and applied to various TMAs with different types of arrival procedures.In the second phase, dependencies between the metrics were examined to create a performance assessment tool tailored to specific operational scopes.Statistical Ordinary Least Squares (OLS) analysis was performed on two subsets of metrics, uncovering that the newly-introduced metric, 'threshold,' correlates significantly with other performance indicators.This metric serves for the predicting TMAs performance based on entry conditions, improving air traffic management.Recently, the focus was placed on Point Merge-specific metrics to assess their potential in improving environmental efficiency of the arrivals.Testing across seven airports revealed that Point Merge (PM) structures are often underutilized.Finally, the PM-specific metrics and the overall performance evaluation framework were refined for more efficient evaluation and further optimization of TMAs featuring different types of arrival operations.

Air Traffic Management (ATM) is defined as a broad domain that consists of complex activities such as Air Traffic Flow Management (ATFM). Within this context this paper presents two prototypes of a module (MAAD) assembled in a decision support device that is based on Artificial Intelligence, which generates suggestions for air traffic flow control according to each traffic saturation scenario. The device known as Distributed Decision Support System for Air Traffic Flow Management (SISCONFLUX) was built with a computational agent based on Reinforcement Learning (Module MAAD), which uses Q-learning algorithm. Two prototypes were built and one of them incorporated the specialist's (human agent) experience. The performance of the prototypes was evaluated through a case study modeled from real traffic demands verified on certain dates. The results obtained in the case study are promising; the behavior of the prototypes demonstrated that the Q-learning algorithm converged in a satisfying way. The prototypes generated actions that contributed effectively to the reduction of saturation in the air traffic scenarios under test.

Adapting a Human – Automation Trust Scale to an Air Traffic Management Environment

Decision support system in tactical air traffic flow management for air traffic flow controllers

The objective of this paper is to examine the use and applications of reinforcement learning (RL) techniques in the production planning and control (PPC) field addressing the following PPC areas: facility resource planning, capacity planning, purchase and supply management, production scheduling and inventory management. The main RL characteristics, such as method, context, states, actions, reward and highlights, were analysed. The considered number of agents, applications and RL software tools, specifically, programming language, platforms, application programming interfaces and RL frameworks, among others, were identified, and 181 articles were sreviewed. The results showed that RL was applied mainly to production scheduling problems, followed by purchase and supply management. The most revised RL algorithms were model-free and single-agent and were applied to simplified PPC environments. Nevertheless, their results seem to be promising compared to traditional mathematical programming and heuristics/metaheuristics solution methods, and even more so when they incorporate uncertainty or non-linear properties. Finally, RL value-based approaches are the most widely used, specifically Q-learning and its variants and for deep RL, deep Q-networks. In recent years however, the most widely used approach has been the actor-critic method, such as the advantage actor critic, proximal policy optimisation, deep deterministic policy gradient and trust region policy optimisation.

The rapid advancement of autonomous robots in domains such as healthcare, transportation, and disaster response necessitates robust ethical decision-making frameworks to mitigate risks associated with uncertain environments. This research delves deeply into the application of reinforcement learning (RL) for enabling "safe" decisions in robots facing multiple risky outcomes. By synthesizing over 80 peer- reviewed sources, including recent 2024-2025 surveys on ethical RL and safe RL benchmarks, it systematically addresses three pivotal research questions: (1) How should a robot choose between two risky paths? (2) Can RL augmented with reward shaping effectively reduce harm in unpredictable environments? (3) How do prominent RL algorithms compare in terms of safety efficacy in robotic applications? Key findings indicate that multi-objective RL (MORL) and constrained policy optimization variants, such as Proximal Policy Optimization (PPO) with Lagrangian constraints, outperform traditional RL in balancing ethical imperatives with task performance, achieving up to 40% reductions in violation rates in simulated benchmarks. Reward shaping, particularly potential-based and ethics-informed variants, accelerates ethical convergence by 3-6x while curbing reward hacking—a common pitfall where agents exploit reward loopholes. A novel student-led simulation in a custom gridworld environment demonstrates that shaped rewards enhance average returns by approximately 12% with near-zero harm incidents over 1,000 episodes. This expanded study proposes an integrated Ethical Safe RL (ESRL) framework, incorporating human-in-the-loop feedback and verifiable constraints, tailored for accessible implementation in educational settings. It underscores persistent challenges like distributional shifts in real-world deployment and calls for interdisciplinary ethics integration. By bridging theoretical surveys with empirical simulations, this work empowers high-school researchers to contribute to responsible AI robotics.

Reinforcement learning (RL) is one of the popular methods for intelligent control and decision making in the field of robotics recently. The goal of RL is to learn an optimal policy of the agent by interacting with the environment via trail and error. There are two main algorithms for RL problems, including model-free and model-based methods. Model-free RL is driven by historical trajectories and empirical data of the agent to optimize the policy, which needs to take actions in the environment to collect the trajectory data and may cause the damage of the robot during training in the real environment. The main different between model-based and model-free RL is that a model of the transition probability in the interaction environment is employed. Thus the agent can search the optimal policy through internal simulation. However, the model of the transition probability is usually estimated from historical data in a single environment with statistical errors. Therefore, an issue is faced by the agent is that the optimal policy is sensitive to perturbations in the model of the environment which can lead to serious degradation in performance. Robust RL aims to learn a robust optimal policy that accounts for model uncertainty of the transition probability to systematically mitigate the sensitivity of the optimal policy in perturbed environments. In this overview, we begin with an introduction to the algorithms in RL, then focus on the model uncertainty of the transition probability in robust RL. In parallel, we highlight the current research and challenges of robust RL for robot control. To conclude, we describe some research areas in robust RL and look ahead to the future work about robot control in complex environments.

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

Accurately predicting the time aircraft spend within Terminal Maneuvering Areas (TMA) is crucial for optimizing air traffic management, minimizing delays, and improving operational efficiency. This study introduces two Graph Neural Network (GNN) models—Graph Convolutional Network (GCN) and Graph Attention Network (GAT)—to forecast aircraft TMA duration based on historical Automatic Dependent Surveillance-Broadcast (ADS-B) data collected from Istanbul TMA. The models leverage spatial-temporal patterns and inter-aircraft relationships to generate predictions. The GAT model employs an attention mechanism to assign varying importance to different nodes, dynamically capturing complex relationships within the TMA. In contrast, the GCN model applies uniform weighting to node connections, offering computational efficiency but limited adaptability to dynamic air traffic conditions. Results demonstrate that the GNN-based approach is effective for TMA duration prediction due to its ability to adapt to traffic density variations. This work underscores the potential of GNN-based approaches in air traffic management applications, facilitating better-informed decision-making processes for traffic controllers and operational planners.

The combination of deep neural networks and reinforcement learning had received more and more attention in recent years, and the attention of reinforcement learning of single agent was slowly getting transferred to multiagent. Regret minimization was a new concept in the theory of gaming. In some game issues that Nash equilibrium was not the optimal solution, the regret minimization had better performance. Herein, we introduce the regret minimization into multiagent reinforcement learning and propose a multiagent regret minimum algorithm. This chapter first introduces the Nash Q-learning algorithm and uses the overall framework of Nash Q-learning to minimize regrets into the multiagent reinforcement learning and then verify the effectiveness of the algorithm in the experiment.

The increasing demand in air transportation is pushing the current air traffic management (ATM) system to its limits in the airspace capacity and workload of air traffic controllers (ATCOs). ATCOs are in an urgent need of assistant tools to aid them in dealing with increased traffic. To address this issue, the application of artificial intelligence (AI) in supporting ATCOs is a promising approach. In this work, we build an AI system as a digital assistant to support ATCOs in resolving potential conflicts. Our system consists of two core components: an intelligent interactive conflict solver (iCS) to acquire ATCOs' preferences, and an AI agent based on reinforcement learning to suggest conflict resolutions capturing those preferences. We observe that providing the AI agent with the human resolutions, which are acquired and characterized by our intelligent interactive conflicts solver, not only improves the agent's performance but also gives it the capability to suggest more human-like resolutions, which could help increase the ATCOs' acceptance rate of the agent's suggested resolutions. Our system could be further developed as personalized digital assistants of ACTOs to maintain their workloads manageable when they have to deal with sectors with increased traffic.