Approximate dynamic programming for CGF air combat maneuvering decision

Adaptive computer-generated forces for simulator-based training

In a typical game scenario, the attention in unmanned aerial vehicle (UAV) air combat should be focused on both sides' maneuvering decision strategies. However, most existing studies focus only on improving their own maneuvering decisions, ignoring the importance of the opponent's strategy in the two-player game. This paper proposes a reinforcement learning (RL)-based air combat decision method considering the opponent's maneuvering strategy. Through the proposed limited imitation offline RL (LIORL) method, the opponent's air combat decision method is refined using existing air combat data. Based on the enemy's superior strategy, an air combat simulation environment is created, and the RL method is applied to train the agent for UAV air combat maneuvering decision strategies. No current research utilizes static datasets for the acquisition of adversary strategies. This study, in contrast, offers a more reliable approach in comparison to methodologies predicated on assuming adversary strategies. Ablation experiments have been meticulously executed to showcase the capability of the LIORL algorithm in identifying the optimal enemy strategy while exerting a lesser impact on the dataset than incumbent algorithms. The employment of the LIORL algorithm enables the agent to refine the adversary's strategy, resulting in an augmented win rate against our UAV. Through air combat simulations, we empirically validate the agent's decision-making process based on the adversary's strategy, thereby affirming the efficacy of the proposed methodology.

Unmanned aircraft systems can perform some more dangerous and difficult missions which manned aircraft systems cannot perform. For tasks with high complexity, such as air combat, maneuvering decision mechanism is required to sense the combat environment and make the optimal strategy in real time. This paper formulates one-to-one air combat maneuvering problem in 3D environment, and proposes an approximate dynamic programming approach to make optimal maneuvering decisions automatically. The aircraft searches for combat strategies based-on Reinforcement Leaning, while sensing the environment, taking available maneuvering actions and receiving feedback reward signals. To solve the problem of dimensional explosion in the air combat, the proposed method is implemented through feature selection, trajectory sampling, function approximation and Bellman backup operation in the air combat simulation environment. This approximate dynamic programming approach provides a fast response to rapid changing tactical situations, and learns effective strategies to fight against the opponent aircraft.

This paper proposes a method to study unmanned aerial vehicles (UAV) maneuvering decision in air combat based on deterministic policy gradient. Aiming at the problem of decision space continuity, based on reinforcement learning theory, the Deep Deterministic Policy Gradient (DDPG) algorithm architecture is used to overcome the dimensional catastrophe caused by the discretization of decision variables and achieve air combat decisions in a continuous decision space for the problem of continuous decision space. In the design of the reward function, based on the traditional distance and angle evaluation factors, the energy function is added to improve the accuracy of the reward function for air combat situation description. Through autonomous reinforcement learning training, UAV gradually learns to acquire strategies for air combat decision making without the priori knowledge, which enables UAV to gain maneuver dominance advantage in air combat. Simulation experiments show that based on the algorithm model proposed in this paper, UAV can conduct the autonomous learning process, complete the air combat maneuver decision and obtain the dominance advantage according to the air combat maneuver decision-making environment.

An approximate dynamic programming approach for solving an air combat maneuvering problem

Computer generated forces (CGFs) must display realistic behavior for tactical training simulations to yield an effective training experience. Tradionally, the behavior of CGFs is scripted. However, there are three drawbacks, viz. (1) scripting limits the adaptive behavior of CGFs, (2) creating scripts is difficult and<br/>(3) it requires scarce domain expertise. A promising machine learning technique<br/>is the dynamic scripting of CGF behavior. In simulating air combat scenarios, team behavior is important, both with and without communication. While dynamic scripting has been reported to be effective in creating behavior for<br/>single fighters, it has not often been used for team coordination. The dynamic<br/>scripting technique is sufficiently flexible to be used for different team coordination methods. In this paper, we report the first results on centralized<br/>coordination of dynamically scripted air combat teams, and compare these results to a decentralized approach from earlier work. We find that using<br/>the centralized approach leads to higher performance and more efficient learning, although creativity of the solutions seems bounded by the reduced complexity.

We simulate and analyze pilot decision making in one-on-one air combat by using an ine uence diagram. Unlike most of the existing approaches, an ine uence diagram graphically describes the factors of a decision process and explicitly handles the decision maker’ s preferences under conditions of uncertainty. In the pilot decision model, the possible combat situations related to each maneuver alternative are associated with a probability and a utility. Ine uence diagram analysis produces a probability distribution of the overall utility that represents the successfulness of a maneuver and gives information to make rational maneuvering decisions. Sensitivity analysis determines the impacts of different factors on the outcome of the maneuvering decision. The effects of sensor information that will reduce the uncertainty of the model are evaluated with Bayesian reasoning. The model can be utilized in the analysis of a single decision situation or as an automated decision-making system that selects combat maneuvers in air combat simulators.

In this paper, air combat simulation data is reconstructed into a dynamic Bayesian network. It gives a compact probabilistic model that describes the progress of air combat and allows efficient computing for study of different courses of the combat. This capability is used in what-if type analysis that investigates the effect of different air combat situations on the air combat evolution and outcome. The utilization of the dynamic Bayesian network is illustrated by analyzing simulation results produced with a discrete event air combat simulation model called X-Brawler.

The recent development of technology helps in the revolutionary war and it controls the war which is influenced by brilliant planning. The maneuver aircraft of intelligent algorithm aids the pilot to decide the particular position on the battlefield. Nowadays the hardware components of radar and missiles are widely used and the beyond visual range is the most popular method applied in air combat. The introduction of close-range air combat maneuver decisions generates the attention of researchers in artificial intelligence. Most of the existing methods are based on autonomous aircraft focused in air combat scenario but manual air combats are widely applied in dual aircraft. Based on the factors mentioned above, a novel hierarchical maneuver decision architecture is applied to a dual-aircraft close-range air combat scenario. Subsequently, the soft actor-critic algorithm is merged with competitive self-play which integrates the knowledge of sub-strategies. Further, the reinforcement learning technique is employed to achieve an approximate Nash equilibrium master strategy. The experimental results show that the hierarchical architecture exhibits good performance, symmetry, and robustness. The research generates a solution for intelligent formation of air combat in the future and guidance for manned or unmanned aircraft cooperative combat.

Reinforcement learning (RL) has recently proven itself as a powerful instrument for solving complex problems and even surpassed human performance in several challenging applications. This signifies that RL algorithms can be used in the autonomous air combat problem, which has been studied for many years. The complexity of air combat arises from aggressive close-range maneuvers and agile enemy behaviors. In addition to these complexities, there may be uncertainties in real-life scenarios due to sensor errors, which prevent estimation of the actual position of the enemy. In this case, autonomous aircraft should be successful even in the noisy environments. In this study, we developed an air combat simulation, which provides noisy observations to the agents, therefore, make the air combat problem even more challenging. Thus, we present a state stacking method for noisy RL environments as a noise reduction technique. In our extensive set of experiments, the proposed method significantly outperforms the baseline algorithms in terms of the winning ratio, where the performance improvement is even more pronounced in the high noise levels. In addition, we incorporate a self-play scheme to our training process by periodically updating the enemy with a frozen copy of the training agent. By this way, the training agent performs air combat simulations to an enemy with smarter strategies, which improves the performance and robustness of the agents. In our simulations, we demonstrate that the self-play scheme provides important performance gains compared to the classical RL training.

Goal of the present paper is providing support to operations planning and management in complex scenarios. The authors are mainly focused on South Asia region, which is subject of experimental analysis by running an Intelligent Agents—driven HLA Federation. Simulation of investments and operations over an asymmetric mission environment with several parties, insurgents, terrorists and dynamic social framework is the aim. The scenario has various degrees of freedom and M&S enables evaluation of human behavior evolution and socio-psychological aspects. The presented models include Computer Generated Forces (CGF) driven by Intelligent Agents (IAs) that represents not only units on the battlefield, but also people and interest groups (i.e. Middle Class, Nomads, Clans). The study is focused on Civil Military Co-operations (CIMIC) and Psychological Operations (PSYOPs). The simulation is based on specific architecture that involves various federates playing different roles. Verification, Validation and Accreditation (VV&A) has been applied along the whole life cycle of the research, in order to determine the correctness and effectiveness of the results. The paper proposes experimental results obtained during the dynamic test of the federations.

The unmanned aerial vehicle (UAV) has been applied in unmanned air combat because of its flexibility and practicality. The short-range air combat situation is rapidly changing, and the UAV has to make the autonomous maneuver decision as quickly as possible. In this paper, a type of short-range air combat maneuver decision method based on deep reinforcement learning is proposed. Firstly, the combat environment, including UAV motion model and the position and velocity relationships, is described. On this basic, the combat process is established. Secondly, some improved points based on proximal policy optimization (PPO) are proposed to enhance the maneuver decision-making ability. The gate recurrent unit (GRU) can help PPO make decisions with continuous timestep data. The actor network's input is the observation of UAV, however, the input of the critic network, named state, includes the blood values which cannot be observed directly. In addition, the action space with 15 basic actions and well-designed reward function are proposed to combine the air combat environment and PPO. In particular, the reward function is divided into dense reward, event reward and end-game reward to ensure the training feasibility. The training process is composed of three phases to shorten the training time. Finally, the designed maneuver decision method is verified through the ablation study and confrontment tests. The results show that the UAV with the proposed maneuver decision method can obtain an effective action policy to make a more flexible decision in air combat.

Air combat decision-making is a process in which the both sides choose certain strategies to maximize the probability of winning according to the current air combat situation. Imitation learning is a way for an agent to make intelligent decisions by imitating expert examples. In this paper, aircraft confrontational strategies are trained by behavior repetition, which is a kind of imitative learning. Aiming at the aircraft model with three degrees of freedom, seven basic maneuvers were designed as alternative strategies by taking 1V1 and 2V2 air combat confrontations as research objects. The victory samples were obtained by Monte Carlo simulation, and state-action pairs were extracted from the victory samples. On this basis, neural network training was carried out, and the model that could be used for air combat decision-making was obtained under the condition of 1V1. In 2V2 condition, we focus on finding the winning strategy through the cooperation between agents. The validity of the decision model is proved by simulation.KeywordsImitation learningAir combat simulationCooperative strategy

This paper presents a new game-theoretic approach toward the validation of discrete-event air combat (AC) simulation models and simulation-based optimization. In this approach, statistical techniques are applied for estimating games based on data produced by a simulation model. The estimation procedure is presented in cases involving games with both discrete and continuous decision variables. The validity of the simulation model is assessed by comparing the properties of the estimated games to actual practices in AC. These games are also applied for simulation-based optimization in a two-sided setting in which the action of the opponent is taken into account. In optimization, the estimated games enable the study of effectiveness of AC tactics as well as aircraft, weapons, and avionics configurations. The game-theoretic approach enhances existing methods for the validation of discrete-event simulation models and techniques for simulation-based optimization by incorporating the inherent game setting of AC into the analysis. It also provides a novel game-theoretic perspective to simulation metamodeling which is used to facilitate simulation analysis. The utilization of the game-theoretic approach is illustrated by analyzing simulation data obtained with an existing AC simulation model.

An air combat maneuver pattern extraction based on time series segmentation and clustering analysis