Research on autonomous maneuvering decision of UCAV based on approximate dynamic programming

An approximate dynamic programming approach was developed to generate intelligence maneuvering decisions for CGF (Computer Generated Forces) during one-on-one air combat simulation. Simplified kinematic equations of aircraft were built up in flight-path coordinate system, the 3 DOF (degree of freedom) control command was improved, and the common basic flight maneuvering set was described quantitatively. Then, a new method of air combat maneuvering decision was designed according to actual battlefield situation, using trajectory sampling, feature extraction, policy learning. With this method, CGF can learn an approximately optimal policy of air-combat maneuvering during real-time simulation. Result of simulation confirms that proposed method can improve the intelligent decision ability of CGF during simulated air combat training.

An approximate dynamic programming approach for solving an air combat maneuvering problem

Many public funding agencies adopt the one-stage call-based mode to select R&D projects. The funding decision-making in this mode is characterized by two primary features: 1) it is static, meaning that decision-makers solely concentrate on projects submitted within the current call, and 2) it is decentralized, indicating that funding decisions are made independently across diverse sectors/disciplines. As a result, the administrative burden on funding agencies is reduced. However, due to the budget constraint, the funding agencies may fail to fund better projects that emerge in future calls or appear in other sectors/disciplines. Therefore, we investigate whether a better project portfolio can be achieved by selecting public R&D projects in a dynamic and centralized manner. We formulate a Markov decision process (MDP) model for the dynamic centralized public R&D project portfolio selection problem. We develop an approximate dynamic programming (ADP) approach that combines learning-based Monte Carlo simulation and a two-stage rollout algorithm to efficiently solve the MDP model. Based on extensive computational experiments, we compare our ADP approach with a threshold heuristic that is representative of the static and decentralized funding mode, as well as two baseline algorithms. The results indicate that our ADP approach is effective, and it is beneficial to adopt the dynamic centralized decision-making mode in public R&D project portfolio selection. We also investigate our ADP approach utilizing a case study based on real-world data from the National Natural Science Foundation of China. Our ADP approach can be integrated into the decision support systems of funding agencies, enabling automated, dynamic, and centralized selection of public R&D projects.

This paper proposes a method to study unmanned aerial vehicles (UAV) maneuvering decision in air combat based on deterministic policy gradient. Aiming at the problem of decision space continuity, based on reinforcement learning theory, the Deep Deterministic Policy Gradient (DDPG) algorithm architecture is used to overcome the dimensional catastrophe caused by the discretization of decision variables and achieve air combat decisions in a continuous decision space for the problem of continuous decision space. In the design of the reward function, based on the traditional distance and angle evaluation factors, the energy function is added to improve the accuracy of the reward function for air combat situation description. Through autonomous reinforcement learning training, UAV gradually learns to acquire strategies for air combat decision making without the priori knowledge, which enables UAV to gain maneuver dominance advantage in air combat. Simulation experiments show that based on the algorithm model proposed in this paper, UAV can conduct the autonomous learning process, complete the air combat maneuver decision and obtain the dominance advantage according to the air combat maneuver decision-making environment.

This article examines a major maritime disaster scenario in which the objective is to maximize the number of survivors. To achieve this objective, we seek to optimize the decision policy to load individuals onto a helicopter for transport from an evacuation site to a forward operating base. Our contributions are twofold. First, we formulate the loading problem as a finite-horizon Markov Decision Process. Second, since the curse of dimensionality renders exact methods not applicable, we use an Approximate Dynamic Programming (ADP) approach to explore the efficacy of these methods to the problem. We generate three ADP policies, each using a value function approximation based on a unique post-decision state variable and a lookup table representation. We compare these ADP policies to a random policy, a myopic policy, and Policy Function Approximation (PFA) which evacuates survivors in order of triage category starting with the most critically ill. Our results show that an ADP-generated policy which prioritizes the evacuation of healthy individuals improved performance by 34 ± 5% versus the random policy, 15 ± 3% versus the myopic policy, and 42 ± 3% versus the PFA policy.

In a typical game scenario, the attention in unmanned aerial vehicle (UAV) air combat should be focused on both sides' maneuvering decision strategies. However, most existing studies focus only on improving their own maneuvering decisions, ignoring the importance of the opponent's strategy in the two-player game. This paper proposes a reinforcement learning (RL)-based air combat decision method considering the opponent's maneuvering strategy. Through the proposed limited imitation offline RL (LIORL) method, the opponent's air combat decision method is refined using existing air combat data. Based on the enemy's superior strategy, an air combat simulation environment is created, and the RL method is applied to train the agent for UAV air combat maneuvering decision strategies. No current research utilizes static datasets for the acquisition of adversary strategies. This study, in contrast, offers a more reliable approach in comparison to methodologies predicated on assuming adversary strategies. Ablation experiments have been meticulously executed to showcase the capability of the LIORL algorithm in identifying the optimal enemy strategy while exerting a lesser impact on the dataset than incumbent algorithms. The employment of the LIORL algorithm enables the agent to refine the adversary's strategy, resulting in an augmented win rate against our UAV. Through air combat simulations, we empirically validate the agent's decision-making process based on the adversary's strategy, thereby affirming the efficacy of the proposed methodology.

Optimal Exercise Patterns and Valuation of Employee Stock Options: An Approximate Dynamic Programming Approach

This report formulates a dual-source model that considers artificial blood along with natural blood as sources of blood platelet supply. The model is formulated for a single hospital by considering deterministic lead time while ordering, uncertainty in supply, and demand. A newsvendor model is used to find the optimal order quantity for each day and approximate dynamic programming (ADP) is used to deal with uncertainty over a year. Three different step size rules are being studied and compared in terms of average value function approximation (VFA) per timestep, with two being deterministic step size and the other being an optimal stochastic step size to deal with non-stationary data. The project aims is to select a policy that can reduce shortages, outages, and maximize the rewards for the developed model. Eleven policies are being investigated while introducing artificial blood platelet rewards. Numerical analysis is implemented by considering varied demand and donations for different acceptance percentages and inventory levels of artificial blood under each policy. Another analysis is done to show how shortage and outage values vary for a single source supply vs a dual source (0% acceptance vs 50% acceptance) and are computed for three policies. Computed results show that first in first out (FIFO) is the best policy with the least shortage and outage for various acceptance rates, for varied demand and donations. As the acceptance rate and the inventory level for artificial blood are increased, for 10% a shortage percentage of 5.8% along with 5.9% of outage for natural and 8.8% for artificial is observed. When increased to 30%, 40%, we see an increase in the outage of natural platelets from5.9% to 6.1% along with shortage levels slightly falling. For 50% shortage values falls below 5.2%, outage values fall below 5.7% with outage of artificial from 9% to 8.6%. For the second analysis 0% vs 50%, considering a secondary source can help reduces shortage from 5.8% to 5.1% and outage from 6.3% to 5.7%.

The recent development of technology helps in the revolutionary war and it controls the war which is influenced by brilliant planning. The maneuver aircraft of intelligent algorithm aids the pilot to decide the particular position on the battlefield. Nowadays the hardware components of radar and missiles are widely used and the beyond visual range is the most popular method applied in air combat. The introduction of close-range air combat maneuver decisions generates the attention of researchers in artificial intelligence. Most of the existing methods are based on autonomous aircraft focused in air combat scenario but manual air combats are widely applied in dual aircraft. Based on the factors mentioned above, a novel hierarchical maneuver decision architecture is applied to a dual-aircraft close-range air combat scenario. Subsequently, the soft actor-critic algorithm is merged with competitive self-play which integrates the knowledge of sub-strategies. Further, the reinforcement learning technique is employed to achieve an approximate Nash equilibrium master strategy. The experimental results show that the hierarchical architecture exhibits good performance, symmetry, and robustness. The research generates a solution for intelligent formation of air combat in the future and guidance for manned or unmanned aircraft cooperative combat.

This paper proposes an Approximate Dynamic Programming (ADP) approach for obtaining the optimal real-time operation strategy of microgrid with power-to-hydrogen (P2H) device. Firstly, the real-time operation model of microgrid with P2H is established, then a piece-wise linear function based approximate dynamic programming is proposed to solve the operation model. With the direction of the slope update and training policies under various training scenarios, the proposed ADP approach can obtain the approximate optimal real-time operation strategies to cope with uncertainties. Case studies verify the superiority of the proposed ADP approach compared with Model Predictive Control approach (MPC).

The unmanned aerial vehicle (UAV) has been applied in unmanned air combat because of its flexibility and practicality. The short-range air combat situation is rapidly changing, and the UAV has to make the autonomous maneuver decision as quickly as possible. In this paper, a type of short-range air combat maneuver decision method based on deep reinforcement learning is proposed. Firstly, the combat environment, including UAV motion model and the position and velocity relationships, is described. On this basic, the combat process is established. Secondly, some improved points based on proximal policy optimization (PPO) are proposed to enhance the maneuver decision-making ability. The gate recurrent unit (GRU) can help PPO make decisions with continuous timestep data. The actor network's input is the observation of UAV, however, the input of the critic network, named state, includes the blood values which cannot be observed directly. In addition, the action space with 15 basic actions and well-designed reward function are proposed to combine the air combat environment and PPO. In particular, the reward function is divided into dense reward, event reward and end-game reward to ensure the training feasibility. The training process is composed of three phases to shorten the training time. Finally, the designed maneuver decision method is verified through the ablation study and confrontment tests. The results show that the UAV with the proposed maneuver decision method can obtain an effective action policy to make a more flexible decision in air combat.

Approximate dynamic programming for an energy-efficient parallel machine scheduling problem

In order to facilitate the introduction of unmanned aircraft (UA) systems (UAS) into nonsegregated airspace, highly reliable radio control and nonpayload communications (CNPC) links will be required. The International Telecommunication Union Radiocommunication Sector (ITU-R) has estimated in report M.2171 that in the year 2030 a total of 34 MHz of spectrum will be required for terrestrial communications and another 56 MHz will be necessary for satellite communications. This paper addresses only the terrestrial (that is, direct ground/air without satellites) links. Its purpose is to validate, via a concrete example, that all of the terrestrial link requirements can be met within a potential 34 MHz allocation in portions of the L-band (960-1164 MHz) and C-band (5030-5091 MHz). There are many potentially conflicting issues that have to be addressed to satisfy the requirements levied in. All of these requirements are somewhat difficult to satisfy simultaneously; however, a system design that meets the known requirements has been devised. The proposed design should be considered as an "existence proof" that the terrestrial system requirements can indeed be met; it is not necessarily the final link design.

In order to the requirement of flexibility and real-time for maneuvering decision in short range air combat, a maneuvering decision model in short range air combat based on reinforcement learning is presented. The model has two characteristics, one is select the situational evaluation function to build the state space, what make the discrete state space get more representative. The other is use the thinking of Monte-Carlo reinforcement learning, the result of air combat was used as the basis for returning the reward, what ensure the continuity of maneuver movement. The model is use the control valve as the decision result and the decision time is less than 0.001 seconds. At last, the feasibility of the decision model was verified by two different experiments.

An approximate dynamic programming approach for the empty container allocation problem