Adaptive computer-generated forces for simulator-based training

An approximate dynamic programming approach was developed to generate intelligence maneuvering decisions for CGF (Computer Generated Forces) during one-on-one air combat simulation. Simplified kinematic equations of aircraft were built up in flight-path coordinate system, the 3 DOF (degree of freedom) control command was improved, and the common basic flight maneuvering set was described quantitatively. Then, a new method of air combat maneuvering decision was designed according to actual battlefield situation, using trajectory sampling, feature extraction, policy learning. With this method, CGF can learn an approximately optimal policy of air-combat maneuvering during real-time simulation. Result of simulation confirms that proposed method can improve the intelligent decision ability of CGF during simulated air combat training.

With the rapid development of Internet of vehicles (IoV), the emerging in-vehicle applications meet people’s demand for intelligent experience, while also increasing the cost of network communication. Driven by the fact that a few popular content accounted for most of the traffic load, caching popular content at the wireless edge has become a trend of content delivery. However, the limitation of cache resources and the ever-changing mobile environment bring a severe challenge to content delivery cost optimization. In this paper, we first investigate the joint optimization of content placement and bandwidth allocation (JOCB) in IoV. In order to minimize the long-term system cost in the mobile scenario, we first establish an innovative vehicle mobility model based on the probability distribution. Then we use the Lagrangian multiplier method to solve the bandwidth allocation sub-problem to reduce the algorithm complexity. Considering the high-dimensional continuity of action and state spaces, the original problem is transformed into a Markov Decision Process (MDP), and we propose a Deep reinforcement learning based Dynamic Content Placement (DDCP) algorithm. DDCP algorithm has low algorithm complexity and can handle high-dimensional continuous state and action space. The simulation results show that compare with other three benchmark algorithms, DDCP algorithm can reduce the system cost effectively.

Safe deployment of a reinforcement learning robot using self stabilization

ConventionalQ‐learning requires pre‐defined quantized state space and action space. It is not practical for real robot applications since discrete and finite numbers of action sets cannot precisely identify the variances in the different positions on the same state element on which the robot is located. In this paper, aQ‐Learning composed continuous action generator, called the fuzzy cerebellar model articulation controller (FCMAC) method, is presented to solve the problem. TheFCMACdisplays continuous action generation by linear combination of the weighting distribution of the state space where the optimal policy of each state is derived fromQ‐learning. This provides better resolution of the weighting distribution for the state space where the robot is located. The algorithm not only solves the single‐agent problem but also solves the multi‐agent problem by extension. An experiment is implemented in a task where two robots are taking action independently and both are connected with a straight bar. Their goal is to cooperate with each other to pass through a gate in the middle of a grid environment.

Automatic decision-making approaches, such as reinforcement learning (RL), have been applied to (partially) solve the resource allocation problem adaptively in the cloud computing system. However, a complete cloud resource allocation framework exhibits high dimensions in state and action spaces, which prohibit the usefulness of traditional RL techniques. In addition, high power consumption has become one of the critical concerns in design and control of cloud computing systems, which degrades system reliability and increases cooling cost. An effective dynamic power management (DPM) policy should minimize power consumption while maintaining performance degradation within an acceptable level. Thus, a joint virtual machine (VM) resource allocation and power management framework is critical to the overall cloud computing system. Moreover, novel solution framework is necessary to address the even higher dimensions in state and action spaces. In this paper, we propose a novel hierarchical framework for solving the overall resource allocation and power management problem in cloud computing systems. The proposed hierarchical framework comprises a global tier for VM resource allocation to the servers and a local tier for distributed power management of local servers. The emerging deep reinforcement learning (DRL) technique, which can deal with complicated control problems with large state space, is adopted to solve the global tier problem. Furthermore, an autoencoder and a novel weight sharing structure are adopted to handle the high-dimensional state space and accelerate the convergence speed. On the other hand, the local tier of distributed server power managements comprises an LSTM based workload predictor and a model-free RL based power manager, operating in a distributed manner.

Real time estimation of impaired aircraft flight envelope using feedforward neural networks

This work explores on-board planning and scheduling for the multi-target, single spacecraft Earth-observing satellite (EOS) scheduling problem. The problem is formulated as a Markov decision process (MDP) where the number of targets included in the state and action space is an adjustable parameter that may account for clusters of targets with varying priorities. As targets are passed or imaged, they are replaced in the state and action space with the next set of upcoming targets. Unlike prior EOS problem formulations, this work explores how the size of the state and action space can be reduced to produce optimal, generalized policies that may be executed on board the spacecraft in a fraction of a second. Performance of the agents is shown to increase with the number of targets in the state and action space. The number of imaged and downlinked targets stays relatively constant, but the reward increases significantly, demonstrating that the agents are prioritizing high priority targets over low priority targets.

This paper investigates the path planning problem of an unmanned aerial vehicle (UAV) for completing a raid mission through ultra-low altitude flight in complex environments. The UAV needs to avoid radar detection areas, low-altitude static obstacles, and low-altitude dynamic obstacles during the flight process. Due to the uncertainty of low-altitude dynamic obstacle movement, this can slow down the convergence of existing algorithm models and also reduce the mission success rate of UAVs. In order to solve this problem, this paper designs a state detection method to encode the environmental state of the UAV’s direction of travel and compress the environmental state space. In considering the continuity of the state space and action space, the SD-TD3 algorithm is proposed in combination with the double-delayed deep deterministic policy gradient algorithm (TD3), which can accelerate the training convergence speed and improve the obstacle avoidance capability of the algorithm model. Further, to address the sparse reward problem of traditional reinforcement learning, a heuristic dynamic reward function is designed to give real-time rewards and guide the UAV to complete the task. The simulation results show that the training results of the SD-TD3 algorithm converge faster than the TD3 algorithm, and the actual results of the converged model are better.

Network slicing has emerged as a key technology to support the coexistence of multi-service in the the 5G/B5G networks. However, due to the resource scarcity and the diversity of slice requests, how to allocate resources efficiently to maximize the long-term reward of the infrastructure network is a challenging issue. In this paper, we model the resource allocation problem as a semi-Markov decision process (SMDP), which is defined by state space, action space, reward and transition probability distribution. The reward function jointly considers the total income, the cost of available resource and the utilization of the total resource for the infrastructure network. Not focusing on one step decisions reward, we apply the Bellman equation to obtain long-term reward by accumulating. Then we exploit value iteration algorithm to determine the resource allocation scheme according to a certain state in such a way that the long-term reward can be maximized. Extensive simulation results show that the proposed SMDP can achieve a superior performance compared with the existing heuristic methods.

Swarm intelligence has brought a new paradise for function optimization, structural optimization, multi-agent systems and other study fields. In our previous work, we proposed a neuro-fuzzy system using reinforcement learning algorithm (actor-critic method with TD error learning algorithm) to acquire optimized swarm behaviors. This paper improves the conventional learning system, which only deals with discrete state space and action space, to solve how a swarm to learn and obtain its adaptive behaviors in the continuous state space. The improved system adopts a new policy function of action which is possible to yield continuous actions corresponding to continuous states. The effectiveness of proposed system is investigated by computer simulations with more kinds of environments for the goal-exploration problem.Keywordsneuro-fuzzy netswarm behaviorreinforcement learningmulti-agent systemactor-critic algorithmgoal-exploration problem

Deep learning has achieved a great success in both visual and acoustic recognition and classification tasks. The accuracy of many state-of-the-art methods have surpassed that of human beings. However, in the field of robotics, it remains to be a big challenge for a real robot to master a high-level skill using deep learning methods, even though human can easily learn the task from demonstration, imitation and practice. Compared to Go and Atari games, this kind of tasks is usually continuous in both state space and action space, which makes value based reinforcement learning methods unavailable. Making a robot learn to return a ball to a desired point in table tennis is such a typical task. It would be a promising step if a robot can learn to play table tennis without the exact knowledge of the models in this sport just as human players do. In this paper, we consider such a kind of motion decision skill learning, a one-step decision making process, and give a Monte-Carlo based reinforcement learning method in the framework of Deep Deterministic Policy Gradient. Then we apply this method in robotic table tennis and test it on two tasks. The first one is to return balls to a desired point first, and the second one is to return balls to randomly selected landing points. The experimental results demonstrate that the trained policy can successfully return balls of random motion state to both a designated point and randomly selected landing points with high accuracy.

This paper proposes a deep Q network (DQN)-based solution framework to minimize UAV's energy consumption in UAV-assisted wireless powered sensor network under the age of information (AoI) constraint, where a UAV wirelessly charges ground sensors and then the sensors use harvested energy to upload their freshly collected information to the UAV. The corresponding non-convex energy-minimization problem is first modeled as a Markov process, and then the state spaces, action spaces and reward function are designed. Simulation results show that the proposed DQN achieves much smaller energy consumption than traditional greedy-based scheme, and when the number of sensors is more than 8, traditional greedy-based scheme becomes very difficult to solve the problem, while our presented DQN method can still find an optimal solution. Moreover, the UAV's energy consumption increases with the decrease of AoI or the increment of sensors' amount, and with the rotation angle constraint, UAV's trajectory becomes smooth.

The traditional cabin capacity control method has limited real-time response ability and is difficult to achieve good results in complex and changing environment. A capacity control algorithm of multi-level fare class based on enhanced exploration PPO algorithm is proposed. The problem of capacity control is transformed into a sequential decision problem. Using deep neural network fits the value function, the state space, action space and reward function for solving sequential decision problem are given. The simulation results show that compared with benchmark algorithms such as EMSR-b, FCFS and PPO, the proposed algorithm could effectively obtain more benefits.

The appropriate design of a vehicular ad hoc network (VANET) has become a pivotal way to build an efficient smart transportation system, which enables various applications associated with traffic safety and highly-efficient transportation. VANETs are vulnerable to the threat of malicious nodes stemming from its dynamicity and infrastructure-less nature and causing performance degradation. Recently, software-defined networking (SDN) has provided a feasible way to manage VANETs dynamically. In this article, we propose a novel software-defined trust based VANET architecture (SD-TDQL) in which the centralized SDN controller is served as a learning agent to get the optimal communication link policy using a deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning approach. The trust of each vehicle and the reverse delivery ratio are considered in a joint optimization problem, which is modeled as a Markov decision process with state space, action space, and reward function. Specifically, we use the expected transmission count ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$ETX$ </tex-math></inline-formula> ) as a metric to evaluate the quality of the communication link for the connected vehicles’ communication. Moreover, we design a trust model to avoid the bad influence of malicious vehicles. Simulation results prove that the proposed SD-TDQL framework enhances the link quality.

We consider positive zero-sum stochastic games with countable state and action spaces. For each player, we provide a characterization of those strategies that are optimal in every subgame. These characterizations are used to prove two simplification results. We show that if player 2 has an optimal strategy then he/she also has a stationary optimal strategy, and prove the same for player 1 under the assumption that the state space and player 2's action space are finite.