Optimizing time-varying performance and mission aborting policy in resource constrained missions

This paper models a real-time warm standby system that has to accomplish a specified amount of task by a hard deadline. The system is subject to corrective replacements (CRs) upon failure of its operating element. It can also be renewed according to a predetermined schedule through preventive replacements (PRs). To facilitate an effective recovery of system operation after replacements, periodic backups are performed so that warm standby elements, upon being activated, can take over the mission task from the last backup point instead of from scratch. This paper presents a novel integrated model that considers effects of periodic backups, CRs and PRs in analyzing and optimizing real-time warm standby systems. Mission success probability and expected mission completion time are evaluated. Impacts of different mission and element parameters on mission success probability, optimal backup and PR policies, and optimal element activation sequence are investigated. It is shown that in warm standby systems with periodic backups and tight deadlines, PRs can improve the mission success probability even when they take the same time as CRs. When the maximum allowed mission time exceeds a certain level, PRs become ineffective and the optimal policy can involve only periodic backups.

Phased-mission systems are widely utilized in aerospace, defense fields, and production processes. These systems are characterized by missions consisting of multiple consecutive phases, each with distinct objectives and operating conditions. Due to complex operational environments and internal degradation, such systems face high failure risks, potentially re- sulting in irreversible economic losses and safety hazards. Therefore, designing a rational mission abort policy during mission execution to balance mission reliability and system survivability is of significant practical importance. This paper addresses the specific charac- teristics of phased-mission systems and proposes a mission abort policy based on the degra- dation process. A multi-phase Wiener process with phase-specific parameters is employed to depict system degradation behavior in different phases. A dynamic decision-making frame- work is developed to determine whether to continue or abort the mission by minimizing the expected total cost. A dynamic programing algorithm is used to find the optimal mission abort policy. The structural properties of the optimal policy are examined. The mission success probability is derived recursively using a matrix method. The practical value of the proposed model is validated through numerical analysis and comparative studies.

Optimal mission abort policy for systems subject to random shocks based on virtual age process

This paper describes an optimal guidance policy for a vehicle to reach a relative position to a target, using information from a single flxed camera. Applying an extended Kalman flltering method, both of a velocity and a position of the vehicle relative to the target, can be estimated. However, estimates of a distance between the vehicle and the target are much worse than those of the other states. Therefore, in this paper, an optimal guidance policy is introduced that can reach the destination while maximizing the predicted accuracy of the range estimation. By limiting vehicle motion to a two dimensionals, the exact solution for control inputs for this optimization problem can be obtained. Simulation results show that the resulting optimal guidance policy gives far more accurate range estimation than a simple linear guidance policy. The automation of unmanned aerial vehicles(UAVs), has progressively developed in recent years. The development of sensors, such as GPS, made a large contribution. In most cases, UAV automated ∞ight control has been achieved by using multi- sensor fusion to estimate vehicle states accurately. However, because this becomes a complex and expensive system and not suitable for small unreusable UAVs, there is a need to make an autonomous ∞ight system which is simpler and less expensive. As seen in nature birds or insects, vision information can be almost exclusively utilized in an autonomous ∞ight system. This has the potential to improve system reliability, performance, and cost. Inspired from this idea, this paper introduces a method to navigate and guide a vehicle to reach a target using vision information from a single camera flxed to the vehicle. In this approach, an extended Kalman fllter(EKF) is utilized to estimate relative velocity and position. A single camera provides measurements of target horizontal and vertical position in its image, with noise. By applying an EKF to these two measurements, estimates of the relative position and velocity are obtained. The EKF estimation performance depends on camera motion, and the estimation will be improved by an appropriate trajectory generation. 1 In the case of the vehicle approaching straight toward the target, the velocity and position in the camera image plane can be estimated with relatively good accuracy. However, the estimation of distance between the vehicle(or camera) and the target includes large errors. If we use these poor range estimates to control a vehicle performing station-keeping with the target, a dangerous overshoot may occur. The reason for this large estimation error is that the vehicle does not necessary have enough lateral motion to estimate this depth. It is well known that the accuracy of range estimation depends on camera translating motion, and the best translation for range estimation is a motion parallel to its image plane. 2 Therefore, if we let the vehicle ∞y to the target with meandering path, we can obtain the more accurate range estimates and avoid the overshoot. Through the EKF, the variance of estimation errors for each state is available. To maximize the estimation accuracy, one should minimize the variance. In other words, to obtain the optimal ∞ight path to the relative position to the target, we have to set the performance index in order to minimize the variance of the estimation error of range and solve the resulting optimization problem subjected to a known camera motion. In this paper, we show that we can solve this problem analytically. Then, comparing simulation results of

Optimal radio access technology selection on heterogeneous networks

Energy-aware control for multiple unmanned aerial vehicles (UAVs) is one of the major research interests in UAV based networking. Yet few existing works have focused on how the network should react around the timing when the UAV lineup is changed. In this work, we study proactive self-remedy of energy-constrained UAV networks when one or more UAVs are short of energy and about to quit for charging. We target at an energy-aware optimal UAV control policy which proactively relocates the UAVs when any UAV is about to quit the network, rather than passively dispatches the remaining UAVs after the quit. Specifically, a deep reinforcement learning (DRL)-based self remedy approach, named SREC-DRL, is proposed to maximize the accumulated user satisfaction scores for a certain period within which at least one UAV will quit the network. To handle the continuous state and action space in the problem, the state-of-the-art algorithm of the actor-critic DRL, i.e., deep deterministic policy gradient (DDPG), is applied with better convergence stability. Numerical results demonstrate that compared with the passive reaction method, the proposed SREC-DRL approach shows a 12.12% gain in accumulative user satisfaction score during the remedy period.

In recent years, using the Unmanned Aerial Vehicle (UAV) as a Base Stations (BS) to cover users in wireless networks has increased dramatically. One of the main goals of integrating UAVs into wireless networks is to deploy UAVs in such a way that user expectations are met with the fewest number of UAVs. To achieve this aim, the coverage area of each UAV should include as many users as possible. Furthermore, the resources assigned to the backhaul links for such UAV deployments must fulfill the requirements of users served by each UAV. In this paper the goal is to position the least number of UAVs in a 3D position to cover cellular network users. To provide appropriate quality of service, we consider a maximum path loss allowed for the network. The path loss of potential links is affected by the propagation environment and might vary depending on network structure. To reflect this uncertainty, path loss is expressed as a random variable with a probability distribution based on environmental characteristics. As a result, we're dealing with an optimization problem with uncertain information. We use stochastic programming to work with uncertain information and formulate the UAV positioning and data rate assignment problem. The implementation results of our proposed mixed-binary linear mathematical model and Monte Carlo simulation in various scenarios show its optimum performance in different dimensions.

In this paper, a novel three dimensional (3D) positioning framework of fluid antenna system (FAS)-enabled unmanned aerial vehicles (UAVs) is developed. In the proposed framework, a set of controlled UAVs including an active UAV and four FAS-enabled passive UAVs cooperatively estimate the real-time 3D position of a target UAV. Here, the active UAV transmits a measurement signal to the passive UAVs via the reflection from the target UAV. Each passive UAV estimates the distance of the active-target-passive UAV link and selects an antenna port to share the distance information with the base station (BS), which calculates the real-time position of the target UAV. As the target UAV is moving due to its task operation, the controlled UAVs must optimize their trajectories and select optimal antenna port for transmitting the positioning information, aiming to estimate the real-time position of the target UAV. We formulate this problem as an optimization problem whose goal is to minimize the target UAV positioning error via optimizing the trajectories of all controlled UAVs and antenna port selection of passive UAVs. To address this problem, an attention-based recurrent multi-agent reinforcement learning (AR-MARL) scheme is proposed, which enables each controlled UAV to use the local Q function to determine its trajectory and antenna port while optimizing the target UAV positioning performance without knowing the trajectories and antenna port selections of other controlled UAVs. Different from current MARL methods that use feedforward neural networks to approximate Q functions, the proposed method uses a recurrent neural network (RNN) that incorporates historical state-action pairs of each controlled UAV, and an attention mechanism to analyze the importance of these historical state-action pairs, thus improving the global Q function approximation accuracy and the target UAV positioning accuracy. Simulation results show that the proposed scheme can reduce the average positioning error by up to 17.5% and 58.5% compared to the value decomposition based-MARL scheme with FAS and the proposed AR-MARL method without FAS.

In this chapter, the regulation of an Unmanned Aerial Vehicle (UAV) communication network is investigated in the presence of dynamic changes in the UAV lineup and user distribution. We target an optimal UAV control policy which is capable of identifying the upcoming change in the UAV lineup (quit or join-in) or user distribution, and proactively relocating the UAVs ahead of the change rather than passively dispatching the UAVs after the change. Specifically, a deep reinforcement learning (DRL)-based UAV control framework is developed to maximize the accumulated user satisfaction (US) score for a given time horizon which is able to handle the change in both the UAV lineup and user distribution. The framework accommodates the changed dimension of the state-action space before and after the UAV lineup change by deliberate state transition design. In addition, to handle the continuous state and action space, deep deterministic policy gradient (DDPG) algorithm, which is an actor-critic based DRL method, is exploited. Furthermore, to promote learning exploration around the timing of the change, the original DDPG scheme is adapted into an asynchronous parallel computing (APC) structure which leads to better training performance in both the critic and actor networks. Finally, extensive simulations are conducted to validate the convergence of the proposed learning approach, and demonstrate its capability in jointly handling the dynamics in UAV lineup and user distribution as well as its superiority over a passive reaction method.KeywordsUnmanned aerial vehicle (UAV)UAV-based communication networksDeep reinforcement learningDynamic user spatial distributionUAV trajectory design

In this paper, the problem of using one active unmanned aerial vehicle (UAV) and four passive UAVs to localize a 3D target UAV in real time is investigated. In the considered model, each passive UAV receives reflection signals from the target UAV, which are initially transmitted by the active UAV. The received reflection signals allow each passive UAV to estimate the signal transmission distance which will be transmitted to a base station (BS) for the estimation of the position of the target UAV. Due to the movement of the target UAV, each active/passive UAV must optimize its trajectory to continuously localize the target UAV. Meanwhile, since the accuracy of the distance estimation depends on the signal-to-noise ratio of the transmission signals, the active UAV must optimize its transmit power. This problem is formulated as an optimization problem whose goal is to jointly optimize the transmit power of the active UAV and trajectories of both active and passive UAVs so as to maximize the target UAV positioning accuracy. To solve this problem, a Z function decomposition based reinforcement learning (ZD-RL) method is proposed. Compared to value function decomposition based RL (VD-RL), the proposed method can find the probability distribution of the sum of future rewards to accurately estimate the expected value of the sum of future rewards thus finding better transmit power of the active UAV and trajectories for both active and passive UAVs and improving target UAV positioning accuracy. Simulation results show that the proposed ZD-RL method can reduce the positioning errors by up to 39.4% and 64.6%, compared to VD-RL and independent deep RL methods, respectively.

Data backups play an important role in effective recovery of system operations, especially, those in computing and Information Technology-related applications. By saving copies of information associated with the completed part of a mission task, a failed system, upon being repaired, can resume its operation from the latest backup point instead of having to repeat the entire work from scratch. This paper considers a repairable, real-time system performing a sequence of nonperiodic full backup (FB) and incremental backup (IB) procedures during its mission. The mission succeeds if a specified amount of work can be accomplished by the system within a deadline. New contributions of this paper are twofold. First, a numerical recursive method is proposed to model effects of the nonperiodic, mixed backup strategy in assessing mission success probability, and expected completion time of the considered system. The method is applicable to any distribution of FBs and IBs and has no limitation on system time-to-failure distributions. Second, the optimal backup policy problem is formulated and solved, which finds the distribution of FBs and IBs maximizing the mission success probability. Examples are provided to illustrate the proposed methodology as well as influence of different parameters on the optimal solution. Advantages of adopting nonperiodic backups over periodic ones are also illustrated.

Unmanned aerial vehicles (UAVs) play an essential role in emergency cases and adverse environments for applications like disaster detection and mine exploration. To process the massive volume of sensing data generated by various sensory payloads in these applications, existing works either compress deep learning (DL) models to conduct onboard computing, or offload raw data back to the resourceful ground station with the help of relay UAVs due to base station damage. However, the former sacrifices the inference accuracy of DL models (up to 10% accuracy loss), while the latter achieves high accuracy at the cost of significant latency, due to limited wireless communication resources in the multi-hop transmission. To address the problem, exploiting the resources of the UAV swarm including both task UAVs and relay UAVs, we build up an <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a</u>ll-<underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">s</u>ky <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a</u>utonomous com<underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</u>uting (ASAP) system to autonomously conduct collaborative computing in the swarm, to achieve both high accuracy and low latency of sensing data processing. In detail, we first propose a novel UAV swarm-native collaborative computing architecture, considering the general hierarchy and clustering structure of UAV swarms, as well as the characteristic of DL model execution. We then design an elastic efficient task scheduler to allocate computing tasks for UAVs, and update the scheduling scheme online when some UAVs are unavailable, with the aid of a lightweight and accurate DL inference performance predictor. Finally, we design an adaptive inter-UAV data compressor, to adapt to the limited and dynamic communication resources between UAVs. Experiment results on 24 airborne computers and five real-world UAVs show that, the proposed system can perform collaborative computing in a timely manner and effectively deal with situations when some UAVs become unavailable.

Joint optimization of trajectory and resource allocation in cellular-connected multi-UAV MEC networks

Synthesis of optimal controllers for a class of maximization problems

Unmanned aerial vehicle (UAV) is regarded as an effective technology in future wireless networks. However, due to the non-convexity feature of joint trajectory design and power allocation (JTDPA) issue, it is challenging to attain the optimal joint policy in multi-UAV networks. In this article, a multi-agent deep reinforcement learning-based approach is presented to achieve the maximum long-term network utility while satisfying the user equipments' quality of service requirements. Moreover, considering that the utility of each UAV is determined based on the network environment and other UAVs' actions, the JTDPA problem is modeled as a stochastic game. Due to the high computational complexity caused by the continuous action space and large state space, a multi-agent deep deterministic policy gradient method is proposed to obtain the optimal policy for the JTDPA issue. Numerical results indicate that our method can obtain the higher network utility and system capacity than other optimization methods in multi-UAV networks with lower computational complexity.