Communication-Enhanced Deep Reinforcement Learning for AoI- and Energy-Aware UAV Trajectory Optimization in MCS Data Collection

In this paper, we study the three-dimensional (3D) trajectory optimization problems of unmanned aerial vehicles (UAV) with a solar energy supply, aiming to provide communication coverage for mobile users on the ground. In general, the higher UAVs fly, the more solar energy they collect, but the smaller the range of coverage they could achieve, and vice versa. How to plan optimal trajectories for UAVs so that more users can be encompassed, while allowing UAVs to collect enough solar energy, is a challenging issue. Moreover, we also consider how geographically fair coverage for each ground user can be achieved. To solve these problems, we designed a multiple solar-powered UAV (SP-UAV) energy consumption model and a fairness model, while designed an observation space, state space, action space, and reward function. Then, we proposed a multiple SP-UAV 3D trajectory optimization algorithm based on deep reinforcement learning (DRL). Our algorithm is able to balance the energy consumption of UAVs to extend the system’s lifetime, while avoiding both collisions and flying out of communication range. Finally, we trained our model through simulation experiments and conducted comparative experiments and analysis based on real network topology data. The results show that our algorithm is superior to the existing typical algorithms in coverage, fairness, and lifetime.

In recent years, the utilization of unmanned aerial vehicles (UAVs) has surged in a variety of applications, including weather monitoring, emergency search and rescue operations, and smart agriculture. In these applications, UAVs are instrumental in executing data collection tasks. However, due to the scarcity of resources such as battery capacity, the duration of work is limited, necessitating the optimization of UAV trajectories and resource allocation. In this paper, we introduce a hierarchical task scheduling and resource allocation scheme for UAV data collection tasks, which incorporates the deep reinforcement learning (DRL) technology into a two-layer training framework. The continuous actions, including UAV trajectories, UAV communication power, and UAV CPU main frequency, are first optimized in the lower layer, while the upper one concentrates on the discrete action space for the target allocation. In particular, the well-trained lower-layer network is integrated into the upper layer, facilitating rapid global reward feedback between the two-layer network and accelerating the training process of the upper-layer network. Experimental results demonstrate that the proposed approach achieves superior performance than the baseline methods in terms of the amount of collected data as well as UAV energy consumption.

The main aim of future mobile networks is to provide secure, reliable, intelligent, and seamless connectivity. It also enables mobile network operators to ensure their customer’s a better quality of service (QoS). Nowadays, Unmanned Aerial Vehicles (UAVs) are a significant part of the mobile network due to their continuously growing use in various applications. For better coverage, cost-effective, and seamless service connectivity and provisioning, UAVs have emerged as the best choice for telco operators. UAVs can be used as flying base stations, edge servers, and relay nodes in mobile networks. On the other side, Multi-access Edge Computing (MEC) technology also emerged in the 5G network to provide a better quality of experience (QoE) to users with different QoS requirements. However, UAVs in a mobile network for coverage enhancement and better QoS face several challenges such as trajectory designing, path planning, optimization, QoS assurance, mobility management, etc. The efficient and proactive path planning and optimization in a highly dynamic environment containing buildings and obstacles are challenging. So, an automated Artificial Intelligence (AI) enabled QoS-aware solution is needed for trajectory planning and optimization. Therefore, this work introduces a well-designed AI and MEC-enabled architecture for a UAVs-assisted future network. It has an efficient Deep Reinforcement Learning (DRL) algorithm for real-time and proactive trajectory planning and optimization. It also fulfills QoS-aware service provisioning. A greedy-policy approach is used to maximize the long-term reward for serving more users with QoS. Simulation results reveal the superiority of the proposed DRL mechanism for energy-efficient and QoS-aware trajectory planning over the existing models.

This paper investigates the energy transfer and data collection problem of multiple unmanned aerial vehicle (UAV)-assisted wireless-powered Internet of Things (IoT) networks. To ensure information freshness for IoT devices and reduce UAVs’ energy consumption, we minimize the average Age of Information (AoI) of IoT devices by jointly optimizing the energy harvesting (EH) and data collection time for IoT devices, the selection of data collection points (DCPs), DCP-IoT associations, and task assignment, flight speed, and trajectories of UAVs, subject to the limited endurance of UAVs. As this problem is nonconvex, we propose a novel DCP association and trajectory-planning scheme that seeks age-optimal solutions through an iterative three-step process. First, we calculate the EH and data collection time for IoT devices using Karush–Kuhn–Tucker (KKT) conditions. Then, we introduce an optimal hovering time allocation-based affinity propagation (OHTAP) clustering algorithm to determine optimal DCP locations and establish DCP-IoT associations. Finally, we develop two algorithms to optimize UAVs’ trajectories: an improved partheno-genetic algorithm with enhancement mechanisms (EIPGA) and a hybrid algorithm that combines improved MinMax k-means clustering with EIPGA. Numerical results confirm that our scheme consistently outperforms benchmark schemes in AoI performance and solution stability across diverse scenarios.

Energy-efficient trajectory optimization algorithm based on K-medoids clustering and gradient-based optimizer for multi-UAV-assisted mobile edge computing systems

With the ongoing advancement of electric power Internet of Things (IoT), traditional power inspection methods face challenges such as low efficiency and high risk. Unmanned aerial vehicles (UAVs) have emerged as a more efficient solution for inspecting power facilities due to their high maneuverability, excellent line-of-sight communication capabilities, and strong adaptability. However, UAVs typically grapple with limited computational power and energy resources, which constrain their effectiveness in handling computationally intensive and latency-sensitive inspection tasks. In response to this issue, we propose a UAV task offloading strategy based on deep reinforcement learning (DRL), which is designed for power inspection scenarios consisting of mobile edge computing (MEC) servers and multiple UAVs. Firstly, we propose an innovative UAV-Edge server collaborative computing architecture to fully exploit the mobility of UAVs and the high-performance computing capabilities of MEC servers. Secondly, we established a computational model concerning energy consumption and task processing latency in the UAV power inspection system, enhancing our understanding of the trade-offs involved in UAV offloading strategies. Finally, we formalize the task offloading problem as a multi-objective optimization issue and simultaneously model it as a Markov Decision Process (MDP). Subsequently, we proposed a task offloading algorithm based on a Deep Deterministic Policy Gradient (OTDDPG) to obtain the optimal task offloading strategy for UAVs. The simulation results demonstrated that this approach outperforms baseline methods with significant improvements in task processing latency and energy consumption.

The use of unmanned aerial vehicles (UAVs) for collecting data from sensors has garnered considerable attention in both academic and industrial fields in recent years. To assess the freshness of the information gathered, the age of information (AoI) metric has emerged as an important tool, particularly in time-sensitive applications such as traffic management or monitoring critical issues like fire hazards. This study explores a scenario where a UAV, supported by multiple intelligent reflecting surfaces (IRSs), periodically gathers data from distributed internet of things devices (IoTDs). To minimize the average AoI across all devices, a comprehensive optimization approach is employed. Key elements such as the UAV’s trajectory, device transmission scheduling, timeslot duration, IRS selection factor, and the phase shift matrix of the selected IRS are jointly optimized. However, the problem’s complexity arises from its mixed-integer nature—IRS selection and device scheduling coefficients require integer values—and the strong interdependence of channel conditions with the UAV’s flight path and IRS phase shifts. These challenges make the problem difficult to solve. To address this, the primary optimization problem is framed as a Markov decision process (MDP). A deep reinforcement learning (DRL)-based approach is then adopted to determine the optimal UAV trajectory, device scheduling, and timeslot duration. Additionally, a location-based method is employed for selecting the suitable IRS, while a closed-form solution is used to design the phase shift matrix of the selected IRS. Results evidence improvements in successful transmission and fairness selection of devices when using multiple IRSs and variable timeslots.

Unmanned aerial vehicle (UAV) has been widely deployed in efficient data collection for Internet of Things (IoT). UAV can not only act as a relay, but also as an energy source to provide information and energy transmission for ground sensor nodes (SNs). This paper studies the efficient multi-UAV-assisted data collection problem in wireless powered IoT. Specifically multiple UAVs wirelessly charge SNs using radio frequency (RF) energy transfer, and the SNs then use the harvested energy to upload the updates of the sensed information to the UAVs, thus improving the freshness of collected data and extending the service time of the SNs. The problem is modeled as a partially observed Markov decision process (POMDP) with a large observation and action space, where each UAV acts as an intelligent agent to learn the environment and make decisions independently. The value-decomposition network (VDN) algorithm is employed to find the optimal strategy in the multi-agent deep reinforcement learning framework. Simulation results validate the effectiveness of the proposed data collection approach compared to two baseline policies.

The rapid expansion of mobile devices with enhanced sensing and computing capabilities has driven the growth of mobile crowd sensing (MCS), enabling applications that collect large datasets from sources like smartphones and smartwatches. However, this data aggregation raises substantial security and privacy concerns, especially when MCS integrates with unmanned aerial vehicles (UAVs), where potential risks are further amplified. This study identifies and analyzes specific security and privacy threats in UAV-based MCS through the framework of the confidentiality, integrity, and availability (CIA) triad. We categorize potential vulnerabilities and propose comprehensive countermeasures targeting hardware, software, and communication models. Our findings outline strategic and actionable countermeasures to mitigate identified risks, thus ensuring data integrity and reliable functionality within MCS systems. Additionally, we present a security scenario involving mitigation suggested for data integrity and recovery. This work underscores the critical need for robust security frameworks in UAV-enhanced MCS applications, offering a holistic approach to mitigate emerging security threats.

Unmanned aerial vehicle (UAV)-assisted mobile communication has been studied in recent years. UAVs can be used as aerial base stations (BSs) to improve the performance of terrestrial mobile network. In this article, mobile data offloading with UAV trajectory optimization is investigated. To tackle with the delay of requesting data and the immediacy of requested data at the same time, a new metric named delanalty is newly proposed. The delanalty metric jointly considers the delay of user requesting data, the immediacy of requested data file, and the quantity of residual requesting data. A find max delanalty user mechanism is proposed to eliminate the user who has the largest delay time. Furthermore, an actor–critic (AC)-based deep reinforcement learning (DRL) algorithm called AC -based delanalty trajectory optimization (ACDTO) algorithm is proposed to solve UAV’s trajectory optimization problem. Simulation results show that the proposed ACDTO algorithm can find an optimal flight trajectory with minimal delanalty for all users.

Mobile crowdsensing (MCS) represents a new sensing paradigm that utilizes the smart mobile devices to collect and share data. Traditional MCS systems mainly leverages the people carried smartphones and other wearable devices which are constrained by the limited sensing capability and battery power. With the popularity of unmanned vehicles like unmanned aerial vehicles (UAVs) and driverless cars, they can provide much more reliable, accurate and cost-efficient sensing services due to to their equipped more powerful sensors. In this paper, we propose a distributed control framework for energy-efficient and DIstributed VEhicle navigation with chaRging sTations, called “e-Divert”. It is a distributed multi-agent deep reinforcement learning (DRL) solution, which uses a convolutional neural network (CNN) to extract useful spatial features as the input to the actor-critic network to produce a real-time action. Also, e-Divert incorporates a distributed prioritized experience replay for better exploration and exploitation, and a long short-term memory (LSTM) enabled N-step temporal sequence modeling module. The solution fully explores the spatiotemporal nature of the considered scenario for better vehicle cooperation and competition between themselves and charging stations, to maximize the energy efficiency, data collection ratio, geographic fairness, and minimize the energy consumption simultaneously. Through extensive simulations, we find an appropriate set of hyperparameters that achieve the best performance, i.e., 5 actors in Ape-X architecture, priority exponent 0.5, and LSTM sequence length 3. Finally, we compare with four baselines including one state-of-the-art approach MADDPG. Results show that our proposed e-Divert significantly improves the energy efficiency, as compared to MADDPG, by 3.62 and 2.36 times on average when varying different numbers of vehicles and charging stations, respectively.

This paper studies the integration of data collection and offloading for maritime Internet of Things (IoT) systems with multiple unmanned aerial vehicles (UAVs). In the considered multi-UAV maritime IoT system, the UAVs act as the aerial base stations to complete the missions of data collection from buoys and data offloading to the offshore base station (OBS). In this case, the UAVs need to adaptively select the mission mode between data collection and data offloading according to the network resources and mission requirements. In this paper, we aimed to minimize the completion time of data collection and offloading missions for all UAVs by jointly optimizing the UAV trajectories, mission mode selection, transmit power of buoys, and association relationships between the UAVs and buoy/OBS. In order to solve the mixed-integer non-convex minimization problem, we first designed a multi-agent deep reinforcement learning algorithm based on a hybrid discrete and continuous action space to preliminarily obtain the UAV trajectories, mission mode selection, and the transmit power of buoys. Then, we propose an algorithm based on the stable marriage problem to determine the buoy–UAV and UAV–OBS association relationships. Finally, the simulation results show that the proposed algorithms can effectively shorten the total mission completion time of data collection and offloading for the multi-UAV-assisted maritime IoT system.

Unmanned aerial vehicle (UAV)-enabled communication has emerged as an appealing technology for efficient data collection in wireless sensor networks (WSNs). This paper considers a scenario where multiple UAVs collect data from a group of sensor nodes (SNs) on the ground. We study the fundamental tradeoff between the aerial cost , which is defined by the propulsion energy consumption and operation costs of all UAVs, and the ground cost , which is defined as the energy consumption of all SNs. To characterize such a tradeoff, an optimization problem is formulated to minimize the weighted sum of the above two costs, by optimizing the UAV trajectory jointly with wake-up time allocation, as well as the transmit power of all SNs. As the formulated problem is non-convex, it is difficult to be optimally solved in general. To tackle this issue, we decouple it into two sub-problems: UAV trajectory and wake-up time allocation optimization, as well as SN transmit power optimization. We propose an iterative algorithm to solve the two sub-problems by leveraging successive convex approximation and alternating optimization techniques. In addition, a new approach is proposed to design the UAV initial trajectory with multiple travelling salesman problem (MTSP) technique. Simulations are conducted to corroborate our study and show the flexible tradeoff achieved by the proposed design for cost balance between UAVs and SNs.

With the advantages of real-time data processing and flexible deployment, unmanned aerial vehicle (UAV)-assisted mobile edge computing systems are widely used in both civil and military fields. However, due to limited energy, it is usually difficult for UAVs to stay in the air for long periods and to perform computational tasks. In this paper, we propose a full-duplex air-to-air communication system (A2ACS) model combining mobile edge computing and wireless power transfer technologies, aiming to effectively reduce the computational latency and energy consumption of UAVs, while ensuring that the UAVs do not interrupt the mission or leave the work area due to insufficient energy. In this system, UAVs collect energy from external air-edge energy servers (AEESs) to power onboard batteries and offload computational tasks to AEESs to reduce latency. To optimize the system’s performance and balance the four objectives, including the system throughput, the number of low-power alarms of UAVs, the total energy received by UAVs and the energy consumption of AEESs, we develop a multi-objective optimization framework. Considering that AEESs require rapid decision-making in a dynamic environment, an algorithm based on multi-agent deep deterministic policy gradient (MADDPG) is proposed, to optimize the AEESs’ service location and to control the power of energy transfer. While training, the agents learn the optimal policy given the optimization weight conditions. Furthermore, we adopt the K-means algorithm to determine the association between AEESs and UAVs to ensure fairness. Simulated experiment results show that the proposed MODDPG (multi-objective DDPG) algorithm has better performance than the baseline algorithms, such as the genetic algorithm and other deep reinforcement learning algorithms.

In recent years, unmanned aerial vehicles (UAVs) have been widely used in wireless communication, attracting intensive attentions. UAVs can not only serve as relays, but also serve as aerial base station for ground users (GUs). However, limited energy means that they cannot work for long and cover a limited area of services. In this paper, we investigate 2D UAV trajectory design and power allocation in order to maximize the UAV's service time and downlink throughput. Based on deep reinforcement learning, we propose a deep deterministic policy gradient (DDPG) algorithm for trajectory design and power allocation (TDPA) to solve the energy efficient and communication service quality problem. The simulation results show that TDPA can extend the service time of UAV, improve the communication service quality, and realize the maximization of downlink throughput, which are significantly improved compared with existing methods.