RIS-Assisted UAV-Based Dynamic Coverage Control for 6G-Enabled Internet of Everything Using Multi-Agent DRL

Due to flexible scheduling, and better transmission channel, unmanned aerial vehicle (UAV) can improve transmit performance of Internet of Things (IoT). In this paper, we propose an UAV-enabled multichannel IoT based on intelligent fog computing, where UAV as a relay forwards IoT's information to the data center under the control of fog computing base station in the case of terrestrial channel fading. The IoT's throughput are maximized by jointly optimizing subcarrier, power of IoT, and UAV, and UAV trajectory, subject to the constraints of information causality, maximum transmit power, and maximum UAV speed. The subcarriers are dynamically allocated according to their channel gains, the water filling algorithm is adopted to optimize the power for UAV, and IoT by fixing UAV trajectory, and the optimal UAV trajectory is achieved with successive convex approximation under the fixed power allocation. Then a jointly iterative optimization on subcarrier, power, and trajectory is presented to get the optimal solution. In addition, we propose a fairness optimization scheme to maximize the minimum transmit rate of IoT nodes. The simulations indicate the IoT with mobile UAV, and dynamic subcarrier allocation may achieve better transmit performance, and the fairness optimization can decrease the rate difference of nodes effectively.

In recent years, the Integrated Satellite Aerial Terrestrial (I‐SAT) network has garnered significant attention as an innovative and integrated communication system. However, it still encounters interference in the face of the complex external environment. In this context, reconfigurable intelligent surface (RIS) provides a key way of solving this problem and effectively improves the performance and stability of the I‐SAT network. This article considers the combination of unmanned aerial vehicle (UAV) and RIS and proposes a novel architecture for sub‐connected active RIS (ARIS) under the energy consumption constraints of UAV and ARIS. The authors first provide a UAV‐ARIS based position prediction strategy for the vehicle. Then, a joint RIS phase shift, amplification and UAV trail optimization algorithm is proposed to pursue a high achievable rate. The interference between each link and the total energy consumption are all taken into consideration. In addition, a deep deterministic policy gradient (DDPG) algorithm is utilized for the optimization problem, and achieves convergence in continuous action space. Finally, the simulation results affirm the precision of the proposed method in significantly enhancing performance compared to other schemes.

Caching the contents of unmanned aerial vehicles (UAVs) could significantly improve the content fetching performance of request users (RUs). In this paper, we study UAV trajectory design, content fetching, power allocation, and content placement problems in multi-UAV-aided networks, where multiple UAVs can transmit contents to the assigned RUs. To minimize the energy consumption of the system, we develop a constrained optimization problem that simultaneously designs UAV trajectory, power allocation, content fetching, and content placement. Since the original minimization problem is a mixed-integer nonlinear programming (MINLP) problem that is difficult to solve, the optimization problem was first transformed into a semi-Markov decision process (SMDP). Next, we developed a new technique to solve the joint optimization problem: option-based hierarchical deep reinforcement learning (OHDRL). We define UAV trajectory planning and power allocation as the low-level action space and content placement and content fetching as the high-level option space. Stochastic optimization can be handled using this strategy, where the agent makes high-level option selections, and the action is carried out at a low level based on the chosen option’s policy. When comparing the proposed approach to the current technique, the numerical results show that it can produce more consistent learning performance and reduced energy consumption.

In recent years, unmanned aerial vehicles (UAVs) have been widely used in wireless communication, attracting intensive attentions. UAVs can not only serve as relays, but also serve as aerial base station for ground users (GUs). However, limited energy means that they cannot work for long and cover a limited area of services. In this paper, we investigate 2D UAV trajectory design and power allocation in order to maximize the UAV's service time and downlink throughput. Based on deep reinforcement learning, we propose a deep deterministic policy gradient (DDPG) algorithm for trajectory design and power allocation (TDPA) to solve the energy efficient and communication service quality problem. The simulation results show that TDPA can extend the service time of UAV, improve the communication service quality, and realize the maximization of downlink throughput, which are significantly improved compared with existing methods.

Unmanned Aerial Vehicle (UAV) has emerged as a promising technology to provide wireless signals from air to the ground users in specific scenarios such as earthquakes, tsunamis and other disasters. The performance of the UAV is degraded when the signals are blocked by obstacles in dense urban scenarios. To address this issue and enhance the signal quality available to the ground users, Reconfigurable Intelligent Surface (RIS) has emerged as a new technological paradigm. It offers an intelligent configuration for the signal propagation environment by redirecting the signals to the users. In this article, we solve a non-convex optimization problem of RIS-assisted UAV network by jointly optimizing the RIS phase shift and 3D trajectory of UAV to maximize the energy efficiency of a rotatory-wing UAV. The considered optimization problem is solved using Deep Reinforcement Learning (DRL) based techniques in an on-line fashion to reduce the computational complexity. We leverage Twin-delayed Deep Deterministic Policy Gradient (TD3) to solve the problem by considering the UAV trajectory as a set of continuous actions. For comparison, we also use the Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG) and Double Deep Q-Network (DDQN) for continuous and discrete optimization of the UAV trajectory, respectively. Extensive simulations show that the TD3 outperforms all the considered DRL techniques with the highest energy efficiency and throughput, and the lowest propulsion energy.

In order to perform the operational advantages of manned aerial vehicle (MAV) /unmanned aerial vehicle (UAV) cooperative system, a method of MAV/UAV intelligent decision-making in air combat based on deep deterministic policy gradient (DDPG) algorithm is proposed. Based on the continuous action space, four typical intentions of MAV were introduced, and the optimization models under different intentions were established. The simulation results show that the multiple UAVs can realize the cooperative operations under the guidance of MAV with a favorable effect. In addition, it makes up the gap for the 1VS1 air combat.KeywordsMAV/UAVIntention guidingDecision-makingDeep deterministic policy gradientMultiple UAVs

In this work, we investigate an active reconfigurable intelligent surface (RIS)-aided non-orthogonal multiple access (NOMA)-enabled space-air-ground integrated network (SAGIN) with cognitive radio, leveraging the flexible deployment of an unmanned aerial vehicle (UAV) and the ubiquitous coverage of satellite networks. The UAV serves uplink and downlink users in the secondary network via NOMA and time division multiple access mechanisms, respectively, while satellites provide wireless backhaul for the UAV and primary users. We aim to maximize the weighted sum mean rate and energy efficiency for the secondary network by jointly the optimizing power allocation, the RIS reflection coefficients (RC), the user matching factors, and the UAV trajectory. We propose an alternating optimization framework based on the block coordinate ascent (BCA) technique, which decouples the problem into multiple variable blocks for alternating optimization until convergence. Moreover, we investigate the performance of energy-efficient active RIS with a sub-connected architecture, decoupling the RIS RC optimization into amplification factor and phase shift subproblems to be solved separately. Finally, simulation results validate the effectiveness of the proposed schemes, and demonstrate weakness of passive RIS and rationality and economics of sub-connected active RIS architecture.

In this paper, we explore the use of reconfigurable intelligent surface (RIS) in unmanned aerial vehicle (UAV) based multiuser downlink communications, where a flying UAV serves multiple single antenna users through multiple RISs mounted on various buildings. More specifically, we consider the selection of RISs based on the outdated and imperfect channel state information (CSI) of the composite UAV-RIS-User channels at the UAV. After selection process, the UAV communicates to the user via the selected RISs and also with the direct link. Particularly, we derive an infinite series based expression for selection probability of RISs under both the outdated and imperfect CSI of composite channels based selection scheme. We also derive the statistical distribution of instantaneously received signal-to-noise ratio (SNR) under outdated and imperfect CSI conditions of both the direct and composite links at the user. Next, using the derived statistics, we analyze the network’s performance in terms of the average coverage probability (ACP) and average bit error rate (ABER) over the complete UAV flight time. Moreover, we discuss the behavior of ACP and ABER for very small and very large values of UAV transmit power, respectively. It is depicted through numerical results that selecting more RISs from a group of small-sized RISs may not be as advantageous as selecting fewer RISs from a group of large-sized RISs. Moreover, we also demonstrate the effect of several system parameters such as number of RIS reflecting elements, number of selected RISs, the severity of UAV-RIS and RIS-User links, and the severity of imperfect and outdated CSI on the network’s performance. The analytical results are corroborated with Monte-Carlo simulations.

In this work, we investigate the performance of cognitive satellite-aerial networks (CSAN), where the satellite operates as the primary network (PN) and the unmanned aerial vehicle (UAV) acts as an aerial base station in the secondary network (SN), employing rate-splitting multiple access (RSMA) to serve user equipment (UEs). We address the problem of joint beamforming, rate allocation, UAV trajectory design, and reconfigurable intelligent surface (RIS) phase shift optimization in a CSAN. To maximize the sum rate while satisfying power and interference constraints, we propose a deep reinforcement learning (DRL) framework based on the twin delayed deep deterministic policy gradient (TD3) algorithm. Simulation results show that the proposed TD3-based scheme outperforms other DRL-based benchmarks.

Beamforming and non-orthogonal multiple access (NOMA) serve as two potential solutions for achieving spectral efficient communication in the fifth generation and beyond wireless networks. In this paper, we jointly apply a hybrid beamforming and NOMA techniques to an unmanned aerial vehicle (UAV)-carried wireless-powered mobile edge computing (MEC) system, within which the UAV is equipped with a wireless power charger and the MEC platform delivers energy and computing services to Internet of Things (IoT) devices. Our aim is to maximize the sum computation rate at all IoT devices whilst satisfying the constraint of energy harvesting and coverage. The resultant optimization problem is non-convex involving joint optimization of the UAV’s 3D placement and hybrid beamforming matrices as well as computation resource allocation in both partial and binary offloading patterns, and thus is quite difficult to tackle directly. By applying the polyhedral annexation method and the deep deterministic policy gradient (DDPG) algorithm, we develop an effective algorithm to derive the closed-form solution for the optimal 3D deployment of the UAV, and find the solution for the hybrid beamformer. Two resource allocation algorithms for partial and binary offloading patterns are thereby proposed. Simulation results verify that our designed algorithms achieve a significant computation performance enhancement as compared to the benchmark schemes.

Piezoelectric inkjet printing has important applications in high-precision manufacturing, but precise control of droplet quality faces challenges such as the nonlinear coupling between drive waveform parameters and droplet characteristics, as well as environmental disturbances. To address these issues, this paper proposes a closed-loop droplet control method based on a curriculum deep deterministic policy gradient (DDPG) algorithm, with the standard deviation of droplet volume and velocity as the main optimization objectives. To tackle the complex mapping relationships and boost training efficiency, we first built a high-fidelity simulation environment using a four-layer deep residual network trained on real-world industrial ink droplet data. Following that, we designed a dynamic weight-adaptive composite reward mechanism and integrated a curriculum learning strategy with the Deep Deterministic Policy Gradient (DDPG) algorithm. Relying on our self-developed droplet observation platform, we achieve closed-loop control of the standard deviation of droplet volume and velocity. Experimental results show that this method can precisely control the droplet volume within ±0.1 pl of the target value, while keeping the velocity standard deviation below 40 mm/s. The adjustment process is stable without severe oscillations, and the method demonstrates strong robustness against environmental disturbances such as ink supply pressure. This provides an efficient and practical solution for the automated control of droplet quality in piezoelectric inkjet printing.

The unmanned aerial vehicle (UAV) enabled mobile edge computing (MEC) is a promising paradigm for providing extensive coverage and additional computation services to the Internet of Things (IoT) devices with smart applications, which are normally computation-intensive and delay-critical, especially in some remote areas or emerging scenarios without available infrastructure. However, considering the restricted computation capabilities of UAVs, even multiple UAVs may not satisfy the quality of service (QoS) requirement of IoT users with heavy-computation applications. In this paper, a heterogenous-multiple-UAVs enabled collaborative aerial-ground MEC framework is proposed. The total computation bits maximization problem under the time-division multiple access (TDMA) scheme and partial offloading mode is formulated by jointly optimizing the user association, the CPU-cycle frequencies of users and computing UAVs, the transmit power of users and UAVs, the offloading time of users and UAVs, as well as the UAVs' trajectories. Due to the non-convexity of the original problem with the nonlinear coupling of different variables, a two-layered alternative optimization (TLAO) algorithm is proposed, where the outer layer optimizes the user association and resource allocation, while the inner layer optimizes the UAV trajectory scheduling. Simulation results demonstrate that our proposed TLAO scheme can improve the total computation bits compared with other benchmark schemes, and achieve fast convergence. Moreover, comparative results also verify that our proposed strategy is flexible to apply to homogeneous multi-UAVs and single-UAV scenarios.

Collaborative computation offloading and wireless charging scheduling in multi-UAV-assisted MEC networks: A TD3-based approach

In this chapter, the regulation of an Unmanned Aerial Vehicle (UAV) communication network is investigated in the presence of dynamic changes in the UAV lineup and user distribution. We target an optimal UAV control policy which is capable of identifying the upcoming change in the UAV lineup (quit or join-in) or user distribution, and proactively relocating the UAVs ahead of the change rather than passively dispatching the UAVs after the change. Specifically, a deep reinforcement learning (DRL)-based UAV control framework is developed to maximize the accumulated user satisfaction (US) score for a given time horizon which is able to handle the change in both the UAV lineup and user distribution. The framework accommodates the changed dimension of the state-action space before and after the UAV lineup change by deliberate state transition design. In addition, to handle the continuous state and action space, deep deterministic policy gradient (DDPG) algorithm, which is an actor-critic based DRL method, is exploited. Furthermore, to promote learning exploration around the timing of the change, the original DDPG scheme is adapted into an asynchronous parallel computing (APC) structure which leads to better training performance in both the critic and actor networks. Finally, extensive simulations are conducted to validate the convergence of the proposed learning approach, and demonstrate its capability in jointly handling the dynamics in UAV lineup and user distribution as well as its superiority over a passive reaction method.KeywordsUnmanned aerial vehicle (UAV)UAV-based communication networksDeep reinforcement learningDynamic user spatial distributionUAV trajectory design

Despite its flexibility, unmanned aerial vehicle (UAV) communications are susceptible to eavesdropping due to the open nature of wireless channels and the broadcasting nature of wireless signals. This paper studies secure UAV communications and proposes a method to optimize the minimum secrecy rate of the system by using interference technology to enhance it. To this end, the system not only deploys multiple UAV base stations (BSs) to provide services to legitimate users but also assigns dedicated UAV jammers to send interference signals to active or potential eavesdroppers to disrupt their eavesdropping effectiveness. Based on this configuration, we formulate the optimization process of parameters such as the user association variables, UAV trajectory, and output power as a sequential decision-making problem and use the single-agent soft actor-critic (SAC) algorithm and twin delayed deep deterministic policy gradient (TD3) algorithm to achieve joint optimization of the core parameters. In addition, for specific scenarios, we also use the multi-agent soft actor-critic (MASAC) algorithm to solve the joint optimization problem mentioned above. The numerical results show that the normalized average secrecy rate of the MASAC algorithm increased by more than 6.6% and 14.2% compared with that of the SAC and TD3 algorithms, respectively.