Optimization strategy of UAV‐ARIS assisted vehicular communication system

The Internet of Everything (IoE) is accelerating the demand for intelligent, low-latency, and highly reliable communication systems to support automation and real-time decision-making. In dense urban and industrial environments, unmanned aerial vehicles (UAVs) are increasingly utilized to extend network coverage, improve connectivity, and enable dynamic data collection. However, managing RIS-assisted UAV-enabled IoE networks poses significant challenges, including accurate signal prediction, high computational complexity, and decentralized task assignment. To address these issues, we propose a novel RIS-empowered UAV-based Dynamic Area of Coverage (DAC) architecture. In this framework, UAVs equipped with reconfigurable intelligent surfaces (RIS) adaptively adjust the phase of reflected signals to optimize wireless channel conditions, suppress interference, and enhance signal quality.We formulate the Dynamic Area of Coverage with Location, Resource Allocation, and Trajectory Optimization (DAC-LRT) problem as a mixed-integer nonlinear programming (MINLP) model, aiming to jointly optimize UAV positioning, power distribution, and trajectory control to maximize real-time downlink capacity and ensure energy efficiency. To solve the DAC-LRT problem in dynamic and large-scale IoE environments, we design a Multi-Agent Distributed Deep Deterministic Policy Gradient (MAD3PG) algorithm. MAD3PG enables decentralized and adaptive policy learning by allowing UAVs to derive optimal actions directly from environmental observations. Simulation results demonstrate that our proposed approach significantly outperforms state-of-the-art methods, achieving improvements of 82.92%, 78.02%, and 71.9% in downlink capacity, coverage ratio, average throughput, and spectral efficiency over Deep Deterministic Policy Gradient (DDPG), Asynchronous Advantage Actor-Critic (A3C), and Greedy algorithms, respectively.

In order to perform the operational advantages of manned aerial vehicle (MAV) /unmanned aerial vehicle (UAV) cooperative system, a method of MAV/UAV intelligent decision-making in air combat based on deep deterministic policy gradient (DDPG) algorithm is proposed. Based on the continuous action space, four typical intentions of MAV were introduced, and the optimization models under different intentions were established. The simulation results show that the multiple UAVs can realize the cooperative operations under the guidance of MAV with a favorable effect. In addition, it makes up the gap for the 1VS1 air combat.KeywordsMAV/UAVIntention guidingDecision-makingDeep deterministic policy gradientMultiple UAVs

This paper presents a parameter selection method of notch filters for suppressing mechanical resonances in industrial servo systems using the deep deterministic policy gradient (DDPG) algorithm. Several methods for tuning the notch filter parameters were studied such as the fast-Fourier-transform-based methods, extended-Kalman-filter-based methods and adaptive notch filter methods. However, these methods do not find the Q parameters of notch filters which play an important role in determining the system stability, and do not consider the cases in which multiple notch filters are required. Deep-Q-network-based method was developed to solve these problems, but the notch filter parameter tuning is limited to discrete action spaces. This paper develops a new parameter selection method of notch filters, using the DDPG algorithm. DDPG algorithm, which is a model-free and actor-critic algorithm using deep neural networks, is utilized for its capability to operate over continuous action spaces. Experiments are performed using an actual industrial servo system to demonstrate that the developed parameter selection method successfully finds the notch filter parameters to suppress the resonances of the system.

In this paper, we explore the use of reconfigurable intelligent surface (RIS) in unmanned aerial vehicle (UAV) based multiuser downlink communications, where a flying UAV serves multiple single antenna users through multiple RISs mounted on various buildings. More specifically, we consider the selection of RISs based on the outdated and imperfect channel state information (CSI) of the composite UAV-RIS-User channels at the UAV. After selection process, the UAV communicates to the user via the selected RISs and also with the direct link. Particularly, we derive an infinite series based expression for selection probability of RISs under both the outdated and imperfect CSI of composite channels based selection scheme. We also derive the statistical distribution of instantaneously received signal-to-noise ratio (SNR) under outdated and imperfect CSI conditions of both the direct and composite links at the user. Next, using the derived statistics, we analyze the network’s performance in terms of the average coverage probability (ACP) and average bit error rate (ABER) over the complete UAV flight time. Moreover, we discuss the behavior of ACP and ABER for very small and very large values of UAV transmit power, respectively. It is depicted through numerical results that selecting more RISs from a group of small-sized RISs may not be as advantageous as selecting fewer RISs from a group of large-sized RISs. Moreover, we also demonstrate the effect of several system parameters such as number of RIS reflecting elements, number of selected RISs, the severity of UAV-RIS and RIS-User links, and the severity of imperfect and outdated CSI on the network’s performance. The analytical results are corroborated with Monte-Carlo simulations.

Piezoelectric inkjet printing has important applications in high-precision manufacturing, but precise control of droplet quality faces challenges such as the nonlinear coupling between drive waveform parameters and droplet characteristics, as well as environmental disturbances. To address these issues, this paper proposes a closed-loop droplet control method based on a curriculum deep deterministic policy gradient (DDPG) algorithm, with the standard deviation of droplet volume and velocity as the main optimization objectives. To tackle the complex mapping relationships and boost training efficiency, we first built a high-fidelity simulation environment using a four-layer deep residual network trained on real-world industrial ink droplet data. Following that, we designed a dynamic weight-adaptive composite reward mechanism and integrated a curriculum learning strategy with the Deep Deterministic Policy Gradient (DDPG) algorithm. Relying on our self-developed droplet observation platform, we achieve closed-loop control of the standard deviation of droplet volume and velocity. Experimental results show that this method can precisely control the droplet volume within ±0.1 pl of the target value, while keeping the velocity standard deviation below 40 mm/s. The adjustment process is stable without severe oscillations, and the method demonstrates strong robustness against environmental disturbances such as ink supply pressure. This provides an efficient and practical solution for the automated control of droplet quality in piezoelectric inkjet printing.

In this work, we investigate the performance of cognitive satellite-aerial networks (CSAN), where the satellite operates as the primary network (PN) and the unmanned aerial vehicle (UAV) acts as an aerial base station in the secondary network (SN), employing rate-splitting multiple access (RSMA) to serve user equipment (UEs). We address the problem of joint beamforming, rate allocation, UAV trajectory design, and reconfigurable intelligent surface (RIS) phase shift optimization in a CSAN. To maximize the sum rate while satisfying power and interference constraints, we propose a deep reinforcement learning (DRL) framework based on the twin delayed deep deterministic policy gradient (TD3) algorithm. Simulation results show that the proposed TD3-based scheme outperforms other DRL-based benchmarks.

In this chapter, the regulation of an Unmanned Aerial Vehicle (UAV) communication network is investigated in the presence of dynamic changes in the UAV lineup and user distribution. We target an optimal UAV control policy which is capable of identifying the upcoming change in the UAV lineup (quit or join-in) or user distribution, and proactively relocating the UAVs ahead of the change rather than passively dispatching the UAVs after the change. Specifically, a deep reinforcement learning (DRL)-based UAV control framework is developed to maximize the accumulated user satisfaction (US) score for a given time horizon which is able to handle the change in both the UAV lineup and user distribution. The framework accommodates the changed dimension of the state-action space before and after the UAV lineup change by deliberate state transition design. In addition, to handle the continuous state and action space, deep deterministic policy gradient (DDPG) algorithm, which is an actor-critic based DRL method, is exploited. Furthermore, to promote learning exploration around the timing of the change, the original DDPG scheme is adapted into an asynchronous parallel computing (APC) structure which leads to better training performance in both the critic and actor networks. Finally, extensive simulations are conducted to validate the convergence of the proposed learning approach, and demonstrate its capability in jointly handling the dynamics in UAV lineup and user distribution as well as its superiority over a passive reaction method.KeywordsUnmanned aerial vehicle (UAV)UAV-based communication networksDeep reinforcement learningDynamic user spatial distributionUAV trajectory design

This work proposes a non-singular fast terminal sliding mode control (NFTSMC) strategy based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and a nonlinear disturbance observer (NDO) to address the issues of modeling errors, motion disturbances, and transmission friction in robotic manipulators. Firstly, a novel modular serial 5-DOF robotic manipulator configuration is designed, and its kinematic and dynamic models are established. Secondly, a nonlinear disturbance observer is employed to estimate the total disturbance of the system and apply feedforward compensation. Based on boundary layer technology, an improved NFTSMC method is proposed to accelerate the convergence of tracking errors, reduce chattering, and avoid singularity issues inherent in traditional terminal sliding mode control. The stability of the designed control system is proved using Lyapunov stability theory. Subsequently, a deep reinforcement learning (DRL) agent based on the TD3 algorithm is trained to adaptively adjust the control gains of the non-singular fast terminal sliding mode controller. The dynamic information of the robotic manipulator is used as the input to the TD3 agent, which searches for optimal controller parameters within a continuous action space. A composite reward function is designed to ensure the stable and efficient learning of the TD3 agent. Finally, the motion characteristics of three joints for the designed 5-DOF robotic manipulator are analyzed. The results show that compared to the non-singular fast terminal sliding mode control algorithm based on a nonlinear disturbance observer (NDONFT), the non-singular fast terminal sliding mode control algorithm integrating a nonlinear disturbance observer and the Twin Delayed Deep Deterministic Policy Gradient algorithm (TD3NDONFT) reduces the mean absolute error of position tracking for the three joints by 7.14%, 19.94%, and 6.14%, respectively, and reduces the mean absolute error of velocity tracking by 1.78%, 9.10%, and 2.11%, respectively. These results verify the effectiveness of the proposed algorithm in enhancing the trajectory tracking accuracy of the robotic manipulator under unknown time-varying disturbances and demonstrate its strong robustness against sudden disturbances.

In this letter, a novel application method of reconfigurable intelligent surfaces (RIS) is proposed, which installs a RIS on an unmanned aerial vehicle (UAV) rather than fixed to buildings. When the location information of the eavesdropper is unknown, the UAV’s rapid deployment and RIS’s ability to change the transmission channel are utilized to dynamically optimize the UAV’s flight trajectory and RIS’s phase shift. The trajectory of the UAV and phase shift of the RIS are optimized to avoid the risk of potential eavesdropping and improve the average downlink secrecy rate of the system. This letter utilizes Q-learning and Deep Q-network (DQN) algorithms to solve the problem. Experiments show that RIS can reduce the flight distance of the UAV on the premise of ensuring security. The scenarios where the two algorithms are applicable are analyzed, and the DQN algorithm is more suitable for the scenarios of large state and action spaces. Also, the experimental results imply that the RIS-assisted UAV improves the average downlink secrecy rate and system security.

Securing communication between multiple users efficiently while there are too many potential eavesdroppers has become an important issue with the rise of the Internet of Things (IoTs). This paper extends on earlier research, moving from a single‐user and single‐eavesdropper scenario to a complex multiuser and multieavesdropper context, and incorporates an advanced physical layer security (PLS) technique for the first time. Using reconfigurable intelligent surfaces (RISs) enhances the strength and quality of signals for intended users, while those to the unintended users are suppressed. Real‐time control of the RIS phase shifts is enabled through a deep deterministic policy gradient (DDPG) algorithm and this control significantly changes the trade‐off between security and energy wastage. The simulation results demonstrate that the developed approach can scale up in densely populated urban centers, while increasing the bit error rate (BER) performance and the overall energy efficiency across different wireless mobile channels.

The research on autonomous driving based on deep reinforcement learning algorithms is a research hotspot. Traditional autonomous driving requires human involvement, and the autonomous driving algorithms based on supervised learning must be trained in advance using human experience. To deal with autonomous driving problems, this paper proposes an improved end-to-end deep deterministic policy gradient (DDPG) algorithm based on the convolutional block attention mechanism, and it is called multi-input attention prioritized deep deterministic policy gradient algorithm (MAPDDPG). Both the actor network and the critic network of the model have the same structure with symmetry. Meanwhile, the attention mechanism is introduced to help the vehicles focus on useful environmental information. The experiments are conducted in the open racing car simulator (TORCS)and the results of five experiment runs on the test tracks are averaged to obtain the final result. Compared with the state-of-the-art algorithm, the maximum reward increases from 62,207 to 116,347, and the average speed increases from 135 km/h to 193 km/h, while the number of success episodes to complete a circle increases from 96 to 147. Also, the variance of the distance from the vehicle to the center of the road is compared, and the result indicates that the variance of the DDPG is 0.6 m while that of the MAPDDPG is only 0.2 m. The above results indicate that the proposed MAPDDPG achieves excellent performance.

Communication with unmanned aerial vehicles (UAVs) in current terrestrial networks suffers from poor signal strength due to the down-tilt of the access points (APs) that are optimized to serve ground users ends (GUEs). To solve this, one could tilt the AP antenna upwards or allocate more power to serve the UAV. However, this negatively affects GUE downlink (DL) rates. In this paper, we propose to solve this challenge using a reconfigurable intelligent surface (RIS) to enhance the UAV communication while preserving the 3GPP-prescribed downwards antenna tilt and potentially improving the DL performance of the GUE. We show that under conjugate beamforming (CB) precoding and proper power split between GUEs and the UAV at the APs, an RIS with phase-shifts configured to reflect radio signals towards the UAV can significantly improve the UAV DL throughput while simultaneously benefiting the GUEs. The presented numerical results show that the RIS-aided system can serve a UAV with a required data rate while improving the GUEs DL performance relative to that in a CF-MIMO system without a UAV and an RIS. We support this conclusion through simulations under a varying numbers of RIS reflecting elements, UAV heights, and power split factor.

Reconfigurable Intelligent Surfaces (RIS) are emerging technology to enhance the 6G wireless communication systems by intelligently reconfiguring the propagation environment. This paper propose a Modified Prioritized Deep Deterministic Policy Gradient (MP-DDPG) algorithm for jointly optimizing beamforming at the Base Station (BS) and RIS phases in a Multiple Input Single Output (MISO) downlink communication system. The primary objective is to minimize transmitted powers while adhering to critical constraints such as maximum power limits and Quality of Service (QoS) requirements of the User's Equipment (UE). The simulation results show that the MP-DDPG algorithm offers a robust and adaptive framework for addressing the inherent non-convexity and dynamic nature of such an optimization problem. Key findings highlight the remarkable resilience to imperfect Channel State Information (CSI), and the crucial trade-offs between performance, computational complexity, and signaling overhead.

In this work, we investigate an active reconfigurable intelligent surface (RIS)-aided non-orthogonal multiple access (NOMA)-enabled space-air-ground integrated network (SAGIN) with cognitive radio, leveraging the flexible deployment of an unmanned aerial vehicle (UAV) and the ubiquitous coverage of satellite networks. The UAV serves uplink and downlink users in the secondary network via NOMA and time division multiple access mechanisms, respectively, while satellites provide wireless backhaul for the UAV and primary users. We aim to maximize the weighted sum mean rate and energy efficiency for the secondary network by jointly the optimizing power allocation, the RIS reflection coefficients (RC), the user matching factors, and the UAV trajectory. We propose an alternating optimization framework based on the block coordinate ascent (BCA) technique, which decouples the problem into multiple variable blocks for alternating optimization until convergence. Moreover, we investigate the performance of energy-efficient active RIS with a sub-connected architecture, decoupling the RIS RC optimization into amplification factor and phase shift subproblems to be solved separately. Finally, simulation results validate the effectiveness of the proposed schemes, and demonstrate weakness of passive RIS and rationality and economics of sub-connected active RIS architecture.

The UAV pursuit-evasion strategy based on Deep Deterministic Policy Gradient (DDPG) algorithm is a current research hotspot. However, this algorithm has the defect of low efficiency in sample exploration. To solve this problem, this paper uses the imitation learning (IL) to improve the DDPG exploration strategy. A kind of quasiproportional guidance control law is designed to generate effective learning samples, which are used as the data of the initial experience pool of DDPG algorithm. The UAV pursuit-evasion strategy based on DDPG and imitation learning (IL-DDPG) is proposed, and the algorithm obtains the data from the experience pool for experience playback learning, which improves the exploration efficiency of the algorithm in the initial stage of training and avoids the problem of too many useless exploration in the training process. The simulation results show that the trained pursuit-UAV can flexibly adjust the flight speed and flight attitude to pursuit the evasion-UAV quickly. It also verifies that the improved DDPG algorithm is more effective than the basic DDPG algorithm to improve the training efficiency.