Multi-Agent DDPG Based Resource Allocation in NOMA-Enabled Satellite IoT

Non-orthogonal multiple access (NOMA) is a key technology for future wireless networks, enabling improved spectral efficiency and massive device connectivity. However, optimizing power allocation, subchannel assignment, and cell association in multi-cell uplink NOMA systems is challenging due to user mobility and the NP-hard nature of the problem. This paper addresses these challenges by formulating the problem as a mixed-integer non-linear programming (MINLP) model to maximize energy efficiency (EE). We propose a deep reinforcement learning framework that employs deep Q-networks (DQN) for cell association and subchannel assignment, and twin delayed deep deterministic policy gradient (TD3) for power allocation. Simulation results reveal significant EE improvements, with multi-agent TD3 (MATD3) outperforming traditional Lagrange methods and multi-agent deep deterministic policy gradient (MADDPG). Furthermore, the proposed method exhibits robust adaptability to user mobility and superior performance in multi-cell environments, effectively mitigating inter-cell interference and enhancing resource allocation in dynamic scenarios.

In the context of the Internet of Things (IoT), aerial computing platforms (ACPs) such as unmanned aerial vehicles and high-altitude platforms with edge computing capabilities have the potential to significantly expand coverage, enhance performance, and handle complex computational tasks for IoT devices (IoTDs). Non-orthogonal multiple access (NOMA) has also emerged as a promising multiple access technology for advanced 5G networks. This paper presents a multi-ACP-enabled NOMA edge network, which enables heterogeneous ACPs to provide computational assistance to IoTDs. To minimize delay and energy consumption, we formulate a joint task offloading and resource allocation problem that considers IoTD association, offloading ratio, transmit power, and computational resource allocation variables. To address the complexity of the optimization problem, it is modeled as a multi-agent Markov decision process and solved using a multi-agent deep deterministic policy gradient (MADDPG)-based solution. Extensive simulation results demonstrate that the proposed MADDPG-based framework can remarkably adapt to the dynamic nature of multi-ACP-enabled NOMA edge networks. It consistently outperforms various benchmark schemes regarding energy efficiency and task processing delay across different simulated scenarios.

This paper investigates the optimal resource allocation of a downlink non-orthogonal multiple access (NOMA) system consisting of one base station and multiple users. Unlike existing short-term NOMA designs that focused on the resource allocation for only the current transmission timeslot, we aim to maximize a long-term network utility by jointly optimizing the data rate control at the network layer and the power allocation among multiple users at the physical layer, subject to practical constraints on both the short-term and long-term power consumptions. To solve this problem, we leverage the recently-developed Lyapunov optimization framework to convert the original long-term optimization problem into a series of online rate control and power allocation problems in each timeslot. The power allocation problem, however, is shown to be non-convex in nature and thus cannot be solved with a standard method. However, we explore two structures of the optimal solution and develop a dynamic programming based power allocation algorithm, which can derive a globally optimal solution, with a polynomial computational complexity. Extensive simulation results are provided to evaluate the performance of the proposed joint rate control and power allocation framework for NOMA systems, which demonstrate that the proposed NOMA design can significantly outperform multiple benchmark schemes, including orthogonal multiple access (OMA) schemes with optimal power allocation and NOMA schemes with non-optimal power allocation, in terms of average throughput and data delay.

Non-orthogonal multiple access (NOMA) and reconfigurable intelligent surfaces (RISs) are recognized as key technologies for beyond 5G and 6G wireless communications. To address the high computational complexity and non-convex optimization challenges, this letter proposes an optimization framework based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. The proposed framework jointly makes use of sensor grouping, power allocation, an RIS computation strategy, and phase shifts to minimize the remote state estimation (RSE) error. Simulation results demonstrate that the MADDPG algorithm, when applied in an RIS-assisted NOMA system, significantly reduces the RSE error.

In the fifth generation (5G) wireless communication network, to better offer service for different vertical industry applications, network slicing is needed to meet the differential requirements. In this paper, we study the joint problem of radio resource allocation and power resource allocation in the radio access network (RAN) to improve the radio resource utilization with multi-user demands in smart grid scenario. Different with conventional single-agent algorithm, we propose a non-orthogonal multiple access (NOMA) based multi-agent deep deterministic policy gradient (MADDPG) algorithm to solve this dynamic optimization problem. The scheme determines the remote radio unit (RRU) and physical resource block (PRB) allocation for each user, and updates the neural network to dynamically adjust the resource allocation strategy to maximize the long-term rate of network slices. The reliability and latency requirements of network slicing are guaranteed as well. Simulation results show the convergence and efficiency of the proposed algorithm.

While existing works about non-orthogonal multiple access (NOMA) have indicated that NOMA can yield a significant performance gain over orthogonal multiple access (OMA) with fixed resource allocation, it is not clear whether such a performance gain will diminish when optimal resource (Time/Frequency/Power) allocation is carried out. In this paper, the performance comparison between NOMA and conventional OMA systems is investigated, from an optimization point of view. Firstly, by using the idea of power splitting, a closed-form expression for the optimum sum rate of NOMA systems is derived. Then, with rigorous mathematical proofs, we reveal the fact that NOMA can always outperform conventional OMA systems, even if both are equipped with the optimal resource allocation policies. Finally, computer simulations are conducted to validate the accuracy of the analytical results.

In the future mobile edge networks, the Internet of things (IoT) applications will be latency-sensitive and computationally intensive. Given the resource limitation of IoT devices, mobile edge computing (MEC) servers are critical to support the efficient processing of IoT tasks. Since MEC servers attached to the ground base stations are generally deployed in fixed locations and vulnerable to physical damage, the unmanned aerial vehicle (UAV) and satellite-assisted MEC framework has been proposed to leverage the flexibility of UAVs and the broad coverage of satellites. However, efficient utilization of the UAV/satellite resources is challenging for the static ground IoT devices because of the dynamic in terms of aerial and space network topology and IoT task arrival rates. To adapt to the changing environment and utilize the interaction among multiple UAVs, we propose a multi-agent deep deterministic policy gradient (MADDPG) framework to jointly optimize the traveling routes of multi-UAVs and the offloading decision of IoT devices. To minimize the processing cost in terms of task processing latency and energy consumption of IoT devices, cooperative UAVs can help find the optimal task offloading location for each IoT device. Simulation results show the proposed algorithm based on MADDPG can averagely decrease 20% of the above processing cost compared with the benchmark approach.

Recently, non-orthogonal multiple access (NOMA) has been proposed to improve spectral efficiency greatly over conventional orthogonal multiple access (OMA). However, it is still challenging to implement NOMA into transmission of video. In this research, we investigate video broadcasting design over NOMA systems. When video transmission is considered, quality of experience (QoE) is a better metric for NOMA design than throughput. Therefore, we design a QoE-driven scalable video broadcasting (SVB) scheme over NOMA systems. To achieve scalable transmission performance, power allocation is optimized among multiple video layers, which are requested by users with different channel conditions. Specifically, we formulate the power allocation problem to maxmize the system overall QoE while guaranteeing the basic service of users. Although this problem is non-convex and discontinuous, an optimal algorithm is proposed based on the hidden monotonicity of the problem. Simulation results show that the proposed SVB scheme over NOMA systems outperforms existing OMA and NOMA based schemes in various broadcast scenarios in terms of QoE.

In this paper, we investigate the throughput performance of single-packet and multi-packet hybrid-automatic repeat request (HARQ) with blanking for downlink non-orthogonal multiple access (NOMA) systems. While conventional single-packet HARQ achieves high throughput at the expense of high latency, multi-packet HARQ, where several data packets are sent in the same channel block, can achieve high throughput with low latency. Previous works have shown that multi-packet HARQ outperforms single-packet HARQ in orthogonal multiple access (OMA) systems, especially in the moderate to high signal-to-noise ratio regime. This work amalgamates multi-packet HARQ with NOMA to achieve higher throughput than the conventional single-packet HARQ and OMA, which has been adopted in the legacy mobile networks. We conduct theoretical analysis for the throughput per user and also investigate the optimization of the power and rate allocations of the packets, in order to maximize the weighted-sum throughput. It is demonstrated that the gain of multi-packet HARQ over the single-packet HARQ in NOMA systems is reduced compared to that obtained in OMA systems due to inter-user interference. It is also shown that NOMA-HARQ cannot achieve any throughput gain with respect to OMA-HARQ when the error propagation rate of the NOMA detector is above a certain threshold.

In this paper, we present the joint optimization of sum rate and fairness for contention based uplink multiple access with non-orthogonal multiple access (NOMA) communication system by resource allocation and user grouping. In particular, we study the cases of many users sharing the same resources that address application of the the internet of things (IoT). The key feature of contention based multiple access is to serve multiple users at the same time and frequency. With different power levels and user grouping, it can achieve better spectral efficiency over conventional orthogonal multiple access (OMA). However, unlike the OMA system, NOMA results in additional inter-user interference (IUI). It has also been shown that, without proper resource allocation for users in the uplink NOMA, the weak users can always be in outage. In this work, we have developed algorithms on subbands assignment, user grouping, and power allocation for joint optimization of sum rate and fairness. The algorithm allocates resources iteratively to handle the IUI in each iteration. Given a number of N <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">s</sub> subbands allocation to each user, we could prevent starvation of poor users, e.g. cell edge users. We have also compare and analyze the sum rate and fairness performance with different combination of L and N <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">s</sub> . We also find that, by properly limiting the maximum number of subbands each user can use, the system could better exploit multi-user diversity to improve the sum rate and hence the energy efficiency. The numerical simulations are also conducted to verify the results.

In this letter, we explore user pairing in a downlink non-orthogonal multiple access (NOMA) network. As power allocation inherently intertwines with user pairing, a joint user pairing and power allocation problem is considered to optimize the achievable sum rate (ASR) with minimum rate constraint for each user, which is a mixed integer programming problem. To solve this non-convex problem, we first obtain the optimal power allocation in an NOMA system with only 2 users; then analyze the user pairing problem in a simplified situation, i.e., an NOMA system with four users. Finally, we obtain the closed-form globally optimal solution in a general NOMA system. Extensive performance evaluations are conducted to compare the ASRs of the NOMA and OMA systems. Results show that the performance of the NOMA system with the proposed optimal user pairing is significantly better than that of the OMA system, as well as the performance of the NOMA system with random user pairing.

The combination of energy harvesting (EH), cognitive radio (CR), and non-orthogonal multiple access (NOMA) is a promising solution to improve energy efficiency and spectral efficiency of the upcoming beyond fifth generation network (B5G), especially for support the wireless sensor communications in Internet of things (IoT) system. However, how to realize intelligent frequency, time, and energy resource allocation to support better performances is an important problem to be solved. In this paper, we study joint spectrum, energy, and time resource management for the EH-CR-NOMA IoT systems. Our goal is to minimize the number of data packets losses for all secondary sensing users (SSU), while satisfying the constraints on the maximum charging battery capacity, maximum transmitting power, maximum buffer capacity, and minimum data rate of primary users (PU) and SSUs. Due to the non-convexity of this optimization problem and the stochastic nature of the wireless environment, we propose a distributed multidimensional resource management algorithm based on deep reinforcement learning (DRL). Considering the continuity of the resources to be managed, the deep deterministic policy gradient (DDPG) algorithm is adopted, based on which each agent (SSU) can manage its own multidimensional resources without collaboration. In addition, a simplified but practical action adjuster (AA) is introduced for improving the training efficiency and battery performance protection. The provided results show that the convergence speed of the proposed algorithm is about 4 times faster than that of DDPG, and the average number of packet losses (ANPL) is about 8 times lower than that of the greedy algorithm.

This paper investigates the problem of age of information (AoI) aware radio resource management for a platooning system. Multiple autonomous platoons exploit the cellular wireless vehicle-to-everything (C-V2X) communication technology to disseminate the cooperative awareness messages (CAMs) to their followers while ensuring timely delivery of safety-critical messages to the Road-Side Unit (RSU). To lower the computational load at the RSU and cope with the challenges of dynamic channel conditions, we exploit a distributed resource allocation framework based on multi-agent reinforcement learning (MARL), where each platoon leader (PL) acts as an agent and interacts with the environment to learn its optimal policy. Motivated by the existing literature in RL, we propose two novel MARL frameworks based on the multi-agent deep deterministic policy gradient (MADDPG), named Modified MADDPG, and Modified MADDPG with task decomposition. Both algorithms train two critics with the following goals: A global critic which estimates the global expected reward and motivates the agents toward a cooperating behavior and an exclusive local critic for each agent that estimates the local individual reward. Furthermore, based on the tasks each agent has to accomplish, in the second algorithm, the holistic individual reward of each agent is decomposed into multiple sub-reward functions where task-wise value functions are learned separately. Numerical results indicate our proposed algorithms' effectiveness compared with other contemporary RL frameworks, e.g., federated reinforcement learning (FRL) in terms of AoI performance and CAM message transmission probability.

With the development of the IoT (Internet of Things), sensors networks can bring a large amount of valuable data. In addition to be utilized in the local IoT applications, the data can also be traded in the connected edge servers. As an efficient resource allocation mechanism, the double auction has been widely used in the stock and futures markets and can be also applied in the data resource allocation in sensor networks. Currently, there usually exist multiple edge servers running double auctions competing with each other to attract data users (buyers) and producers (sellers). Therefore, the double auction market run on each edge server needs efficient mechanism to improve the allocation efficiency. Specifically, the pricing strategy of the double auction plays an important role on affecting traders’ profit, and thus, will affect the traders’ market choices and bidding strategies, which in turn affect the competition result of double auction markets. In addition, the traders’ trading strategies will also affect the market’s pricing strategy. Therefore, we need to analyze the double auction markets’ pricing strategy and traders’ trading strategies. Specifically, we use a deep reinforcement learning algorithm combined with mean field theory to solve this problem with a huge state and action space. For trading strategies, we use the Independent Parametrized Deep Q‐Network (I‐PDQN) algorithm combined with mean field theory to compute the Nash equilibrium strategies. We then compare it with the fictitious play (FP) algorithm. The experimental results show that the computation speed of I‐PDQN algorithm is significantly faster than that of FP algorithm. For pricing strategies, the double auction markets will dynamically adjust the pricing strategy according to traders’ trading strategies. This is a sequential decision‐making process involving multiple agents. Therefore, we model it as a Markov game. We adopt Multiagent Deep Deterministic Policy Gradient (MADDPG) algorithm to analyze the Nash equilibrium pricing strategies. The experimental results show that the MADDPG algorithm solves the problem faster than the FP algorithm.

Federated deep reinforcement learning for task offloading and resource allocation in mobile edge computing-assisted vehicular networks