DCN-LB: A Deep Learning-Driven Cloud Load ‎BalancingFramework for Efficient Resource ‎Optimization

A balanced and reliable data replica placement scheme based on reinforcement learning in edge–cloud environments

Optimum efficiency and fault tolerance are the most demanding and challenging issues in the domain of performance and reliability management in cloud computing environments. Optimized resource utilization is a key aspect for yielding efficiency in cloud platforms. Workload balancing through resource sharing is one of the key solutions for attaining performance in cloud environments. In addition, multiple cloud environments join hands to offer performance and fault tolerance through resource sharing. We provide a better and cloud-instances’ priority-based efficient load balancing method for collaborative cloud platforms. The recommended efficient load balancing method shortens the waiting timespan and overcomes the starvation problem of low priority instances in intercloud environments. A functional prototype of the recommended load balancing method was deployed on a physical cloud infrastructure which was setup with the OpenStack cloud software on the Fedora Linux operating system. The pilot project execution findings exhibit a reduction in the timespan borne by instances for executing load balancing. This technique is useful for attaining fault tolerance and efficient resource utilization in intracloud and intercloud environments.

Performance efficiency is important in mobile application development because mobile devices have limitations in terms of power and resources. Performance efficiency can be improved by applying architecture patterns. In this paper, we use the Model View ViewModel (MVVM) architecture. The application of the architecture is carried out to analyze how practical the application of the MVVM architecture pattern is in increasing performance efficiency in the mobile application. Performance efficiency is measured based on CPU usage, memory usage, and execution Time. The case study shows that the CPU usage and execution Time on MVVM are smaller than Base architecture pattern from the AR Ruler. This is due to the third-party library RxSwift in the MVVM architecture that increases the application's response so that CPU usage and execution time is better than Base architecture pattern. However, the existence of the third-party library RxSwift has a negative impact on memory usage, resulting in higher memory usage than the Base Architecture Pattern. The MVVM pattern is highly recommended for mobile application development to improve performance efficiency.

A deep reinforcement learning based algorithm for a distributed precast concrete production scheduling

Adaptive traffic signal control (ATSC) based on deep reinforcement learning (DRL) has shown promising prospects to reduce traffic congestion. Most existing methods keeping traffic signal phases fixed adopt two agent actions to match a four-phase suffering unstable performance and undesirable operation in a four-phase signalized intersection. In this paper, a Double Deep Q-Network (DDQN) with a dual-agent algorithm is proposed to obtain a stable traffic signal control policy. Specifically, two agents are denoted by two different states and shift the control of green lights to make the phase sequence fixed and control process stable. State representations and reward functions are presented by improving the observability and reducing the leaning difficulty of two agents. To enhance the feasibility and reliability of two agents in the traffic control of the four-phase signalized intersection, a network structure incorporating DDQN is proposed to map states to rewards. Experiments under Simulation of Urban Mobility (SUMO) are carried out, and results show that the proposed traffic signal control algorithm is effective in improving traffic capacity.

Intelligent energy management for an on-grid hydrogen refueling station based on dueling double deep Q network algorithm with NoisyNet

Cyber-physical systems (CPSs) in autonomous vehicles must handle highly dynamic and uncertain settings, where unanticipated impediments, shifting traffic conditions, and environmental changes all provide substantial decision-making issues. Deep reinforcement learning (DRL) has emerged as a strong tool for dealing with such uncertainty, yet current DRL models struggle to ensure safety and optimal behaviour in indeterminate settings due to the difficulties of understanding dynamic reward systems. To address these constraints, this study incorporates double deep Q networks (DDQN) to improve the agent's adaptability under uncertain driving conditions. A structured reward system is established to accommodate real-time fluctuations, resulting in safer and more efficient decision-making. The study acknowledges the technological limitations of automobile CPSs and investigates hardware acceleration as a potential remedy in addition to algorithmic enhancements. Because of their post-manufacturing adaptability, parallel processing capabilities, and reconfigurability, field programmable gate arrays (FPGAs) are used to execute reinforcement learning in real-time. Using essential parameters, including collision rate, behaviour similarity, travel distance, speed control, total rewards, and timesteps, the suggested method is thoroughly tested in the TORCS Racing Simulator. The findings show that combining FPGA-based hardware acceleration with DDQN successfully improves computational efficiency and decision-making reliability, tackling significant issues brought on by uncertainty in autonomous driving CPSs. In addition to advancing reinforcement learning applications in CPSs, this work opens up possibilities for future investigations into real-world generalisation, adaptive reward mechanisms, and scalable hardware implementations to further reduce uncertainty in autonomous systems.

In order to better understand state-of-the-art reinforcement learning agent, deep Q-network, a brief review of reinforcement learning and Q-learning are first described. Then recent advances of deep Q-network are presented, and double deep Q-network and dueling deep Q-network that go beyond deep Q-network are also given.

On GPU-based clusters, the training workloads of machine learning (ML) models, particularly neural networks (NNs), are often structured as Directed Acyclic Graphs (DAGs) and typically deployed for parallel execution across heterogeneous GPU resources. Efficient scheduling of these workloads is crucial for optimizing performance metrics such as execution time, under various constraints including GPU heterogeneity, network capacity, and data dependencies. DAG-structured ML workload scheduling could be modeled as a Nonlinear Integer Program (NIP) problem, and is shown to be NP-complete. By leveraging a positive correlation between Scheduling Plan Distance (SPD) and Finish Time Gap (FTG) identified through an empirical study, we propose to develop a Running Time Gap Strategy for scheduling based on Whale Optimization Algorithm (WOA) and Reinforcement Learning, referred to as WORL-RTGS. The proposed method integrates the global search capabilities of WOA with the adaptive decision-making of Double Deep Q-Networks (DDQN). Particularly, we derive a novel function to generate effective scheduling plans using DDQN, enhancing adaptability to complex DAG structures. Comprehensive evaluations on practical ML workload traces collected from Alibaba on simulated GPU-enabled platforms demonstrate that WORL-RTGS significantly improves WOA’s stability for DAG-structured ML workload scheduling and reduces completion time by up to 66.56% compared with five state-of-the-art scheduling algorithms.

Reinforcement Learning (RL) has made several advances in the machine learning domain especially Deep Reinforcement Learning. AlphaGo developed by DeepMind is a good example of how the deep neural network can train an agent to play and outperform professional Go players. Auto optimizing parameter is a relatively challenging research area. To automatically tune the parameters of the system, we formalize the problem into the setting of reinforcement learning model, where the aim is to train an agent whose goal is to find the optimum parameters by observing the state of the system and taking actions to maximize a cumulative reward. In this work, we solve this problem with the matrix version of Q-learning and the neural network version of Q-learning. We developed a new heuristic of Q-learning algorithm, called Dynamic Action Space (DAS), to further improve the robustness of the algorithm in finding the optimum state. The DAS approach significantly improved the convergence and stability of the algorithm. Then we tested the approach on three deep neural network variants, namely Deep Q-Networks (DQN), Double Deep Q-Networks (DDQN) and Dueling Networks. We show that the heuristic DAS model helps the Deep RL networks to converge better than the baseline Q-Learning model.

In this paper, we propose a federated deep reinforcement learning framework to solve a multi-objective optimization problem, where we consider minimizing the expected long-term task completion delay and energy consumption of IoT devices. This is done by optimizing offloading decisions, computation resource allocation, and transmit power allocation. Since the formulated problem is a mixed-integer non-linear programming (MINLP), we first cast our problem as a multi-agent distributed deep reinforcement learning (DRL) problem and address it using double deep Q-network (DDQN), where the actions are offloading decisions. The immediate cost of each agent is calculated through solving either the transmit power optimization or local computation resource optimization, based on the selected offloading decisions (actions). Then, to enhance the learning speed of IoT devices (agents), we incorporate federated learning (FDL) at the end of each episode. FDL enhances the scalability of the proposed DRL framework, creates a context for cooperation between agents, and minimizes their privacy concerns. Our numerical results demonstrate the efficacy of our proposed federated DDQN framework in terms of learning speed compared to federated deep Q network (DQN) and non-federated DDQN algorithms. In addition, we investigate the impact of batch size, network layers, DDQN target network update frequency on the learning speed of the FDL.

Unmanned Aerial Vehicle (UAV) has emerged as a promising technology to provide wireless signals from air to the ground users in specific scenarios such as earthquakes, tsunamis and other disasters. The performance of the UAV is degraded when the signals are blocked by obstacles in dense urban scenarios. To address this issue and enhance the signal quality available to the ground users, Reconfigurable Intelligent Surface (RIS) has emerged as a new technological paradigm. It offers an intelligent configuration for the signal propagation environment by redirecting the signals to the users. In this article, we solve a non-convex optimization problem of RIS-assisted UAV network by jointly optimizing the RIS phase shift and 3D trajectory of UAV to maximize the energy efficiency of a rotatory-wing UAV. The considered optimization problem is solved using Deep Reinforcement Learning (DRL) based techniques in an on-line fashion to reduce the computational complexity. We leverage Twin-delayed Deep Deterministic Policy Gradient (TD3) to solve the problem by considering the UAV trajectory as a set of continuous actions. For comparison, we also use the Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG) and Double Deep Q-Network (DDQN) for continuous and discrete optimization of the UAV trajectory, respectively. Extensive simulations show that the TD3 outperforms all the considered DRL techniques with the highest energy efficiency and throughput, and the lowest propulsion energy.

Non-orthogonal multiple access (NOMA) provides higher spectral efficiency and access to more users than orthogonal multiple access. However, the issue of resource allocation in NOMA is dynamic and produces a high computation burden when using traditional methods. In this paper, a symmetry-aware double deep Q network (DDQN) algorithm in deep reinforcement learning is employed to allocate power to users in NOMA while guaranteeing quality of service for the weakest users. The research process is divided into two parts. Firstly, users in the communication system are grouped using a method that synergistically considers gain difference and similarity, exploiting symmetrical properties within the user groups. Secondly, the DDQN algorithm is used to allocate power to multiple users in a NOMA system, which utilizes the inherent symmetry in the signal-to-interference noise ratio of each user as an objective function. By recognizing and leveraging these symmetrical patterns, the algorithm can dynamically adjust the power allocation to optimize system performance. Finally, the proposed algorithm is compared with conventional NOMA power allocation algorithms and shows significant improvements in system performance. The results of the convergence function show that the algorithm proposed in this paper can converge in approximately 1800 iterations, which effectively solves the problem of large arithmetic and complex processes existing in the traditional method.

Energy efficient task scheduling based on deep reinforcement learning in cloud environment: A specialized review

The rapid emergence of the Internet of Things (IoT) has given rise to immense volumes of data, which require proper processing, storage, and analysis. Cloud computing can scale to meet the needs of the Internet of Things (IoT), but latency, bandwidth usage, and inefficient resource utilization are bottlenecks affecting performance. The capacity of deep learning (DL) to build and optimize connections amidst assets and the capacity to describe complicated associations is turning into a groundbreaking method of advancing the capability of IoT in the cloud-based setting. The article provides a comprehensive summary of the deep learning-based strategies to maximize the functionality of the IoT. It also talks about how it can be used in smart task scheduling and smart energy management, anomaly discovery, smart resource provisioning, and smart latency reduction. Deep neural networks based on convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, and reinforcement learning are considered as architectures in the framework of IoT-cloud integration. It is demonstrated that DL can significantly increase throughput, reliability, and responsiveness and reduce costs and energy usage. Nevertheless, pressing issues include high computational costs, the interpretability of deep learning systems, data confidentiality, and counterexamples. To offer solutions to them, in addition to federated learning, edge-cloud interaction, and explainable AI, it is proposed to combine them in the future. This paper concludes that IoT systems optimized through the use of deep learning-based cloud-IoT frameworks can be considered as a promising trend that can ensure scalability, resilience, and effectiveness in the next-generation smart environments.