A Hierarchical Framework of Cloud Resource Allocation and Power Management Using Deep Reinforcement Learning

Recently, deep neural networks (DNNs) have been widely studied and achieved tremendous success in a variety of real-world applications, such as computer vision, natural language processing, medical diagnosis, and machine translation. Deep reinforcement learning (DRL), as an emerging powerful deep learning technique, combines DNNs with reinforcement learning into an interactive system that enables the agent to learn the best policy from the environment to optimize their rewards. DRL opens up many new applications in domains such as healthcare, robotics, and smart grids. With the rapid evolution of IT infrastructures, cloud computing has been witnessed as the most common computing paradigm. The underlying infrastructure of cloud computing relies on a large amount of data centers. Nowadays, with the fast development of communication technologies, such as 5G, cloud computing is now evolving into yet another new paradigm called edge computing, providing for emerging applications, especially in the Internet-of-Things. The energy efficiency issue from both "cloud" and "edge" are becoming more crucial and calls for more attention. To solve the real-world energy efficiency problems, we take advantage of the deep learning and deep reinforcement learning techniques for efficient modeling of both "edge" and "cloud" applications. On the "edge" side, wearable technologies and the Internet of Things (IoT) are becoming increasingly popular in both personal and commercial applications with the employment of the embedded system technologies. Non-Volatile Processors (NVPs) have been proposed for the intermittent power supply problem, to ensure the instant on/off for embedded systems under the unstable power supply. We proposed neural network-based prediction techniques to derive the best power extraction policy and converter parameters optimization technique for NVPs. In cloud computing, we formulated a novel optimal power management problem of data centers to minimize the total cost by proposing the long short term memory (LSTM) neural network-based method for the prediction of future data center power consumption, and further we presented a novel DRL-based hierarchical framework for solving the overall resource allocation and power management problem in cloud computing systems. Furthermore, we explored the effectiveness of the DRL-based framework for high dimensional action and state spaces problem-dynamic treatment regimes, to model real-life complexity in heterogeneous disease progression and treatment choices, to provide doctors and patients the data-driven personalized decision recommendations. Last but not least, we explored the efficiency of deep neural networks. Structured weight pruning is a representative model compression technique of DNNs to reduce both the storage and computation requirements and meanwhile accelerate inference. An automatic hyperparameter determination process is necessary due to a large number of flexible hyperparameters. We propose an automatic structured pruning framework to compress the DNNs. Our framework outperforms the prior works on automatic model compression by up to 33× in pruning rate (120× reduction in the actual parameter count) under the same accuracy. Significant inference speedup has been observed from the proposed framework on actual measurements on the smartphone.

Safe deployment of a reinforcement learning robot using self stabilization

Job scheduling is a necessary prerequisite for performance optimization and resource management in the cloud computing system. Focusing on accurate scaled cloud computing environment and efficient job scheduling under Virtual Machine (VM) resource and Server Level Agreement (SLA) constraints, we introduce the architecture of cloud computing platform and optimization job scheduling scheme in this study. The system model is comprised of clearly defined separate constituent parts, including portal, job scheduler, and resources pool. By analyzing the execution process of user jobs, we designed a novel job scheduling scheme based on reinforcement learning to minimize the makespan and Average Waiting Time (AWT) under the VM resource and deadline constraints, and employ parallel multi-age parallel technologies to balance the exploration and exploitation in learning process and accelerate the convergence of Q -learning algorithm. Both simulation and real cloud platform experiment results demonstrate the efficiency of the proposed job scheduling scheme.

Cloud computing allows execution and deployment of different types of applications such as interactive databases or web-based services which require distinctive types of resources. These applications lease cloud resources for a considerably long period and usually occupy various resources to maintain a high quality of service (QoS) factor. On the other hand, general big data batch processing workloads are less QoS-sensitive and require massively parallel cloud resources for short period. Despite the elasticity feature of cloud computing, fine-scale characteristics of cloud-based applications may cause temporal low resource utilization in the cloud computing systems, while process-intensive highly utilized workload suffers from performance issues. Therefore, ability of utilization efficient scheduling of heterogeneous workload is one challenging issue for cloud owners. In this paper, addressing the heterogeneity issue impact on low utilization of cloud computing system, conjunct resource allocation scheme of cloud applications and processing jobs is presented to enhance the cloud utilization. The main idea behind this paper is to apply processing jobs and cloud applications jointly in a preemptive way. However, utilization efficient resource allocation requires exact modeling of workloads. So, first, a novel methodology to model the processing jobs and other cloud applications is proposed. Such jobs are modeled as a collection of parallel and sequential tasks in a Markovian process. This enables us to analyze and calculate the efficient resources required to serve the tasks. The next step makes use of the proposed model to develop a preemptive scheduling algorithm for the processing jobs in order to improve resource utilization and its associated costs in the cloud computing system. Accordingly, a preemption-based resource allocation architecture is proposed to effectively and efficiently utilize the idle reserved resources for the processing jobs in the cloud paradigms. Then, performance metrics such as service time for the processing jobs are investigated. The accuracy of the proposed analytical model and scheduling analysis is verified through simulations and experimental results. The simulation and experimental results also shed light on the achievable QoS level for the preemptively allocated processing jobs.

With the rapid development of Internet of vehicles (IoV), the emerging in-vehicle applications meet people’s demand for intelligent experience, while also increasing the cost of network communication. Driven by the fact that a few popular content accounted for most of the traffic load, caching popular content at the wireless edge has become a trend of content delivery. However, the limitation of cache resources and the ever-changing mobile environment bring a severe challenge to content delivery cost optimization. In this paper, we first investigate the joint optimization of content placement and bandwidth allocation (JOCB) in IoV. In order to minimize the long-term system cost in the mobile scenario, we first establish an innovative vehicle mobility model based on the probability distribution. Then we use the Lagrangian multiplier method to solve the bandwidth allocation sub-problem to reduce the algorithm complexity. Considering the high-dimensional continuity of action and state spaces, the original problem is transformed into a Markov Decision Process (MDP), and we propose a Deep reinforcement learning based Dynamic Content Placement (DDCP) algorithm. DDCP algorithm has low algorithm complexity and can handle high-dimensional continuous state and action space. The simulation results show that compare with other three benchmark algorithms, DDCP algorithm can reduce the system cost effectively.

With the widespread application of cloud computing technology, the energy consumption problem of data centers is becoming increasingly severe. This research is dedicated to presenting a highly efficient and smart solution for optimizing energy usage in cloud computing systems, especially since effectively managing energy consumption through resource allocation and task scheduling has now become a critical challenge in the field. In view of this, the study first constructs a resource allocation model based on redundant resource recovery to dynamically identify and recover redundant resources. Second, a task scheduling optimization method based on dual deep reinforcement learning is established, which utilizes directed acyclic graphs to model task dependencies and optimizes task execution order and resource allocation strategies through reinforcement learning agents. In the testing of the resource allocation optimization model, when the number of tasks is 500, the redundant resource recovery rate is 87.0%, and the computational complexity is 16.7 s. The response time in the regional resource environment is 2.28 s, and the load adaptability is 95.8%. In the task scheduling method test, the proposed method has a memory occupancy of 85 MB and a scheduling delay of 0.13 s in a high-bandwidth environment of 300 Mbps. The results show that the proposed method outperforms traditional scheduling strategies in terms of resource utilization and energy consumption, and contributes to achieving the goal of green cloud computing. The research aims to provide an efficient and intelligent solution for energy optimization in cloud computing systems.

In this paper, we present a new methodology for managing power consumption of networks-on-chips (NOCs). A power management problem is formulated for the first time using closed-loop control concepts. We introduce an estimator and a controller that implement our power management methodology. The estimator is capable of very fast and accurate tracking of changes in the system parameters. Parameters estimated are used to form the system model. Our system model combines node and network centric power management decisions. Node centric power management assumes no a priori knowledge of requests coming in from outside the core. Thus, it implements a more traditional dynamic voltage scaling and power management control algorithms. Network-centric power management utilizes interaction with the other system cores regarding the power and the quality of service (QoS) needs. The overall system model is based on Renewal theory and, thus, guarantees globally optimal results. We introduce a fast optimization method that runs multiple orders of magnitude faster than the previous optimization approaches while still having the same accuracy in obtaining the power management control. Finally, our controller implements the results of optimization in either hardware or software. The new methodology for power management of NOCs is tested on a system consisting of four satellite units, each implementing an estimator and a controller capable of both node and network centric power management. Our results show large savings in power with good QoS.

Though enhancing the capacity and coverage of cellular networks to meet the explosive increasing of traffic demands, Ultra-Dense Network (UDN) suffers from great power consumption and greenhouse gas emission. Turning small base stations (SBS) off dynamically according to the network operating conditions is a promising solution to enhance energy efficiency (EE). Inspired by the success of Deep Q-learning Network(DQN) on solving complicated real-time control problems and the usage of energy-harvesting(EH) in UDNs, without any prior knowledge about energy arrival, data arrival and channel state information, we present a DQN-based framework for dynamic resource allocation in EH-UDN. Specially, joint optimization problem is formulated, considering both EE and quality of service (QoS) of the network. Then according to the application scenario, the action space, state space and reward function of the proposed DQN-based framework are defined and formulated. We evaluate the performance of the proposed framework by comparing it with the classic Q-learning framework via simulation. Numerical results show that our proposed scheme can enhance EE while taking good control of the QoS.

Considering the proliferation in the number of cloud users on an everyday basis, the task of resource provisioning in order to support all these users becomes a challenging problem. When resource allocation is non-optimal, users may face high costs or performance issues. So, in order to maximize profit and resource utilization while satisfying all client requests, it is essential for Cloud Service Providers to come up with ways to allocate resources adaptively for diverse conditions. This is a constrained optimization problem. Each client that submits a request to the cloud has its own best interests in mind. But each of these clients competes with other clients in the quest to obtain required quantum of resources. Hence, every client is a participant in this competition. So, a preliminary analysis of the problem reveals that it can be modelled as a game between clients. A game theoretic modelling of this problem provides us an ability to find an optimal resource allocation by employing game theoretic concepts. Resource allocation problems are NP-Hard, involving VM allocation and migration within and possibly, among data centres. Owing to the dynamic nature and number of requests, static methods fail to surmount race conditions. Using a Min-Max Game approach, we propose an algorithm that can overcome the problems mentioned. We propose to employ a utility maximization approach to solve the resource provisioning and allocation problem. We implement a new factor into the game called the utility factor which considers the time and budget constraints of every user. Resources are provisioned for tasks having the highest utility for the corresponding resource.

Network slicing has emerged as a key technology to support the coexistence of multi-service in the the 5G/B5G networks. However, due to the resource scarcity and the diversity of slice requests, how to allocate resources efficiently to maximize the long-term reward of the infrastructure network is a challenging issue. In this paper, we model the resource allocation problem as a semi-Markov decision process (SMDP), which is defined by state space, action space, reward and transition probability distribution. The reward function jointly considers the total income, the cost of available resource and the utilization of the total resource for the infrastructure network. Not focusing on one step decisions reward, we apply the Bellman equation to obtain long-term reward by accumulating. Then we exploit value iteration algorithm to determine the resource allocation scheme according to a certain state in such a way that the long-term reward can be maximized. Extensive simulation results show that the proposed SMDP can achieve a superior performance compared with the existing heuristic methods.

We consider positive zero-sum stochastic games with countable state and action spaces. For each player, we provide a characterization of those strategies that are optimal in every subgame. These characterizations are used to prove two simplification results. We show that if player 2 has an optimal strategy then he/she also has a stationary optimal strategy, and prove the same for player 1 under the assumption that the state space and player 2's action space are finite.

This paper presents a method of vision-based reinforcement learning by which a robot learns to shoot a ball into a goal. We discuss several issues in applying the reinforcement learning method to a real robot with vision sensor by which the robot can obtain information about the changes in an environment. First, we construct a state space in terms of size, position, and orientation of a ball and a goal in an image, and an action space is designed in terms of the action commands to be sent to the left and right motors of a mobile robot. This causes a “state-action deviation” problem in constructing the state and action spaces that reflect the outputs from physical sensors and actuators, respectively. To deal with this issue, an action set is constructed in a way that one action consists of a series of the same action primitive which is successively executed until the current state changes. Next, to speed up the learning time, a mechanism of Learning from Easy Missions (or LEM) is implemented. LEM reduces the learning time from exponential to almost linear order in the size of the state space. The results of computer simulations and real robot experiments are given.

This work explores on-board planning and scheduling for the multi-target, single spacecraft Earth-observing satellite (EOS) scheduling problem. The problem is formulated as a Markov decision process (MDP) where the number of targets included in the state and action space is an adjustable parameter that may account for clusters of targets with varying priorities. As targets are passed or imaged, they are replaced in the state and action space with the next set of upcoming targets. Unlike prior EOS problem formulations, this work explores how the size of the state and action space can be reduced to produce optimal, generalized policies that may be executed on board the spacecraft in a fraction of a second. Performance of the agents is shown to increase with the number of targets in the state and action space. The number of imaged and downlinked targets stays relatively constant, but the reward increases significantly, demonstrating that the agents are prioritizing high priority targets over low priority targets.

With the advancement of information technology need to perform computing tasks everywhere and all the time there, in cloud computing environments and heterogeneous users have access to different sources with different characteristics that these resources are geographically in different areas. Due to this, the allocation of resources in cloud computing comes to the main issue is considered a major challenge to achieve high performance. Due to the nature of cloud computing is a distributed system to account, comes to business, economic methods such as auctions are used to allocate resources for decentralization. As an important economic bilateral hybrid auction model is the perfect solution for the allocation of resources in cloud computing, on the other hand, providers of cloud resources similarly, their sources of supply combined addressing. One of the problems auction two-way combination with maximum benefit for the parties to the transaction is the efficient allocation of resources to the problem of determining an auction winner is known. Given that the winning auction is NP-Hard. It results in a problem, several methods have been proposed to solve it. In this dissertation, taking into account the strength simulated annealing algorithm, a modified version of it is proposed for solving the winner determination in combinatorial double auction problem in cloud computing. The proposed approach is simulated along with genetic and simulated annealing algorithms and the results show that the proposed approach finds better solutions than the two mentioned algorithms.

A novel system power management technique is proposed that employs a novel cost function based on state‐action. Compared with the conventional algorithm, by using multiple parameter constraints in cost function of power management framework, the improved Q‐learning can effectively make decisions to achieve a rational optimisation room. The proposed power management framework does not need any prior data and is running on a power model. As uncertainties can be effectively captured and modelled, the framework based on the model can help to explore an ideal trade‐off and converge to the best power management policy. The results obtained showed that improved algorithm achieved remarkable significance.