Using reinforcement learning algorithms to dynamically allocate computing resources in cloud environments

Cloud computing technology is getting more advanced these days. Cloud computing is a pay-as-you-go service since it allows for variable resource allocation for services. Task scheduling and resource allocation are becoming major requirements for efficient and effective load balancing among cloud resources. For scheduling resources in cloud environment, different tasks need to be executed simultaneously by the available resources in order to minimize the completion time. To achieve this, we proposed a new technique called Enhanced Max-Min Algorithm (EMM). This algorithm takes less time to complete all the tasks with minimum completion time. In the proposed algorithm, small size tasks are concurrently executed and resource allocation is done accordingly. When compared to Min-Min algorithm the Enhanced Max-Min algorithm provides high resource utilization and reduces the makespan.

. ., , . . , .,

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

The article investigates the efficiency of DevOps processes by integrating artificial intelligence and machine learning technologies. Traditional DevOps approaches face limited monitoring, the complexity of managing CI/CD processes, a significant amount of logs and incidents, and inefficient scaling of resources in cloud environments. Attention is focused on implementing intelligent methods for analyzing logs, predicting failures, automating software deployment, and optimising load balancing. Particular attention is paid to deep learning, natural language processing, and time series forecasting algorithms to improve anomaly detection and provide more accurate load forecasting. Mechanisms for optimising CI/CD processes using reinforcement learning, which allows for automatically adapting pipeline settings to changing operating conditions, are also discussed. Using clustering and pattern recognition methods in the code allows the detection of potential errors even before the testing stage, significantly reducing the risk of failure when deploying new versions. The introduction of intelligent monitoring systems allows for the prompt detection of deviations from normal operation, which facilitates proactive problem solving and minimises human intervention at critical moments. The study also considers prospects for further development, including developing integrated platforms that combine different AI models for monitoring, resource management, and deployment automation. In addition, the article analyses the application of generative AI models for coding automation, including the creation of optimized test scenarios and automatic correction of errors in the code. Particular attention is paid to the possibility of creating autonomous DevOps agents that can independently manage the software life cycle, analyse the effectiveness of deployments, and make decisions on rolling back changes or further optimising resources.

Remote geovisualization has gained momentum to support large‐scale geospatial data analysis and complex decision‐making over the last few years. Cloud computing, due to its capabilities to deliver on‐demand computing resources, has been embraced to develop and deploy interactive and scalable remote geovisualization applications. However, current cloud computing frameworks do not offer a versatile resource management scheme that is readily applicable for online remote visualization services, which usually require maintaining a satisfactory service level over time under dynamic workloads. To address this gap, we propose an automatic cloud resource management approach based on a bi‐level scheduling and horizontal scaling scheme to exploit cloud resources efficiently. At the lower level, a dynamic task‐scheduling scheme using collaborative filtering techniques is proposed to allocate virtual cloud resources to execute sub‐tasks. The scheduling scheme considers spatio‐temporal patterns presented in visualization views. At the upper level, reinforcement learning is adopted to perform resource auto‐scaling based on a reward function that integrates three different facets, namely: time cost, resource cost, and service stability. The original reinforcement learning algorithm is improved in two main aspects: (1) considering the delay of resource provisioning that is common in cloud environments; and (2) using online Gaussian estimation to estimate Q values. Task scheduling and auto‐scaling interact with each other and are integrated to deliver a comprehensive and responsive resource management solution. Experimental results demonstrate that our approach outperforms several existing cloud resource management methods. The proposed approach is also applicable for other interactive visualization applications, which have similar workload characteristics and performance requirements as interactive remote geovisualization.

P2P power trading based on reinforcement learning for nanogrid clusters

A family of heuristics for agent-based elastic Cloud bag-of-tasks concurrent scheduling

Dynamic graph reinforcement learning algorithm for portfolio management: A novel time–frequency correlated model

This paper proposes a reinforcement learning (RL) algorithm for the security problem of state estimation of cyber-physical system (CPS) under denial-of-service (DoS) attacks. The security of CPS will inevitably decline when faced with malicious cyber attacks. In order to analyze the impact of cyber attacks on CPS performance, a Kalman filter, as an adaptive state estimation technology, is combined with an RL method to evaluate the issue of system security, where estimation performance is adopted as an evaluation criterion. Then, the transition of estimation error covariance under a DoS attack is described as a Markov decision process, and the RL algorithm could be applied to resolve the optimal countermeasures. Meanwhile, the interactive combat between defender and attacker could be regarded as a two-player zero-sum game, where the Nash equilibrium policy exists but needs to be solved. Considering the energy constraints, the action selection of both sides will be restricted by setting certain cost functions. The proposed RL approach is designed from three different perspectives, including the defender, the attacker and the interactive game of two opposite sides. In addition, the framework of Q-learning and state–action–reward–state–action (SARSA) methods are investigated separately in this paper to analyze the influence of different RL algorithms. The results show that both algorithms obtain the corresponding optimal policy and the Nash equilibrium policy of the zero-sum interactive game. Through comparative analysis of two algorithms, it is verified that the differences between Q-Learning and SARSA could be applied effectively into the secure state estimation in CPS.

The introduction of network function virtualization (NFV) leads to a new business model in which the Telecommunication Service Provider needs to rent cloud resources to infrastructure provider (InP) at prices as low as possible. Lowest prices can be achieved if the cloud resources can be rented in advance by allocating long-term virtual machines (VM). This is in contrast with the short-term VMs that are rented on demand and have higher costs. For this reason, we propose a proactive solution in which the cloud resource rent is planned in advance based on peak traffic knowledge. We illustrate the problem of determining the cloud resources in cloud infrastructures managed by different InPs and so as to minimize the sum of cloud resource, bandwidth and deployment costs. We formulate an integer linear problem (ILP) and due to its complexity, we introduce an efficient heuristic approach allowing for a remarkable computational complexity reduction. We compare our solution to a reactive solution in which the cloud resources are rented on demand and dimensioned according to the current traffic. Though the proposed proactive solution needs more cloud and bandwidth resources due to its peak allocation, its total resources cost may be lower than the one achieved when a reactive solution is applied. That is a consequence of the higher cost of short-term VMs. For instance, when a reactive solution is applied with traffic variation times of ten minutes, our proactive solution allows for lower total costs when the long-term VM rent is lower than the short-term VM one by 33%.

This paper investigates the error accumulation problem in centralized training and decentralized execution (CTDE) policy‐based multi‐agent reinforcement learning (MARL) algorithms, which arises from local observation inaccuracies. To address this issue, we propose a novel MARL algorithm that incorporates imitation learning using local observations. Firstly, by analysing the multi‐agent proximal policy optimization algorithm and examining the problems arising when global states are replaced with local observations, it is proved that insufficient observations can lead to information loss, thereby introducing errors of advantage function, and it is demonstrated that the generalized advantage estimation method accumulates errors during the training process. Then, imitation learning is introduced and a novel training framework that combines reinforcement learning and imitation learning is proposed. During the reinforcement learning phase, an MARL agent trained with global observations acts as an expert. Subsequently, imitation learning is applied to train another agent that mimics the expert's decisions using only local observations. Finally, the effectiveness of this algorithm is verified in some commonly used multi‐agent environments, which demonstrates its superior performance compared to traditional multi‐agent reinforcement learning algorithms.

Recent advancement on reinforcement learning (RL) algorithms shows that effective learning of parametric action- selection policies can often be achieved through direct opti- mization of a performance lower bound subject to pre-defined policy behavioral constraints. Driven by this understanding, this paper seeks to develop new policy search techniques where RL is achieved through maximizing a performance lower bound ob- tained originally based on an Expectation-Maximization method. For reliable RL, our new learning techniques must also simul- taneously guarantee constrained policy behavioral changes mea- sured through KL divergence. Two separate approaches will be pursued to tackle our constrained policy optimization problems, resulting in two new RL algorithms. The first algorithm utilizes a conjugate gradient technique and a Bayesian learning method for approximate optimization. The second algorithm focuses on minimizing a loss function derived from solving the Lagrangian for constrained policy search. Both algorithms have been experi- mentally examined on several benchmark problems provided by OpenAI GYM. The experiment results clearly demonstrate that our algorithms can be highly effective in comparison to several well-known RL algorithms.

Background: In recent years, the computational memory and energy conservation have become a major problem in cloud computing environment due to the increase in data size and computing resources. Since, most of the different cloud providers offer different cloud services and resources use limited number of user’s applications. Objective: The main objective of this work is to design and implement a cloud resource allocation and resources scheduling model in the cloud environment. Methods: In the proposed model, a novel cloud server to resource management technique is proposed on real-time cloud environment to minimize the cost and time. In this model different types of cloud resources and its services are scheduled using multi-level objective constraint programming. Proposed cloud server-based resource allocation model is based on optimization functions to minimize the resource allocation time and cost. Results: Experimental results proved that the proposed model has high computational resource allocation time and cost compared to the existing resource allocation models. Conclusion: This cloud service and resource optimization model is efficiently implemented and tested in real-time cloud instances with different types of services and resource sets.

Given its current development trajectory, the complexity of cloud computing ecosystems are evolving to where traditional resource management strategies will struggle to remain fit for purpose. These strategies have to cope with ever-increasing numbers of heterogeneous resources, a proliferation of new services, and a growing user-base with diverse and specialized requirements. This growth not only significantly increases the number of parameters needed to make good decisions, it increases the time needed to take these decisions. Consequently, traditional resource management systems are increasingly prone to poor decisions making. Devolving resources management decisions to the local environment of that resource can dramatically increase the speed of decisions making; moreover, the cost of gathering global information can thus be eliminated; saving communication costs. Experimental data, provided in this paper, illustrate that extant cloud deployments can be used as effective vehicles for devolved decision making. This finding strengthens the case for the proposed paradigm shift, since it does not require a change to the architecture of existing cloud systems. This shift would result in systems in which resources decide for themselves how best they can be used. This paper takes this idea to its logical conclusion and proposes a system for supporting self-managing resources in cloud environments. It introduces the concept of coalitions, consisting of collaborating resources, formed for the purpose of service delivery. It suggests the utility of restricting the interactions between the end-user and the cloud service provider to a well-defined services interface. It shows how clouds can be considered functionally, as engines for delivering an appropriate set of resources in response to service requests. And finally, since modern applications are increasingly constructed from sophisticated workflows of complex components, it shows how combinatorial auctions can be used to effectively deliver packages of resources to support those workflows.

The access infrastructure to the Cloud is usually a major drawback that limits the uptake of Cloud services. Attention has turned to rethinking a new architectural deployment of the overall Cloud service delivery. We argue that it is not sufficient to integrate the cloud domain with the operator's network domain based on the current models. In this work we envision a full integration of the Cloud and the network, where cloud resources are no longer confined to a data center, but are spread throughout the network and owned by the network operator. In such an environment, challenges arise at different levels, such as at the resource management, where both cloud and network resources need to be managed in an integrated approach. We particularly address the resource allocation problem through joint virtualization of network and cloud resources, by proposing an algorithm to allocate cloud and network resources in an integrated way. This algorithm is evaluated through both simulation and experimental results in a real virtualization platform.