Modeling Instability for Large Scale Processing Tasks Within HEP Distributed Computing Environments

SUMMARYIt is well known that data centers are consuming a large amount of energy that incurs significant financial and environmental costs. Recently, there has been an increasing interest in utilizing green energy for data centers, where green energy sources include solar and wind. This paper studies the crucial problem of maximizing the utilization of green energy through scheduling complex jobs in data centers in order to reduce the use of traditional brown energy. However, it is highly challenging for data centers to make use of green energy. First, the availability of typical green energy is variable to dynamic changes of natural environments, for example, weather. Second, although predictions can be made for the future availability of green energy, it is inevitable that such predictions have errors. Third, jobs are associated with strict deadlines, and it is required that jobs are completed before their deadlines. Finally, because the reliability in a data center relies upon temperature, the awareness of temperature should be taken into account while maximizing the green energy. In this paper, we consider online scheduling of jobs whose arrivals to the data center system dynamically. In addition, we explicitly take the power consumption of switches into account when scheduling jobs onto computing nodes. Two solar energy‐aware algorithms called SEEDMin and SEEDMax have been proposed. Then, we extend SEED to RSEED with the awareness of reliability. To evaluate the effectiveness of the proposed algorithms, comprehensive simulations have been conducted, and the proposed algorithms are compared with other state‐of‐art algorithms. Experimental results demonstrate that both SEEDMin and SEEDMax can significantly increase the utilization of solar energy without violating job deadlines and overall energy budget. The amount of solar energy utilized by SEEDMin and SEEDMax is 33.4%and35.3% larger than that of two traditional scheduling algorithms, MinMin and MinMax, respectively. Also, it can be seen that RSEED greatly improves the reliability by decreasing the temperature. Copyright © 2013 John Wiley & Sons, Ltd.

For MapReduce jobs in data center, network traffic is generated in shuffling phase, causing east-west communication bottleneck. Aiming at this problem, an optimization scheme is proposed to aggregate relevant network traffic flows into local areas. Firstly, the characteristics of pre-scheduling of MapReduce jobs are extracted, the communication activity degree of jobs is defined and the computing jobs are divided into two types: active or inactive communication job. Then the Bayesian classification with active learning is used as the prediction model, and this model after training by sample data can determine job type. The active communication jobs are deployed in the same rack to improve network bandwidth utilization. The experiment results of small-scale data center show that the proposed communication optimization scheme has a significant effect on shuffling intensive jobs, reaching 4.2%-5.6%. In the case of larger amount of input data, this scheme has better robustness and can effectively reduce east-west communication delay in data center.

Data analytic applications built upon big data processing frameworks such as Apache Spark are an important class of applications. Many of these applications are not latency-sensitive and thus can run as batch jobs in data centers. By running multiple applications on a computing host, task co-location can significantly improve the server utilization and system throughput. However, effective task co-location is a non-trivial task, as it requires an understanding of the computing resource requirement of the co-running applications, in order to determine what tasks, and how many of them, can be co-located. State-of-the-art co-location schemes either require the user to supply the resource demands which are often far beyond what is needed; or use a one-size-fits-all function to estimate the requirement, which, unfortunately, is unlikely to capture the diverse behaviors of applications.

Demand of data storage and processing of datasets in cloud computing consumes a huge volume of power and energy. Exhaustive usage of power and energy by the cloud service providers globally increases carbon footprint which is an environmental concern. Latest research results on carbon emission indicate a warning sign for environmental sustainability. Data centers, their resources and the digital waste created in the cloud environment have created a major impact on the environment. Taking this as an alarm sign on a global perspective, we rigorously worked on to develop “green software” in order to minimize the carbon waste from the environment. The design of this software focuses on to measure energy consumption and the carbon footprint level. This green software shall motivate the cloud services to uphold the concern on carbon emission and to take immediate attention to predict the waste of CPU power. This green software design framework provides a separate track session on energy consumption by the data center and cloud service provider along with the acquired energy efficiency level by the specific center. The software architecture insights are the working elements to analyze the level of emission, time period of the effect and the impact assessment of the carbon footprint. We propose an energy-efficient and cost-effective solution to minimize carbon emission and its impact on the environment.

Nowadays, Cloud Computing acts a major role in every field. These days, more large data centers are in service and many small cloud data centers are enlarging all over the universe. Cloud Computing is a catchword in the domain of HPC and offers on-demand services to the resources on the internet. The VMs (Virtual Machines) specified in the cloud data centres may have different specifications and instable resource usage, which causes imbalanced resource utilization within servers. Thus, it leads to performance degradation. Hence to achieve efficient selection of VM, these challenges must be addressed and solved by using meta-heuristics algorithms. In order to process the data, the VMs are placed on the PMs (Physical Machines). There will be multiple and dynamic request of input in the IaaS(Infrastructure as a Service) framework, hence the system’s responsibility is to create a VMs without knowing the types of tasks. Therefore, the fixed tasks scheduling is not right for this system. The most important research area that needs to be addressed is its performance in scheduling. The best and optimal solution is to find out in the cloud environment. Metaheuristics-based algorithms provide the near-optimal solution. In this paper, we proposed an Improved Particle Swarm Optimization algorithm to reduce the makespan and improve the throughput. We have compared our results with adaptive three-threshold energy-aware (ATEA) algorithm and PSO. The investigational results display the proposed Improved PSO algorithm will schedule and balance the load in the dynamic cloud environment better than the other approaches.

Cloud computing is rapidly taking over the information technology industry because it makes computing a lot easier without worries of buying the physical hardware needed for computations, rather, these services are hosted by companies with provide the cloud services. These companies contain a lot of computers and servers whose main source of power is electricity, hence, design and maintenance of these companies is dependent on the availability of steady and cheap electrical power supply. Cloud centers are energy-hungry. With recent spikes in electricity prices, one of the main challenges in designing and maintenance of such centers is to minimize electricity consumption of data centers and save energy. Efficient data placement and node scheduling to offload or move storage are some of the main approaches to solve these problems. In this article, we propose an Extreme Gradient Boosting (XGBoost) model to offload or move storage, predict electricity price, and as a result reduce energy consumption costs in data centers. The performance of this method is evaluated on a real-world dataset provided by the Independent Electricity System Operator (IESO) in Ontario, Canada, to offload data storage in data centers and efficiently decrease energy consumption. The data is split into 70% training and 30% testing. We have trained our proposed model on the data and validate our model on the testing data. The results indicate that our model can predict electricity prices with a mean squared error (MSE) of 15.66 and mean absolute error (MAE) of 3.74% respectively, which can result in 25.32% cut in electricity costs. The accuracy of our proposed technique is 91% while the accuracy of benchmark algorithms RF and SVR is 89% and 88%, respectively.

With the rise in cyber-attacks on cloud environments like Brute Force, Malware or Distributed Denial of Service attacks, information security officers and data center administrators have a monumental task on hand. Organizations design data center and service delivery with the aim of catering to maximize device provisioning & availability, improve application performance, ensure better server virtualization and end up securing data centers using security solutions at internet edge protection level. These security solutions prove to be largely inadequate in times of a DDoS cyber-attack. In this paper, traditional data center design is reviewed and compared to the proposed three tier data center. The resilience to withstand against DDoS attacks is measured for Real User Monitoring parameters, compared for the two infrastructure designs and the data is validated using T-Test.

With the emerging diverse applications in data centers, the demands on quality of service in data centers also become diverse, such as high throughput of elephant flows and low latency of deadline-sensitive flows. However, traditional TCPs are ill-suited to such situations and always result in the inefficiency (e.g. missing the flow deadline, inevitable throughput collapse) of data transfers. This further degrades the user-perceived quality of service (QoS) in data centers. To reduce the flow completion time of mice and deadline-sensitive flows along with promoting the throughput of elephant flows, an efficient and deadline-aware priority-driven congestion control (PCC) protocol, which grants mice and deadline-sensitive flows the highest priority, is proposed in this paper. Specifically, PCC computes the priority of different flows according to the size of transmitted data, the remaining data volume, and the flows' deadline. Then PCC adjusts the congestion window according to the flow priority and the degree of network congestion. Furthermore, switches in data centers control the input/output of packets based on the flow priority and the queue length. Different from existing TCPs, to speed up the data transfers of mice and deadline-sensitive flows, PCC provides an effective method to compute and encode the flow priority explicitly. According to the flow priority, switches can manage packets efficiently and ensure the data transfers of high priority flows through a weighted priority scheduling with minor modification. The experimental results prove that PCC can improve the data transfer performance of mice and deadline-sensitive flows while guaranting the throughput of elephant flows.

Cloud computing data centers are becoming increasingly popular for the provisioning of computing resources. The cost and operating expenses of data centers have skyrocketed with the increase in computing capacity. Several governmental, industrial, and academic surveys indicate that the energy utilized by computing and communication units within a data center contributes to a considerable slice of the data center operational costs.In this paper, we present a simulation environment for energy-aware cloud computing data centers. Along with the workload distribution, the simulator is designed to capture details of the energy consumed by data center components (servers, switches, and links) as well as packet-level communication patterns in realistic setups.The simulation results obtained for two-tier, three-tier, and three-tier high-speed data center architectures demonstrate the effectiveness of the simulator in utilizing power management schema, such as voltage scaling, frequency scaling, and dynamic shutdown that are applied to the computing and networking components.

Cloud computing data centers are becoming increasingly popular for the provisioning of computing resources. The cost and operating expenses of data centers have skyrocketed with the increase in computing capacity. Several governmental, industrial, and academic surveys indicate that the energy utilized by computing and communication units within a data center contributes to a considerable slice of the data center operational costs. In this paper, we present a simulation environment for energy-aware cloud computing data centers. Along with the workload distribution, the simulator is designed to capture details of the energy consumed by data center components (servers, switches, and links) as well as packet-level communication patterns in realistic setups. The simulation results obtained for two-tier, three- tier, and three-tier high-speed data center architectures demonstrate the effectiveness of the simulator in utilizing power management schema, such as voltage scaling, frequency scaling, and dynamic shutdown that are applied to the computing and networking components.

The explosive growth of GPU data centers is driven by the surging demand for machine learning, artificial intelligence, and high-performance computing applications, with the pressure on the power grid intensifying. Meanwhile, as renewable energy becomes more integrated into the grid, the carbon intensity of grid electricity varies at different times. This variability presents an opportunity for data centers to align their workloads with periods of high renewable energy generation and low carbon intensity, thereby reducing their operational carbon emissions. This paper proposes a multi-objective scheduling strategy to minimize carbon emissions and maximize quality of service (QoS) by identifying the Pareto front. The proposed strategy employs an integrated reinforcement learning model to dynamically integrate action strategies for multiple objectives. Results from a case study based on a real-world, large-scale data center illustrate the method’s effectiveness. This effectiveness arises from its dual-objective job scheduling strategy, which simultaneously reduces carbon emissions from electricity consumption and maintains a high quality of service in data centers. The method aims to promote the low-carbon operation of data centers, provide decision support, and facilitate coordination between data centers and power systems in real-world applications.

Based on the virtualization technology and pushed by the softwarization paradigm and the actual demand for services and resources, commercial cloud data centers know an unprecedented expansion.The systematic presence of software and the services generated enabled the development of the already dense application expenses. This causes, not only a cost explosion, especially when proprietary solutions protected by licenses are in hand, but also, represents a critical need in terms of software asset and resource management at the SaaS level. In addition to these costs, inefficient resource utilization, and the resulting energy represent an important part of the operational expenditure of data centers and are still a hot topic despite the consolidation initiatives put in place. The main objective of the consolidation service is to maximize resource exploitation while minimizing energy consumption and costs, among others. Even so, we have noticed that the reported literature doesn’t treat license management in the cloud environment as a whole, especially, from the resource management perspective and the overwhelming majority of the consolidation work focuses on resource optimization at the IaaS level. Therefore, we propose a reinforcement learning-based scheme that allows efficient use of resources and optimizes costs, energy consumption, and resource wastage, while remaining compliant. The experimental results show that our intelligent consolidator outperforms the baseline approaches according to the evaluation metrics used regardless of the resource heterogeneity and the data center dimensionality.

Our research aims to improve energy efficiency in data centres by combining cloud computing infrastructure with machine learning techniques. We propose that dynamic resource assignment, combined with intelligent optimisation of cooling systems, can reduce power waste and operational costs. Real-time sensor data from various data centre components such as servers, cooling systems, and power distribution units is collected and fed into machine learning models for analysis. In this way, we can create power arrangements that are tailored to the resources and needs of various applications. The experimental results show that energy consumption has been reduced by an average of 30% compared to traditional methods. Furthermore, our machine learning models are quite accurate in predicting cooling system performance. For example, an Artificial Neural Network (ANN) has an accuracy rate of 98.78%. This result demonstrates the efficacy of our approach in promoting energy efficiency and operational performance in data centres: it not only provides a scalable, cost-effective solution to industry energy efficiency challenges, but it also improves day-to-day data centre operations by reducing electrical consumption. Our approach, which is based on dynamic allocation of computational resources and real-time data analysis for optimising cooling systems, not only saves energy but also improves operational efficiency. With energy in mind, we must all work towards a more sustainable and green approach to data centre management. Future research can look into other potential optimisations, as well as issues with scalability and application in real-world data centre environments.

Cloud computing as a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal management effort or service provider interaction. Cloud environment comprises of abundant nodes with distributed computing resources in different geographic locations. The main aim is to maintain all the data centres to be available and also to distribute the load uniformly to all the data centres. Load balancing with switching mechanism contains two different algorithms in which, one algorithm is used to distribute the jobs in idle state divisions and another, token-based load balancing (TLB) algorithm is used to distribute the jobs in the normal state divisions. The basic demand of each user is to complete requested jobs in the short time, so the public cloud need a mechanism, which can complete the jobs of all users with reasonable response time. The proposed work chooses different strategies for different load conditions. Whenever the server is overloaded, the TLB algorithm migrates the job to the available data centre (DC) either within a cloud division or to the data centre in other cloud division.

The widespread demand for hosting application services in the cloud has been fueled by the deployment of cloud data centers (CDCs) on a global scale. Furthermore, modern apps' resource needs have sharply increased, especially in industries that use a lot of data. As a result, more cloud servers have been made available, resulting in higher energy usage and, ecological problems. Large-scale data centers have been developed as a result of the rapidly increasing demand for cloud services, allowing application service providers to rent data center space for application deployment by user-required quality of service (QoS). These data centers use a lot of electricity, which raises running expenses and produces more carbon dioxide (CO2) emissions. Modern cloud computing environments must also provide QoS for their users, necessitating a trade-off between power performance, energy consumption, and service-level agreement (SLA) compliance. We present an intelligent resource management policy using enforcement learning for CDCs. The objective is to continuously consolidate and dynamically allocate virtual machines (VMs). Utilizing live migration and disabling inactive nodes to reduce power consumption in this cloud environment while maintaining service quality. To enable dynamic resource management, a better power-performance tradeoff, and significantly lower energy consumption, we integrate several artificial intelligence concepts. Based on the result the proposed approach is more efficient as compared with other techniques.