Cost-Aware Dynamic Resource Allocation in Distributed Computing Infrastructures

A growing number of very simple parallel applications can benefit from the availability of very large computing resource pools that are exploited in an opportunistic way to considerably speed up the processing of these applications. In particular, many applications can scale out to take advantage of the availability of thousands, or even millions, of processors that can be simultaneously used. Some large-scale distributed computing infrastructures (DCI) have already been successfully assembled to this end. However, these DCI are normally designed to cater to specific kinds of applications, and their assemblage is not only costly, but also time consuming. We propose a novel architecture, named OddCI (On-demand DCI), for the cheap, fast and on-demand instantiation and dismantle of very large DCI. These characteristics allow application users to dynamically commission DCI that can cost-effectively be customized to their particular needs, and disposed after being used. Our approach leverages on broadcast communication as an efficient mechanism to enable aggregation of geographically distributed computing resources, including millions of non-traditional processing devices such as mobile phones, and digital television (DTV) receivers. We discuss how the proposed architecture can be implemented on top of a network of DTV receivers, providing evidences of both the technical, as well as the operational feasibility of the OddCI architecture.

Nowadays visualization-based knowledge discovery can play an important role in astrophysics. Collaborative visualization can enable multiple users to share visualization experiences, e.g. by interacting simultaneously with astrophysical datasets giving feedback on what other participants are doing/seeing. Further, workflow-driven applications allow reproduction of specific visualization results, a challenging task as selecting suitable visualization parameters may not be a straightforward process. This paper presents VisIVO Science Gateway, a web-based workflow-enabled framework integrating large-scale, multidimensional datasets and applications for visualization and data filtering on Distributed Computing Infrastructures (DCIs). Advanced users are able to create, change, invoke, and monitor workflows while standard users are provided with easy-to-use customised web interfaces hiding all technical aspects of the visualization algorithms and DCI configurations.

OF THE THESIS Pilot based frameworks for Weather Research Forecasting by Dinesh Prasanth Ganapathi Thesis Director: Dr. Shantenu Jha The Weather Research Forecasting (WRF) domain consists of complex workflows that demand the use of Distributed Computing Infrastructure (DCI). Weather forecasting requires that weather researchers use different set of initial conditions and one or a combination of physics models on the same set of input data. For these type of simulations an ensemble based computing approach becomes imperative. Most DCIs have local job-schedulers that have no smart way of dealing with the execution of an ensemble type of computational problem as the job-schedulers are built to cater to the bare essentials of resource allocation. This means the weather scientists have to submit multiple jobs to the job-scheduler. In this dissertation we use Pilot-Job based tools to decouple work-load submission and resource allocation therefore streamlining the complex workflows in Weather Research and Forecasting domain and reduce their overall time to completion. We also achieve location independent job execution, data movement, placement and processing. Next, we create the necessary enablers to run an ensemble of tasks bearing the capability to run on multiple heterogeneous distributed computing resources there by creating the opportunity to minimize the overall time consumed in running the models. Our experiments show that the tools developed exhibit very good, strong and weak scaling characteristics. These results bear the potential to change the way weather researchers are submitting traditional WRF jobs to the

Distributed Computing Infrastructure is characterized by interfaces that are heterogeneous—syntactically and semantically. SAGA represents the most comprehensive community effort to date to address the heterogeneity by defining a simple, uniform access layer. In this paper, we describe the basic concepts underpinning its design and development. We also discuss RADICAL-SAGA which is the most widely used implementation of SAGA.

Virtual high-throughput screening (vHTS) is an invaluable method in modern drug discovery. It permits screening large datasets or databases of chemical structures for those structures binding possibly to a drug target. Virtual screening is typically performed by docking code, which often runs sequentially. Processing of huge vHTS datasets can be parallelized by chunking the data because individual docking runs are independent of each other. The goal of this work is to find an optimal splitting maximizing the speedup while considering overhead and available cores on Distributed Computing Infrastructures (DCIs). We have conducted thorough performance studies accounting not only for the runtime of the docking itself, but also for structure preparation. Performance studies were conducted via the workflow-enabled science gateway MoSGrid (Molecular Simulation Grid). As input we used benchmark datasets for protein kinases. Our performance studies show that docking workflows can be made to scale almost linearly up to 500 concurrent processes distributed even over large DCIs, thus accelerating vHTS campaigns significantly.

In this paper, we address job scheduling in Distributed Computing Infrastructures, that is a loosely coupled network of autonomous acting High Performance Computing systems. In contrast to the common approach of mutual workload exchange, we consider the more intuitive operator's viewpoint of load-dependent resource reconfiguration. In case of a site's over-utilization, the scheduling system is able to lease resources from other sites to keep up service quality for its local user community. Contrary, the granting of idle resources can increase utilization in times of low local workload and thus ensure higher efficiency. The evaluation considers real workload data and is done with respect to common service quality indicators. For two simple resource exchange policies and three basic setups we show the possible gain of this approach and analyze the dynamics in workload-adaptive reconfiguration behavior.

WS-PGRADE/gUSE is a gateway framework that offers a set of high-level grid and cloud services by which interoperation between grids, clouds, and scientific user communities can be achieved. gUSE is also a workflow system that enables scientific communities to compose and execute a series of computational or data manipulation steps in a scientific application on Distributed Computing Infrastructures (DCIs). The current chapter summarizes the most important features of WS-PGRADE/gUSE.

The Cherenkov Telescope Array (CTA) - an array of several tens of Cherenkov telescopes - is the next generation of ground-based instrument in the field of very high energy gamma-ray astronomy. The CTA observatory is expected to produce a main data stream for permanent storage of the order of 1-to-5 GB/s for about 1000 hours of observation per year, thus producing a total data volume of the order of several PB per year. The CPU time needed to calibrate and process one hour of data taking will be of the order of some thousands CPU hours with current technology. The high data rate of CTA, together with the large computing power requirements for Monte Carlo (MC) simulations, need dedicated computing resources. Massive MC simulations are needed to study the physics of cosmic-ray atmospheric showers as well as telescope response and performance for different detectors and layout configurations. Given these large storage and computing requirements, the Grid approach is well suited, and a vast number of MC simulations are already running on the European Grid Infrastructure (EGI). In order to optimize resource usage and to handle all production and future analysis activities in a coherent way, a high-level framework with advanced functionalities is desirable. For this purpose we have preliminarly evaluated the DIRAC framework for distributed computing and tested it for the CTA workload and data management systems. In this paper we present a possible implementation of a Distributed Computing Infrastructure (DCI) Computing Model for CTA as well as the benchmark test results of DIRAC.

The availability of large quantities of processors is a crucial enabler of many-task computing. Voluntary computing systems have proven that it is possible to build computing platforms with millions of nodes to support the execution of embarrassingly parallel applications. These systems, however, lack the flexibility of more traditional grid infrastructures. On the other hand, flexible infrastructures currently available can gather only dozens of thousands nodes. We propose a novel architecture for generic Distributed Computing Infrastructures (DCI) that can be instantiated on demand to be, at the same time, flexible and highly-scalable. Bringing the scalability from voluntary computing, the flexibility from grid computing and the elasticity from cloud computing in a single arrangement, our proposal allows for fast setup, fast initialization and fast dismantle of customized DCI supported by both dedicated and shared underlying infrastructures. Our approach leverages broadcast communication as an efficient mechanism to enable aggregation of geographically distributed computing resources, including millions of non-traditional processing devices such as PDA, mobile phones and Digital TV receivers, using both opportunistic and non-opportunistic models. We show the feasibility of the proposed architecture by implementing it atop a digital television system. We also assess the performance of such system and show that it can be used to execute several classes of many-tasks computing applications with very high efficiency, substantially decreasing their response time.

Distributed computing owes its success to the development of infrastructure, middleware, and standards (e.g., CORBA) to support interoperability. The computing community has slowly recognized the need to protect information and has begun to develop commercial security infrastructures and standards. The US Government must protect national security information against unauthorized information flow. To support MLS distributed computing, a MLS infrastructure must be built that enables information sharing among users at different classification levels. This infrastructure should provide MLS services for protection of classified information and use both the emerging distributed computing and commercial security infrastructures, when possible. The resulting infrastructure will enable users to integrate commercial information technology products into their systems. In this paper, we examine the philosophy that has led to successful distributed computing among heterogeneous, autonomous components and propose an analogous approach for MLS distributed computing. We identify some services that are required to support MLS distributed computing, argue that these services are needed regardless of the MLS architecture used, present an approach for designing these services, and provide design guidance for a critical building block of the MLS infrastructure.

Recent research has shown that the cooperative diversity can efficiently combat the fading effects caused by the hostile wireless channels just as the multiple-input multiple-output (MIMO) antenna systems do. Besides, due to the inherent cost and size constraints by many small area wireless networks, e.g. wireless ad hoc networks and wireless sensor networks, in such environments, it would be almost impossible to deploy the antenna array in a mobile node. However, cooperative diversity can provide an economical and viable solution by sharing each mobile node’s single-antenna with others to form a virtual antenna array, while it still can maintain significant performance gains similar to MIMO systems. Meanwhile, the adaptive modulation is a proven technique which can significantly increase the spectral efficiency when perfect channel estimation can be achieved, and many wireless standards have adopted it as an implementation option. It also has been shown that combining these two techniques can greatly raise the system efficiency. Nevertheless, in the multiple nodes cooperative communication networks, the randomness of demodulation process at each relay node makes the task of correctly selecting the forwarding relay nodes very complicated. In view of this problem, a novel adaptive node selection algorithm is proposed for the destination node to effectively select the forwarding relay nodes to help the source node transfer the signals. In this paper, the decode-and-forward (DF) protocol is employed as the cooperative diversity signaling, and the constant-power variable-rate M-ary phase shift keying (M-PSK) is adopted as the adaptive modulation scheme. At the destination node, the maximal-ratio-combining (MRC) method is used to combine all of the signals from diversity branches with the main path transmitted directly from the source node. Because of the random phenomenon of the demodulation process, the outage requirement is used as the baseline in the first stage of the proposed node selection algorithm to choose the maximal transmission rate. Hence the maximal overall system throughput in terms of rate can be achieved. In the meantime, the first candidate forwarding sets of the relay nodes which collectively meet the required outage probability are also been chosen. Then in the second stage, we can further refine the number of the relay nodes among the first candidate forwarding sets by using the following criterion — maximum average rate per node. It will screen out the first candidate sets with the minimum number of the relay nodes. If only one candidate set is left at the end of the second stage, it will be the end of the proposed algorithm. However, if the result of the screening still has more than one set, then the final election among the second candidate sets would be the one which can give the minimum outage probability. In the simulation results, our proposed node selection algorithm can even further improve the spectral efficiency while the deleteriously fading effect induced by the wireless channels can also be effectively removed.

In the cloud computing environment, task scheduling with multiple objectives optimization becomes a highly challenging problem in such a dynamic and bursty environment. Previous studies have mostly emphasized assigning the incoming tasks in a specific scenario, with a weak generalization ability to various objectives automatically. Thus, they suffer the inefficient issue under large-scale and heterogeneous cloud workloads. To address this issue, we propose a deep reinforcement learning (DRL)-based intelligent cloud task scheduler, which makes the optimal scheduling decision only dependent on learning directly from its experience without any prior knowledge. We formulate task scheduling as a dynamical optimization problem with constraints and then adopt the deep deterministic policy gradients (DDPG) network to find the optimal task assignment solution while meeting the performance and cost constraints. We propose a correlation-aware state representation method to capture the inherent characteristics of demands, and a dual reward model is designed to learn the optimal task allocation strategy. Extensive experimental results on Alibaba cloud workloads show that compared with other existing solutions, our proposed DDPG-based task scheduler enjoy superiority and effectiveness in performance and cost optimization.

The effectiveness of resource allocation directly affects process performance. In order to optimize resource allocation, this paper proposes a resource allocation model in view of the relationship between resource allocation and process performance, which minimizes process execution time in terms of resource preference, cost constraints and resource availability criteria. Resource coordination is paid less attention in previous resource allocation studies. Therefore, this paper presents the corresponding resource allocation method in consideration of resource coordination, the interval between adjacent activities and distinguishing turnaround time between different resources from event logs. The experiments show that the proposed method can effectively optimize the resource allocation.

Cloud computing is one among the emerging platforms in business, IT enterprise and mobile computing applications. Resources like Software, CPU, Memory and I/O devices etc. are utilized and charged as per the usage, instead of buying it. A Proper and efficient resource allocation in this dynamic cloud environment becomes the challenging task due to drastic increment in cloud usage. Various promising technologies have been developed to improve the efficiency of resource allocation process. But still there is some incompetency in terms of task scheduling and power consumption, when the system gets overloaded. So an energy efficient task scheduling algorithm is required to improve the efficiency of resource allocation process. In this paper an improved task scheduling and an optimal power minimization approach is proposed for efficient dynamic resource allocation process. Using prediction mechanism and dynamic resource table updating algorithm, efficiency of resource allocation in terms of task completion and response time is achieved. This framework brings an efficient result in terms of power reduction since it reduces the power consumption in data centers. The proposed approach gives accurate values for updating resource table. An efficient resource allocation is achieved by an improved task scheduling technique and reduced power consumption approach. The Simulation result gives 8% better results when comparing to other existing methods.

Stream processing is an emerging paradigm to handle data streams upon arrival, powering latency-critical application such as fraud detection, algorithmic trading, and health surveillance. Though there are a variety of Distributed Stream Processing Systems (DSPSs) that facilitate the development of streaming applications, resource management and task scheduling is not automatically handled by the DSPS middleware and requires a laborious process to tune toward specific deployment targets. As the advent of cloud computing has supported renting resources on-demand, it is of great interest to review the research progress of hosting streaming systems in clouds under certain Service Level Agreements (SLA) and cost constraints. In this article, we introduce the hierarchical structure of streaming systems, define the scope of the resource management problem, and present a comprehensive taxonomy in this context covering critical research topics such as resource provisioning, operator parallelisation, and task scheduling. The literature is then reviewed following the taxonomy structure, facilitating a deeper understanding of the research landscape through classification and comparison of existing works. Finally, we discuss the open issues and future research directions toward realising an automatic, SLA-aware resource management framework.