A taxonomy and survey on scheduling algorithms for scientific workflows in IaaS cloud computing environments

The public cloud environment has emerged as a promising platform for exe-cuting scientific workflows. These executions involve leasing virtual machines (VMs) from public services for the duration of the workflow. The structure of the workflows significantly impacts the performance of any proposed scheduling approach. A task within a workflow cannot begin its execution before receiving all the required data from its preceding tasks. In this paper, we introduce a multi-priority scheduling approach for executing workflow tasks in the cloud. The key component of the proposed approach is a mechanism that logically or-ders and groups workflow tasks based on their data dependencies and locality. Using the proposed approach, the number of available VMs influences the num-ber of groups (partitions) obtained. Based on the locality of each group’s tasks, the priority of each group is determined to reduce the overall execution delay and improve VM utilization. As the results demonstrate, the proposed approach achieves a significant reduction in both execution costs and time in most scenar-ios

We consider the problem of optimizing the execution of data-intensive scientific workflows in the Cloud. We address the problem under the following scenario. The tasks of the workflows communicate through files; the output of a task is used by another task as an input file and if these tasks are assigned on different execution sites, a file transfer is necessary. The output files are to be stored at a site. Each execution site is to be assigned a certain percentage of the files and tasks. These percentages, called target weights, are pre-determined and reflect either user preferences or the storage capacity and computing power of the sites. The aim is to place the data files into and assign the tasks to the execution sites so as to reduce the cost associated with the file transfers, while complying with the target weights. To do this, we model the workflow as a hypergraph and with a hypergraph-partitioning-based formulation, we propose a heuristic which generates data placement and task assignment schemes simultaneously. We report simulation results on a number of real-life and synthetically generated scientific workflows. Our results show that the proposed heuristic is fast, and can find mappings and assignments which reduce file transfers, while respecting the target weights.

The scale of scientific applications becomes increasingly large not only in computation, but also in data. Many of these applications also concern inter-related tasks with data dependencies, hence, they are scientific workflows. The efficient coordination of executing/running scientific workflows is of great practical importance. The core of such coordination is scheduling and resource allocation. In this paper, we present three scheduling heuristics for running large-scale, data-intensive scientific workflows in clouds. In particular, the three heuristic algorithms are designed to leverage slot queue threshold, data locality and data prefetching, respectively. We also demonstrate how these heuristics can be collectively used to tackle different issues in running "data-intensive" workflows in clouds although each of these heuristics can be used independently. The practicality of our algorithms has been realized by actually implementing and incorporating them into our workflow execution system (DEWE). Using Montage, an astronomical image mosaic engine, as an example workflow, and Amazon EC2 as the cloud environment, we evaluate the performance of our heuristics in terms primarily of completion time (make span). We also scrutinize workflow execution showing different execution phases to identify their impact on performance. Our algorithms scale well and reduce make span by up to 27%.

Next-generation e-Science features large-scale, compute-intensive workflows of many computing modules that are typically executed in a distributed manner. With the recent emergence of cloud computing and the rapid deployment of cloud infrastructures, an increasing number of scientific workflows have been shifted or are in active transition to cloud environments. As cloud computing makes computing a utility, scientists across different application domains are facing the same challenge of reducing financial cost in addition to meeting the traditional goal of performance optimization. We develop a prototype generic workflow system by leveraging existing technologies for a quick evaluation of scientific workflow optimization strategies. We construct analytical models to quantify the network performance of scientific workflows using cloud-based computing resources, and formulate a task scheduling problem to minimize the workflow end-to-end delay under a user-specified financial constraint. We rigorously prove that the proposed problem is not only NP-complete but also non-approximable. We design a heuristic solution to this problem, and illustrate its performance superiority over existing methods through extensive simulations and real-life workflow experiments based on proof-of-concept implementation and deployment in a local cloud testbed.

Usually, scientists need to execute experiments that demand high performance computing environments and parallel techniques. This is the scenario found in many bioinformatics experiments modeled as scientific workflows, such as phylogenetic and phylogenomic analyses. To execute these experiments, scientists have adopted virtual machines (VMs) instantiated in clouds. Estimating the number of VMs to instantiate is a crucial task to avoid negative impacts on the execution performance and on the financial costs with under or overestimations. Previously, the necessary number of VMs to execute bioinformatics workflows have been estimated by a GRASP heuristic and have been coupled to a Cloud-based Parallel Scientific Workflow Management System. Although this work was a step forward, this approach only provided a static dimensioning. If the characteristics of the environment change (processing capacity, network speed), this static dimensioning may not be suitable. In this way, it is of interest that the dimensioning is adjusted at runtime. To achieve this, we developed a novel framework for monitoring and dynamically dimensioning resources during the execution of parallel scientific workflows in clouds, called Dynamic Dimensioning of Cloud Computing Framework (DDC-F). We have evaluated DDC-F in real executions of bioinformatics workflows. Experiments showed that DDC-F is able to efficiently calculate the number of VMs necessary to execute bioinformatics workflows of Comparative Genomics (CG), also reducing the financial costs, when compared with other works of the related literature.

Cloud computing emerges with high performance computing, massive data storage and easy access of the Internet. By deploying cloud computing, scientific workflows can be more cost-effective. During workflow execution, large volume data transmission may occur among multiple datacenters, hence incur large cost. Traditional approaches aim at finding data placement strategies for individual workflows only to reduce data transmission cost. However, workflows may share some datasets. Therefore, optimal data placement considering individual workflows in isolation is not necessarily optimal for the situation of multiple workflows as a whole. In this paper, by facilitation of Particle Swarm Optimization (PSO), we build a novel data transmission cost model for developing a new multi-datacenter cost-effective data placement strategy. The experimental results show that our strategy is much better than its traditional counterparts.

Workflow scheduling is one of the burning topics that has drawn enormous attention recently in the research community of cloud computing due to its wide applications in astronomy, physics, bioinformatics, health care and so on. This is a well-known NP-complete problem. It presents an interesting aspect of achieving minimum processing time of all the tasks and maximum resource utilization in cloud resources. Therefore, many algorithms have been developed for workflow scheduling. However, most of them consider a static priority of the tasks which is non-realistic for heterogeneous cloud computing environment. In this paper, we propose a workflow scheduling algorithm which considers dynamic priority of the tasks. The algorithm undergoes a process of min–max normalization followed by the calculation of the dynamic threshold to dispatch the tasks into one of the virtual machines. The algorithm is extensively simulated using various benchmark, scientific and real-life workflows. All the simulated results are compared with other four existing workflow scheduling algorithms. The simulated results confirm that the proposed algorithm lags behind all the four existing algorithms in terms of makespan and average cloud resource utilization. The simulation results are also validated through analysis of variance statistical test.

Executing scientific workflows in high-performance cloud computing platforms requires the use of scheduling algorithms that allow workflows execution as fast as possible, while minimizing the monetary cost of such executions. In this work we study how the use of supporting information can offer guidance to scheduling algorithms, helping them to devise more efficient execution plans in terms of the total execution time (makespan) and the total monetary cost. Using two large-scale scientific workflows, our experiments showed that simple modifications on a classical scheduling algorithm (HEFT), in conjunction with the appropriate supporting information, could reduce the monetary cost of an execution in up to 59% and reduce the makespan in up to 8.6%.

Data-intensive scientific applications are posing many challenges in distributed computing systems. In the scientific field, the application data are expected to double every year over the next decade and further. With this continuing data explosion, high performance computing systems are needed to store and process data efficiently, and workflow technologies are facilitated to automate these scientific applications. Scientific workflows are typically very complex. They usually have a large number of tasks and need a long time for execution. Running scientific workflow applications usually need not only high performance computing resources but also massive storage. The emergence of cloud computing technologies offers a new way to develop scientific workflow systems. Scientists can upload their data and launch their applications on the scientific cloud workflow systems from everywhere in the world via the Internet, and they only need to pay for the resources that they use for their applications. As all the data are managed in the cloud, it is easy to share data among scientists. This kind of model is very convenient for users, but remains a big challenge to the system. This paper proposes several research topics of data management in scientific cloud workflow systems, and discusses their research methodologies and state-of-the-art solutions.

A heterogeneous system can be portrayed as a variety of unlike resources that can be locally or geologically spread, which is exploited to implement data-intensive and computationally intensive applications. The competence of implementing the scientific workflow applications on heterogeneous systems is determined by the approaches utilized to allocate the tasks to the proper resources. Cost and time necessity are evolving as different vital concerns of cloud computing environments such as data centers. In the area of scientific workflows, the difficulties of increased cost and time are highly challenging, as they elicit rigorous computational tasks over the communication network. For example, it was discovered that the time to execute a task in an unsuited resource consumes more cost and time in the cloud data centers. In this paper, a new cost- and time-efficient planning algorithm for scientific workflow scheduling has been proposed for heterogeneous systems in the cloud based upon the Predict Optimistic Time and Cost (POTC). The proposed algorithm computes the rank based not only on the completion time of the current task but also on the successor node in the critical path. Under a tight deadline, the running time of the workflow and the transfer cost are reduced by using this technique. The proposed approach is evaluated using true cases of data-exhaustive workflows compared with other algorithms from written works. The test result shows that our proposed method can remarkably decrease the cost and time of the experimented workflows while ensuring a better mapping of the task to the resource. In terms of makespan, speedup, and efficiency, the proposed algorithm surpasses the current existing algorithms—such as Endpoint communication contention-aware List Scheduling Heuristic (ELSH)), Predict Earliest Finish Time (PEFT), Budget-and Deadline-constrained heuristic-based upon HEFT (BDHEFT), Minimal Optimistic Processing Time (MOPT) and Predict Earlier Finish Time (PEFT)—while holding the same time complexity.

In this paper, we address the scheduling of scientific workflows in hybrid clouds considering data placement and present the Hybrid Scheduling for Hybrid Clouds (HSHC) algorithm. HSHC is a two-phase scheduling algorithm with a genetic algorithm based static phase and dynamic programming based dynamic phase. We evaluate HSHC with both a real-world scientific workflow application and random workflows in terms of makespan and costs.

Many intelligent applications of Cloud and Edge can be modeled to be stochastically arrived workflows. Elastically provisioning resources such as containers to these applications is crucial to minimize resource costs while fulfilling deadlines. Most of existing workflow scheduling algorithms are either not for randomly arrived workflows from users of Edge Computing or only consider workflows in pure Cloud Computing. In this article, an elastic resource scheduling method, which integrates loosely coupled workflow scheduling with resource auto-scaling, is developed for stochastically arrived workflows in Cloud and Edge to minimize long-term operating costs while meeting deadline constraints. Experimental results illustrate that our approach achieves a lower overall cost and a higher success ratio compared with the state-of-art algorithm.KeywordsEdge computingCloud computingMicroserviceContainersWorkflow scheduling

Comparative Analysis of Scientific Workflow Scheduling Algorithms in Cloud Under Budget Constraint

Most of the large-scale scientific experiments modeled as scientific workflows produce a large amount of data and require workflow parallelism to reduce workflow execution time. Some of the existing Scientific Workflow Management Systems (SWfMS) explore parallelism techniques - such as parameter sweep and data fragmentation. In those systems, several computing resources are used to accomplish many computational tasks in homogeneous environments, such as multiprocessor machines or cluster systems. Cloud computing has become a popular high performance computing model in which (virtualized) resources are provided as services over the Web. Some scientists are starting to adopt the cloud model in scientific domains and are moving their scientific workflows (programs and data) from local environments to the cloud. Nevertheless, it is still difficult for the scientist to express a parallel computing paradigm for the workflow on the cloud. Capturing distributed provenance data at the cloud is also an issue. Existing approaches for executing scientific workflows using parallel processing are mainly focused on homogeneous environments whereas, in the cloud, the scientist has to manage new aspects such as initialization of virtualized instances, scheduling over different cloud environments, impact of data transferring and management of instance images. In this paper we propose SciCumulus, a cloud middleware that explores parameter sweep and data fragmentation parallelism in scientific workflow activities (with provenance support). It works between the SWfMS and the cloud. SciCumulus is designed considering cloud specificities. We have evaluated our approach by executing simulated experiments to analyze the overhead imposed by clouds on the workflow execution time.

In recent years, intelligent flow sensors have been applied to many fields. Cloud operation is a design method to further improve the intelligence of such sensors. However, the cloud workflows of intelligent flow sensors consume too much energy, making it imperative to schedule cloud workflows. With the growing awareness of energy conservation, it is a hot topic to design an energy-efficient workflow scheduling algorithm. Therefore, this paper puts forward the predict minimum energy consumption (PMEC) algorithm, a cloud workflow scheduling algorithm that strikes a balance between energy consumption and execution time. Firstly, the optimistic cost table (OCT) was adopted to rank the tasks by priority. Then, the resources, i.e. virtual machines, were assigned statically to the tasks, in the light of task priority and energy consumption. After that, the workflow was scheduled according to the assignments. Simulation results show that the PMEC is much more energy efficient than traditional listbased scheduling algorithms.