Multi-priority scheduling algorithm for scientific workflows in cloud

The scale of scientific applications becomes increasingly large not only in computation, but also in data. Many of these applications also concern inter-related tasks with data dependencies, hence, they are scientific workflows. The efficient coordination of executing/running scientific workflows is of great practical importance. The core of such coordination is scheduling and resource allocation. In this paper, we present three scheduling heuristics for running large-scale, data-intensive scientific workflows in clouds. In particular, the three heuristic algorithms are designed to leverage slot queue threshold, data locality and data prefetching, respectively. We also demonstrate how these heuristics can be collectively used to tackle different issues in running "data-intensive" workflows in clouds although each of these heuristics can be used independently. The practicality of our algorithms has been realized by actually implementing and incorporating them into our workflow execution system (DEWE). Using Montage, an astronomical image mosaic engine, as an example workflow, and Amazon EC2 as the cloud environment, we evaluate the performance of our heuristics in terms primarily of completion time (make span). We also scrutinize workflow execution showing different execution phases to identify their impact on performance. Our algorithms scale well and reduce make span by up to 27%.

We consider the problem of optimizing the execution of data-intensive scientific workflows in the Cloud. We address the problem under the following scenario. The tasks of the workflows communicate through files; the output of a task is used by another task as an input file and if these tasks are assigned on different execution sites, a file transfer is necessary. The output files are to be stored at a site. Each execution site is to be assigned a certain percentage of the files and tasks. These percentages, called target weights, are pre-determined and reflect either user preferences or the storage capacity and computing power of the sites. The aim is to place the data files into and assign the tasks to the execution sites so as to reduce the cost associated with the file transfers, while complying with the target weights. To do this, we model the workflow as a hypergraph and with a hypergraph-partitioning-based formulation, we propose a heuristic which generates data placement and task assignment schemes simultaneously. We report simulation results on a number of real-life and synthetically generated scientific workflows. Our results show that the proposed heuristic is fast, and can find mappings and assignments which reduce file transfers, while respecting the target weights.

Usually, scientists need to execute experiments that demand high performance computing environments and parallel techniques. This is the scenario found in many bioinformatics experiments modeled as scientific workflows, such as phylogenetic and phylogenomic analyses. To execute these experiments, scientists have adopted virtual machines (VMs) instantiated in clouds. Estimating the number of VMs to instantiate is a crucial task to avoid negative impacts on the execution performance and on the financial costs with under or overestimations. Previously, the necessary number of VMs to execute bioinformatics workflows have been estimated by a GRASP heuristic and have been coupled to a Cloud-based Parallel Scientific Workflow Management System. Although this work was a step forward, this approach only provided a static dimensioning. If the characteristics of the environment change (processing capacity, network speed), this static dimensioning may not be suitable. In this way, it is of interest that the dimensioning is adjusted at runtime. To achieve this, we developed a novel framework for monitoring and dynamically dimensioning resources during the execution of parallel scientific workflows in clouds, called Dynamic Dimensioning of Cloud Computing Framework (DDC-F). We have evaluated DDC-F in real executions of bioinformatics workflows. Experiments showed that DDC-F is able to efficiently calculate the number of VMs necessary to execute bioinformatics workflows of Comparative Genomics (CG), also reducing the financial costs, when compared with other works of the related literature.

SummaryLarge‐scale scientific problems are often modeled as workflows. The ever‐growing data and compute requirements of these applications has led to extensive research on how to efficiently schedule and deploy them in distributed environments. The emergence of the latest distributed systems paradigm, cloud computing, brings with it tremendous opportunities to run scientific workflows at low costs without the need of owning any infrastructure. It provides a virtually infinite pool of resources that can be acquired, configured, and used as needed and are charged on a pay‐per‐use basis. However, along with these benefits come numerous challenges that need to be addressed to generate efficient schedules. This work identifies these challenges and studies existing algorithms from the perspective of the scheduling models they adopt as well as the resource and application model they consider. A detailed taxonomy that focuses on features particular to clouds is presented, and the surveyed algorithms are classified according to it. In this way, we aim to provide a comprehensive understanding of existing literature and aid researchers by providing an insight into future directions and open issues.

Executing scientific workflows in high-performance cloud computing platforms requires the use of scheduling algorithms that allow workflows execution as fast as possible, while minimizing the monetary cost of such executions. In this work we study how the use of supporting information can offer guidance to scheduling algorithms, helping them to devise more efficient execution plans in terms of the total execution time (makespan) and the total monetary cost. Using two large-scale scientific workflows, our experiments showed that simple modifications on a classical scheduling algorithm (HEFT), in conjunction with the appropriate supporting information, could reduce the monetary cost of an execution in up to 59% and reduce the makespan in up to 8.6%.

Multi-objective scheduling of Scientific Workflows in multisite clouds

Next-generation e-Science features large-scale, compute-intensive workflows of many computing modules that are typically executed in a distributed manner. With the recent emergence of cloud computing and the rapid deployment of cloud infrastructures, an increasing number of scientific workflows have been shifted or are in active transition to cloud environments. As cloud computing makes computing a utility, scientists across different application domains are facing the same challenge of reducing financial cost in addition to meeting the traditional goal of performance optimization. We develop a prototype generic workflow system by leveraging existing technologies for a quick evaluation of scientific workflow optimization strategies. We construct analytical models to quantify the network performance of scientific workflows using cloud-based computing resources, and formulate a task scheduling problem to minimize the workflow end-to-end delay under a user-specified financial constraint. We rigorously prove that the proposed problem is not only NP-complete but also non-approximable. We design a heuristic solution to this problem, and illustrate its performance superiority over existing methods through extensive simulations and real-life workflow experiments based on proof-of-concept implementation and deployment in a local cloud testbed.

In recent years, intelligent flow sensors have been applied to many fields. Cloud operation is a design method to further improve the intelligence of such sensors. However, the cloud workflows of intelligent flow sensors consume too much energy, making it imperative to schedule cloud workflows. With the growing awareness of energy conservation, it is a hot topic to design an energy-efficient workflow scheduling algorithm. Therefore, this paper puts forward the predict minimum energy consumption (PMEC) algorithm, a cloud workflow scheduling algorithm that strikes a balance between energy consumption and execution time. Firstly, the optimistic cost table (OCT) was adopted to rank the tasks by priority. Then, the resources, i.e. virtual machines, were assigned statically to the tasks, in the light of task priority and energy consumption. After that, the workflow was scheduled according to the assignments. Simulation results show that the PMEC is much more energy efficient than traditional listbased scheduling algorithms.

Energy-efficient VM opening algorithms for real-time workflows in heterogeneous clouds

Due to the increasing number of constituting jobs and input data size, the execution of modern complex workflow-based applications on cloud requires a large number of virtual machines (VMs), which makes the cost a great concern. Under the constraints of VM processing and storage capabilities and communication bandwidths between VMs, how to quickly figure out a cost-optimal resource provisioning and scheduling solution for a given cloud workflow is becoming a challenge. The things become even worse when taking the infrastructure-related failures with transient characteristics into account. To address this problem, this paper proposes a soft error aware VM selection and task scheduling approach that can achieve near-optimal the lowest possible cost. Under the reliability and completion time constraints by tenants, our approach can figure out a set of VMs with specific CPU and memory configurations and generate a cost-optimal schedule by allocating tasks to appropriate VMs. Comprehensive experimental results on well-known scientific workflow benchmarks show that compared with state-of-the-art methods, our approach can achieve up to 66% cost reduction while satisfying both reliability and completion time constraints.

Cloud computing(CC) delivers multiple services to users by processing complex tasks through internet connections to serve as a dynamic, dependable, and flexible computing solution. With the increased Internet speed, the user demands to perform many tasks through computing provided by cloud virtual machines (VMs) created in heterogeneous servers. The complexity of cloud computing(CC) increases through the diversity of servers and the increased demand for resources to perform more tasks, which makes the cloud data center(CDC) unbalanced in load. Therefore, future cloud systems will require more effective resource management(RM) methods to balance load and improve Quality of Service (QoS). This paper proposes KMPPVM-DRM, a dynamic clustering and scheduling approach for VM resource allocation(RA) based on the KMeans++ algorithm. VMs are clustered into three groups (high, medium, low) according to their real-time normalized CPU and RAM weights, using weighting parameters α = 0.6 and β = 0.4. Tasks are categorized into three levels based on length and mapped to appropriate VM clusters. Within each cluster, a Utilization-Level Comparator (ULC) dynamically selects the optimal VM for task execution. The experimental results were conducted using CloudSim, with 5 and 10 VMs and task counts ranging from 1000 to 10000. The proposed model was compared with PSO, ACO, and DBSCAN algorithms using execution time, average start time, and average finish time. Results show that with 10 VMs, the proposed model achieved a harmonic mean execution time of 886.72 ms, compared to 4102.99 ms (PSO), 4672.25 ms (ACO), and 3071.51 ms (DBSCAN). It also attained the lowest harmonic mean start and finish times of 166.41 ms and 167.19 ms, respectively. Relative reduction in execution time against DBSCAN ranged from 69.90% to 71.52%, with start and finish time improvements between 52.65% and 81.51%.To ensure statistical reliability, a paired t-test confirmed that the performance improvements were statistically significant (p < 0.05) across all task sizes. The findings confirm that KMPPVM-DRM enhances resource allocation(RA) efficiency and scheduling effectiveness, maintaining balanced loads even with limited resources (e.g., only 5 VMs) and outperforming PSO, ACO, and DBSCAN in all tested scenarios.

In this paper, we address the scheduling of scientific workflows in hybrid clouds considering data placement and present the Hybrid Scheduling for Hybrid Clouds (HSHC) algorithm. HSHC is a two-phase scheduling algorithm with a genetic algorithm based static phase and dynamic programming based dynamic phase. We evaluate HSHC with both a real-world scientific workflow application and random workflows in terms of makespan and costs.

Scheduling large-scale and resource-intensive workflows in cloud infrastructure is one of the main challenges for cloud service providers (CSPs). Cloud infrastructure is more efficient when virtual machines and other resources work up to their full potential. The main factor that influences the quality of cloud services is the distribution of workflow on virtual machines (VMs). Scheduling tasks to VMs depends on the type of workflow and mechanism of resource allocation. Scientific workflows include large-scale data transfer and consume intensive resources of cloud infrastructures. Therefore, scheduling of tasks from scientific workflows on VMs requires efficient and optimized workflow scheduling techniques. This paper proposes an optimised workflow scheduling approach that aims to improve the utilization of cloud resources without increasing execution time and execution cost.

Data-intensive scientific applications are posing many challenges in distributed computing systems. In the scientific field, the application data are expected to double every year over the next decade and further. With this continuing data explosion, high performance computing systems are needed to store and process data efficiently, and workflow technologies are facilitated to automate these scientific applications. Scientific workflows are typically very complex. They usually have a large number of tasks and need a long time for execution. Running scientific workflow applications usually need not only high performance computing resources but also massive storage. The emergence of cloud computing technologies offers a new way to develop scientific workflow systems. Scientists can upload their data and launch their applications on the scientific cloud workflow systems from everywhere in the world via the Internet, and they only need to pay for the resources that they use for their applications. As all the data are managed in the cloud, it is easy to share data among scientists. This kind of model is very convenient for users, but remains a big challenge to the system. This paper proposes several research topics of data management in scientific cloud workflow systems, and discusses their research methodologies and state-of-the-art solutions.

Deadline-constrained cost-aware workflow scheduling in hybrid cloud