Grid Workflow Scheduling Based on Task and Data Locations

Currently the research on the grid workflow scheduling algorithms commonly suffer by one or several of the following drawbacks: grid workflow model of directed acyclic graph (DAG), not enough considerations of multi-dimensional Quality of Service (QoS) parameters and single objective optimal scheduling. Based on abstract grid workflow language (AGWL) model, this paper presents an algorithm for multi-objective optimal grid workflow scheduling with QoS constraints, which make uses of an improved NSGA-II algorithm. Theoretical analysis and experimental results indicate the feasibility and efficiency of the algorithm.KeywordsQuality of service (QoS)grid workflow schedulingNSGA-II

Workflow technology has been adopted in scientific domains to orchestrate and automate scientific processes in order to facilitate experimentation. Such scientific workflows often involve large data sets and intensive computation that necessitate the use of the Grid, which offers supercomputing power through shared distributed resources. To execute a scientific workflow in the Grid, tasks within the workflow that represent steps in the scientific process are assigned to Grid resources for execution. To ensure efficient execution of the workflow, Grid workflow scheduling is required to manage the allocation of Grid resources. Although many Grid workflow scheduling techniques exist, they are mainly designed for the execution of a single workflow. This is not the case with parameter sweep workflows which focus on parametric study and parameter optimisation. A parameter sweep workflow is executed numerous times with different input parameters in order to determine the effect of each parameter combination on the experiment. While executing multiple instances of a parameter sweep workflow in parallel can reduce the time required for the overall execution, this execution introduces new challenges to Grid workflow scheduling. Not only is a scheduling algorithm that is able to manage multiple workflow instances required, but this algorithm also needs the ability to schedule tasks across multiple workflow instances judiciously, as tasks may require the same set of Grid resources. Without appropriate resource allocation, resource competition problem could arise. In the thesis, we propose a new Grid workflow scheduling technique for parameter sweep workflow called the Besom scheduling algorithm. The scheduling decision of our algorithm is based on the resource dependencies of tasks in the workflow, as well as conventional Grid resource-performance metrics. In addition, the proposed technique is extended to handle loop structures in scientific workflows without using existing loop-unrolling techniques. We evaluate the Besom algorithm under a variety of conditions. A comparison between the simulation results of the Besom algorithm and of the three existing Grid workflow scheduling algorithms shows that the Besom algorithm is able to perform better than the existing algorithms for workflows that have complex structures and that involve overlapping resource dependencies of tasks. The Besom scheduling algorithm advances the ability to schedule parallel execution of parameter sweep workflows in the Grid. The outcomes of this thesis justify continuing research in this area to increase our understanding of scheduling multiple Grid workflow instances and to provide support to those involved in parametric study and scientific workflow management.

Grid computing technology is able to integrate and share large-scale distributed computation and data resource to facilitate the scientific researches. Recently, the grid workflow support and large-scale distributed data management are becoming two main requirements of scientists and researchers in many fields, such as bioinformatics, high-energy physics etc. In this paper, we proposed to support grid workflow for data intensive applications using CSF4 scheduling plug-ins. The grid workflow scheduling and data aware scheduling policies are implemented in two scheduling plug-ins, grid workflow plug-in and grid data aware plug-in, respectively. The two scheduling plug-ins can work together smoothly. The data aware plug-in will automatically dispatch the workflow tasks to the grid sites which are close to data replicas. At last, the experiment results are given to show the improvement of system performance and optimization of scheduling.

A Scheduler based on Resource Competition for Parameter Sweep Workflow

For the grid workflow system include a large of candidate services, the activity scheduling in workflow’s ex-ecution base on quality of service (QoS) constraints can pro-vide optimal service for users. This paper established the QoS’s parameter system with applied technology of grid workflow, and introduced the each parameter’s calculation model of activity’s QoS, the grid work-flow’s QoS estimation method based on the basic flow control structures defined by the abstract grid workflow language (AGWL). The experimental results show that the calculation method in this paper can estimate stably QoS parameter values of grid workflow combining with the multi candidate services, and, can provide a reasonable basis for process scheduling based on QoS constraints.

Grid computing platforms provide middleware and services for coordinating the use of data and computational resources available throughout the network. Grids are used to implement a wide range of distributed applications and systems, including frameworks for distributed data mining and knowledge discovery. This chapter presents a framework we developed to support the execution of knowledge discovery workflows in Grid environments by executing data mining and computation intelligence algorithms on a set of Grid nodes. Our framework is an extension of Weka, an open-source toolkit for data mining and knowledge discovery, and makes use of Web Service technologies to access Grid resources and distribute the computation. We present the implementation of the framework and show through some applications how it supports the design of knowledge discovery workflows and their execution on a Grid.

In grid workflow systems, temporal correctness is critical to assure the timely completion of grid workflow execution. To monitor and control the temporal correctness, fixed-time constraints are often assigned to a grid workflow and then verified. A checkpoint selection strategy is used to select checkpoints along grid workflow execution for verifying fixed-time constraints. The problem of existing representative strategies is that they do not differentiate fixed-time constraints as once a checkpoint is selected, they verify all fixed-time constraints. However, these checkpoints do not need to be taken for those constraints whose consistency can be deduced from others. The corresponding verification of such constraints is consequently unnecessary and can severely impact the efficiency of overall temporal verification. To address the problem, in this paper, we develop a new temporal dependency based checkpoint selection strategy which can select checkpoints according to different fixed-time constraints. With our strategy, the corresponding unnecessary verification can be avoided. The comparison and experimental simulation further demonstrate that our new strategy can improve the efficiency of overall temporal verification significantly over the existing representative strategies.

The computational grid provides the global computing infrastructure for users to access the services over a network. However, grid service providers charge users for the services based on their usage and QoS level specified. Therefore, in order to optimize the grid workflow execution, a robust multi-objective scheduling algorithm is needed considering economic cost along with execution performance. Generally, in multi-objective problems, simulations rely on running large number of evaluations to obtain the accurate results. However, algorithms that consider the preferences of decision maker, convergence to optimal tradeoff solutions is faster. Thus, in this chapter, the author proposed the preference-based guided search mechanism into MOEAs. To obtain solutions near the pre-specified regions of interest, the author has considered two MOEAs, namely R-NSGA-II and R-ε-MOEA. Further, to improve the diversity of solutions, a modified form called M-R-NSGA-II is used. Finally, the experimental settings and performance metrics are presented for the evaluation of the algorithms.

In the recent years, many efforts have been made for the development of the Grid workflow systems. Currently most Grid workflow systems focus on data analysis function or specific Grid applications, while building large-scale scientific collaborative applications is still not easy for common users. In this paper, requirement of Grid workflow spanning multiple grids was introduced firstly. Then, a highly scalable, more usable Grid workflow framework is proposed. The framework provides abstract components that facilitate Grid-based workflow construction, and can integrate multiple Grids' resources into a single system with an intuitive graphical user interface. Furthermore, a prototype system based on this framework was developed, along with a simple bioinformatics application workflow as a demonstration.

Grid workflow applications are emerging as an important new alternative to develop truly distributed applications for the grid platform. The modeling of grid workflow system is a challenging task, due to the high degree of autonomy and heterogeneity of the cooperative organizations under distributed environments. This paper presents a formal specification methodology for grid workflow modeling which is founded upon the Pi-Calculus process algebra. This method works well for characterizing the behaviors and interactions of the grid workflow processes that belong to different organizations in terms of the semantics of Pi-Calculus.

Many important scientific and engineering problems may be solved by combining multiple applications in the form of a Grid workflow. We consider that for the wide acceptance of Grid technology it is important that the user has the possibility to express requirements on Quality of Service (QoS) at workflow specification time. However, most of the existing workflow languages lack constructs for QoS specification. In this paper we present an approach for high level workflow specification that considers a comprehensive set of QoS requirements. Besides performance related QoS, it includes economical, legal and security aspects. For instance, for security or legal reasons the user may express the location affinity regarding Grid resources on which certain workflow tasks may be executed. Our QoS-aware workflow system provides support for the whole workflow life cycle from specification to execution. Workflow is specified graphically, in an intuitive manner, based on a standard visual modeling language. A set of QoS-aware service-oriented components is provided for workflow planning to support automatic constraint-based service negotiation and workflow optimization. For reducing the complexity of workflow planning, we introduce a QoS-aware workflow reduction technique. We illustrate our approach with a real-world workflow for maxillo facial surgery simulation.

Workflow management is one of the most important parts of ShanghaiGrid, which has a complex service-oriented infrastructure. Because grid environment is very dynamic and services in it are shared among many users, it is impossible to optimise grid workflow from the point of view of execution ahead of time. In fact, one may want to make decisions about the execution locations and the access to a particular data set as late as possible, which is the main objective of grid workflow mapping. In this paper, three types of workflow are denoted as abstract workflow (AW), concrete workflow (CW) and executable workflow (EW). We propose a method of using Belief-desire-intention (BDI) agents to implement service-oriented workflow mapping in the agent-based grid workflow management system (AGWMS) of ShanghaiGrid. BDI agent technology helps the system meet challenges from the grid context. Contract net protocol (CNP) provides a very proper negotiation model for these agents. The problem of workflow mapping has been transferred to the problem of multi-agent negotiation with the help of CNP model in AGWMS. AW2CW and CW2EW mapping algorithms are also given to accomplish service-oriented workflow mapping.

An optimal grid scheduling method considering coupling degree of nodes and duality of reciprocal inverse mapping

Manufacturing grid (MG) workflow can be defined as the composition of manufacturing activities which execute on heterogeneous and distributed manufacturing resources in virtual organization to accomplish a specific goal. Uncertainties within MG environment pose new challenges for MG workflow. Dynamic scheduling is one of the most critical components in MG workflow. Through analyzing the QoS properties in MG, a scheduling architecture based on QoS for MG workflow is presented. Workflow engine can dynamically schedule the resources according the activity requirements and resource QoS capabilities. If multiple resources that meet the QoS requirements are available to an activity at a time, workflow scheduling can select the 'best match' resource according to scheduling algorithms based on AHP (analytic hierarchy process).

New technological advancements in synchrotron light sources enable data acquisitions at unprecedented levels. This emergent trend affects not only the size of the generated data but also the need for larger computational resources. Although beamline scientists and users have access to local computational resources, these are typically limited and can result in extended execution times. Applications that are based on iterative processing as in tomographic reconstruction methods require high-performance compute clusters for timely analysis of data. Here, time-sensitive analysis and processing of Advanced Photon Source data on geographically distributed resources are focused on. Two main challenges are considered: (i) modeling of the performance of tomographic reconstruction workflows and (ii) transparent execution of these workflows on distributed resources. For the former, three main stages are considered: (i) data transfer between storage and computational resources, (i) wait/queue time of reconstruction jobs at compute resources, and (iii) computation of reconstruction tasks. These performance models allow evaluation and estimation of the execution time of any given iterative tomographic reconstruction workflow that runs on geographically distributed resources. For the latter challenge, a workflow management system is built, which can automate the execution of workflows and minimize the user interaction with the underlying infrastructure. The system utilizes Globus to perform secure and efficient data transfer operations. The proposed models and the workflow management system are evaluated by using three high-performance computing and two storage resources, all of which are geographically distributed. Workflows were created with different computational requirements using two compute-intensive tomographic reconstruction algorithms. Experimental evaluation shows that the proposed models and system can be used for selecting the optimum resources, which in turn can provide up to 3.13× speedup (on experimented resources). Moreover, the error rates of the models range between 2.1 and 23.3% (considering workflow execution times), where the accuracy of the model estimations increases with higher computational demands in reconstruction tasks.