An Adaptive Scheduler Framework for Complex Workflow Jobs on Grid Systems

The aim of this thesis is to design and implement a new Grid Resource Management methodology, where non-massive resources owners would be able to share their resources and integrate human collaboration across multiple domains regardless of network technology, operative platform or administrative domain.<br/>This thesis proposes a distributed and heuristic policy-based resource management architecture for large-scale Grids. The resource management architecture proposed herein is composed of four main building blocs: services management, resource discovery and monitoring, resource scheduling and jobs allocation and activation. The Grid Services Management (GSM) and Jobs Allocation and Activation (JAA) are supported by means of a Policy-based Grid Resource Management Architecture (PbGRMA). This architecture is able to identify service needs arising from diverse sources during the deployment and management of Grid Services, such as requirements demanded by customers, applications and network conditions. Afterwards, the PbGRMA merges these requirements into deployment policies for the corresponding Grid Services. The Grid Resource Discovery and Monitoring (GRDM) is supported by the introduction of the SNMP-based Balanced Load Monitoring Agents for Resource Scheduling (SBLOMARS), in which network and computational resources are monitored by distributed agents. This allows for a flexible, heterogeneous and scalable monitoring system. The Grid Resource Scheduling (GRS) is based on the Balanced Load Multi-Constrained Resource Scheduler (BLOMERS). This heuristic scheduler represents an alternate way of solving the inherent NP-hard problem for resource scheduling in large-scale distributed networks by means of the implementation of a Genetic Algorithm.<br/>Finally, based on the outcome of both the GRDM and GRS, the PbGRMA allocates the corresponding Grid Services by means of its interfaces with Globus ToolKit Middleware and Unix-based CLI commands along of any large-scale Grid Infrastructure. <br/>The synergy obtained by these components allows Grid administrators to exploit the available resources with predetermined levels of Quality of Service (QoS), reducing computational costs and makespan in resource scheduling while ensuring that the resource load is balanced throughout the Grid. The makespan of a schedule is the time required for all jobs to be processed when no one job could be interrupted during its execution and each node can perform at most one operation at any time.<br/>This new approach has been successfully tested in a real large-scale scenario such as Grid5000. The results presented along this Thesis show that our general solution is a reliable, flexible and scalable architecture to deploy and manage Grid Services in large-scale Grid Infrastructures. Moreover, the substitution of the heuristic algorithm approach used into the Grid Resource Scheduling (GRS) phase by other non-heuristics selection algorithms could make our solution useful in smaller Grid Infrastructures.

Job scheduling in grid computing is a very important problem. To utilize grids efficiently, we need a good job scheduling algorithm to assign jobs to resources in grids. The main scope of this article is to propose a new Ant Colony Optimization (ACO) algorithm for balanced job scheduling in the Grid environment. To achieve the above goal, we will indicate a way to balance the entire system load while minimizing the makespan of a given set of jobs. Based on the experimental results, the proposed algorithm confidently demonstrates its practicability and competitiveness compared with other job scheduling algorithms.

Ad hoc network is an infra structure less network, which is formed by mobile devices like laptops, PDAs, cell phones etc. Each device has different computational capability, power, hardware and software, which forms a heterogeneous network. These devices can be integrated to form an infrastructure known as grid. A grid integrates and coordinates resources and users that are within the same network with different capabilities. Hence we can visualize a grid over an ad hoc network that effectively utilizes the heterogeneity in the mobile devices. The major challenge in forming a grid over an ad hoc network is the mobility of the nodes. In this paper, we address the challenges due to mobility by considering a trace model for the movement of the nodes. Next, we demonstrate the feasibility of forming a grid over a mobile ad hoc network by proposing lightweight algorithms for grid formation, resource discovery, negotiation, job scheduling, and resource sharing. We have analyzed the performance of mobile ad hoc grid both by using a theoretical model and by simulation. The results point to a promising approach to form a mobile ad hoc grid.

\n\t\t\t\t\tThis paper addresses the problem of resource scheduling in a grid computing environment. One of the main goals of grid computing is to share system resources among geographically dispersed users, and schedule resource requests in an efficient manner. Grid computing resources are distributed, heterogeneous, dynamic, and autonomous, which makes resource scheduling a complex problem. This paper proposes a new approach to resource scheduling in grid computing environments, the hierarchical stochastic Petri net (HSPN). The HSPN optimizes grid resource sharing, by categorizing resource requests in three layers, where each layer has special functions for receiving subtasks from, and delivering data to, the layer above or below. We compare the HSPN performance with the Min-min and Max-min resource scheduling algorithms. Our results show that the HSPN performs better than Max-min, but slightly underperforms Min-min.\n\t\t\t\t

Improving the grid resource matching performance under large-scale users and computing resources environment is an important and challenging problem. Many existing grid resource matching and scheduling algorithms need search and compare every grid computing resource node without considering characteristics of grid resource and users' behavior. At the same time, with service oriented grid development, grid system should provide personalized service in scheduling process. Therefore, this paper proposes a novel grid resource matching algorithm based improved collaborative filtering algorithm that can give user recommendation resource set by computing the best approximate resource based on user's behavior.

One of the main challenges in Grid systems is designing an adaptive, scalable, and model-independent method for job scheduling to achieve a desirable degree of load balancing and system efficiency. Centralized job scheduling methods have some drawbacks, such as single point of failure and lack of scalability. Moreover, decentralized methods require a coordination mechanism with limited communications. In this paper, we propose a multi-agent approach to job scheduling in Grid, named Centralized Learning Distributed Scheduling (CLDS), by utilizing the reinforcement learning framework. The CLDS is a model free approach that uses the information of jobs and their completion time to estimate the efficiency of resources. In this method, there are a learner agent and several scheduler agents that perform the task of learning and job scheduling with the use of a coordination strategy that maintains the communication cost at a limited level. We evaluated the efficiency of the CLDS method by designing and performing a set of experiments on a simulated Grid system under different system scales and loads. The results show that the CLDS can effectively balance the load of system even in large scale and heavy loaded Grids, while maintains its adaptive performance and scalability.

Grid resource management and scheduling are confronted with a great challenge. This paper presented a dynamic resource price-adjusting (RPA) strategy in computational Grid. Three new concepts and two new evaluation standards are defined. On the basis of these theories, a Grid resource scheduling algorithm based on marker-driven using dynamic RPA strategy is proposed. With the help of GridSim toolkit, the simulation of resource scheduling algorithm proposed in the paper is realized, and the new algorithm is compared with GridSim DBC (deadline and budget constrained) scheduling algorithms. The experimental results show that the new algorithm improves the performance of Grid in aspect of increasing the task accomplishment ratio and solving the problem of load imbalance.

Grid resource providers can use gossiping to disseminate their available resource state to remote regions of the grid to attract application load. Pairwise gossiping protocols exchange information about limited subsets of other resources between pairs of potentially remote participants. In epidemic gossiping protocols, the provider disseminates information to multiple neighbors, who in turn forward it to their neighbors, and so on. One important metric for these protocols is their coverage, which characterizes how many and which resources receive the information. Coverage characteristics of epidemic protocols are non-uniform, concentrated within the vicinity of a disseminating node; they can exhibit bi-modal behavior where information either reaches distant nodes or dies out quickly. Pairwise gossiping protocols, on the other hand, provide a more uniform coverage, but it can take longer for the dissemination to reach desired uniformity. In this paper, we study performance characteristics of three gossiping protocols: (1) epidemic gossiping, (2) pairwise gossiping, and (3) adaptive information dissemination (which is based on a form of epidemic gossiping). We report experimental results based on our simulation framework that compare the three protocols in terms of packet overhead and query satisfaction rates. We show that pairwise gossiping protocols work best when resource distribution on the grid is uniform, but that they can be configured to perform well in support of grid scheduling.

Abstract—Grid computing is characterized by the existence of a collection of heterogeneous geographicallydistributed resources that are connected over high speed networks.Job scheduling and resource management have been a great challenge to researchers in the area of grid computing.Very often, there are applications having alarge number of fine-grainedjobs.Sending these finegrained jobsindividually to be executed on grid resources that have high processing power reduces resource utilization and is thus uneconomical. This paper presents efficient grouping-based scheduling models that group fine-grained jobs to form coarsegrained jobs which are sent for execution on grid resources. Our groupingstrategy is based on the processing capability of resources and the processing requirements of grouped jobs.A load balancing approach is also presented to achieve efficient utilization of resources. Simulation experiments were conducted using the Gridsim toolkit. Results show that thetotal simulation time and the cost are improved by grouping.Furthermore, our load balancing approach enhances resource utilization and achieves load balancing among resources. Keywords—grid computing;job scheduling; job grouping; load balancing; resource utilization; Gridsim I.

The paper is targeted at concerning pricing approach for grid resource scheduling with QoS guarantees. The two major parties in Grid, namely, resource consumers who submit various applications, and resources providers who share their resources, usually have different motivations when they join the Grid. These incentives are presented by objective functions in scheduling. Economic grid QoS scheduling algorithm adopt both application-centric and resource-centric scheduling objective function aim to optimize the performance of each individual grid user and the performance of the grid resources. The economic cost and profit are considered by grid users and resource providers respectively. Objective functions and QoS scheduling optimization algorithms are proposed. QoS scheduling optimization is decomposed into two sub optimizations of grid user's cost and grid resource provider's profit by utilizing dynamic programming. QoS scheduling algorithm is an interactive process in economic market coordinating grid users and grid resource providers to achieve optimal QoS scheduling. Experiments are to compare the proposed economic QoS scheduling with three partial optimisation approaches, which only consider one criterion.

Job scheduling is a fundamental issue in achieving high performance on the Grids. In this paper, a hierarchical job scheduling architecture has been proposed for the Grids. The stochastic high-level Petri net (SHLPN) model of the proposed architecture and the refined model are presented. The job scheduling algorithms for an incoming job to be distributed to a waiting queue of any processor in any node of the Grid architecture is integrated by the node-selection algorithm at the global scheduler, the job-dispatching algorithm at the local scheduler, and job-scheduling algorithm at the processor scheduler. The integrated job scheduling policy is investigated using SHLPN performance analysis methods. The performance metrics concerned include the delay time of jobs in the Grid node and the response time perceived by the users. The numerical example shows the efficiency of the performance evaluation technique.

Grid Computing is a widespread distributed computing environment that consists of heterogeneous resources and it provides huge computational power for the large scale distributed application. In this job scheduling is an essential objective to prioritize the jobs based on the job characteristics. It is a big challenge to design and implement an efficient Job Scheduling algorithm and the Job Scheduling model. This paper proposes the priority based Job Scheduling algorithm with Nonpreemptive Priority (NPRP) Job Scheduling model. The priority based job scheduling algorithm is the combination of Backfill algorithm and Earliest Deadline based algorithm. The proposed model has been integrated with Care Resource Broker (CRB) which supports virtualized grid environment. The proposed model has been integrated with CRB helps to make the intelligent scheduling decisions for prioritizing the jobs and also the proposed model minimizes the turnaround time, waiting time and response time of the jobs available in the queue and the user's quality of service has been maximized in the heterogenous and virtualized grid environment.

Aiming at realizing a high performance computing (HPC) cloud with vector supercomputers, this paper presents the world’s first prototype of wide-area vector meta computing infrastructure named a vector computing cloud by virtualizing remote vector computing resources as an HPC service over the Internet. The prototype system consists of two remote NEC SX-9 nodes connected through a long fat-pipe network (LFN), and each node is located at Tohoku University and Osaka University with a distance of 800 km. The vector computing cloud also provides a single sign on environment and jobs are automatically assigned to appropriate sites. Wide-area co-operation of distributed vector supercomputers is realized by adopting the NAREGI Grid Middleware, and a virtual machine for NEC SX vector supercomputer series, job scheduling algorithms, and an MPI operating environment are newly developed to enhance the job and resource management capabilities of the NAREGI Grid Middleware. In addition, to achieve fairness and efficient job scheduling on the vector computing cloud, this paper presents a history-based job scheduling mechanism for a queue system. Based on the estimation, the job scheduling mechanism automatically allocates the job to an appropriate site, which can execute the job earlier. The operation tests and experiment results indicate that the prototype system realizes single sign on for multiple vector resources and has enough potential for transparently operating jobs between the two SX-9 systems. This paper also evaluates and discusses the performance of the proposed job scheduling mechanism and MPI operation between both SX-9 systems using the High Performance Linpack (HPL) benchmark.

The two major problems in grid computing applications are, resource management and job scheduling. These problems do occur due to distributed and heterogeneous nature of the resources. This paper introduces a model in job scheduling in grid computing environments. A dynamic scheduling algorithm is proposed to maximize the resource utilization and minimize processing time of the jobs. The proposed algorithm is based on job grouping. The results show that the proposed scheduling algorithm efficiently reduces the processing time of jobs.

Distributed job scheduling based on Swarm Intelligence: A survey