A Scheduling Middleware for Data Intensive Applications on a Grid

We present some theoretical and experimental results of an important caching problem which arises frequently in data intensive scientific applications that are run in data-grids. Such applications often need to process several files simultaneously, i.e., the application runs only if all its needed files are present in some disk cache accessible to the compute resource of the application. The set of files requested by an application, all of which must be in cache for the application to run, is called a file-bundle. This requirement introduces the need for cache replacement algorithms that are based on file-bundles rather then individual files. We show that traditional caching algorithms such as Least Recently Used (LRU) and GreedyDual-Size (GDS) are not optimal in this case since they are not sensitive to file-bundles and may hold in the cache non-relevant combinations of files. We propose and analyze a new cache replacement algorithm specifically adapted to deal with file-bundles. Results of experimental studies of the new algorithm, using a disk cache simulation model under a wide range of conditions such as file request distributions, relative cache size, file size distribution, and incoming job queue size, show significant improvement over traditional caching algorithms such as GDS.KeywordsCache SizeCache StrategyCache ReplacementZipf DistributionData IntensiveThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

High level QoS-driven model for Grid applications in a simulated environment

As grid applications evolve, the need for user controlled network infrastructure is apparent in order to support emerging dynamic and interactive services. Due to the inherent bandwidth offered by optical wavelength division multiplexed (WDM) networks, they prove to be a potential candidate to support the bandwidth intensive grid applications. In a grid environment, the available resources can be classified into two broad categories: grid resources which consist of computing and storage components that reside on each node of the network and network resources which provide bandwidth for the execution of a grid application. A typical grid application (job) is usually divided into a number of smaller tasks which need to be scheduled, on possibly different nodes of the network in order to ensure job completion. There usually exists some dependency between these tasks and a strict time-deadline within which the job needs to be completed. Rather than using an independent scheduling approach (at the grid and network levels), we address the co-scheduling problem in lambda-grids for advance reservation requests and aim at minimizing the job blocking probability. We use the anycasting communication paradigm referred to as co-anycasting, to allocate grid and network resources to all tasks of a job. We propose three heuristics: first free (FF), shortest hop (SH), and least used (LU) to solve the co-scheduling problem. Moreover, we compare the proposed co-anycasting approach to a grid-anycasting approach, wherein the anycasting flexibility is offered only at the grid level, and show through extensive simulations the performance benefit of using co-anycasting to support grid applications in a time-deadline environment.

Proceedings Eleventh International Workshop on Research Issues in Data Engineering. Document Management for Data Intensive Business and Scientific Applications. RIDE 2001

One of the advantages of following a MDA-based approach in the development of interactive applications is the possibility of generating multiple platform-specific user interfaces (UI) from the same platform independent UI model. However, the effort required to create the UI model may be significant. In the case of data-intensive applications, a large part of the UI structure and functionality is closely related with the structure and functionality of the domain entities described in the domain model, and the access rules specified in the use case model. This paper presents an approach to reduce the effort required to create platform independent UI models for data intensive applications, by automatically generating an initial UI model from domain and use case models. For that purpose, UML-aligned metamodels for domain and use case models are defined, together with a MOF-based metamodel for user interface models. The transformation rules that drive the UI model generation are introduced. It is also proposed a MDA-based process for the development of data intensive interactive applications based on the proposed model architecture and transformations.

The aim of this thesis is to design and implement a new Grid Resource Management methodology, where non-massive resources owners would be able to share their resources and integrate human collaboration across multiple domains regardless of network technology, operative platform or administrative domain.<br/>This thesis proposes a distributed and heuristic policy-based resource management architecture for large-scale Grids. The resource management architecture proposed herein is composed of four main building blocs: services management, resource discovery and monitoring, resource scheduling and jobs allocation and activation. The Grid Services Management (GSM) and Jobs Allocation and Activation (JAA) are supported by means of a Policy-based Grid Resource Management Architecture (PbGRMA). This architecture is able to identify service needs arising from diverse sources during the deployment and management of Grid Services, such as requirements demanded by customers, applications and network conditions. Afterwards, the PbGRMA merges these requirements into deployment policies for the corresponding Grid Services. The Grid Resource Discovery and Monitoring (GRDM) is supported by the introduction of the SNMP-based Balanced Load Monitoring Agents for Resource Scheduling (SBLOMARS), in which network and computational resources are monitored by distributed agents. This allows for a flexible, heterogeneous and scalable monitoring system. The Grid Resource Scheduling (GRS) is based on the Balanced Load Multi-Constrained Resource Scheduler (BLOMERS). This heuristic scheduler represents an alternate way of solving the inherent NP-hard problem for resource scheduling in large-scale distributed networks by means of the implementation of a Genetic Algorithm.<br/>Finally, based on the outcome of both the GRDM and GRS, the PbGRMA allocates the corresponding Grid Services by means of its interfaces with Globus ToolKit Middleware and Unix-based CLI commands along of any large-scale Grid Infrastructure. <br/>The synergy obtained by these components allows Grid administrators to exploit the available resources with predetermined levels of Quality of Service (QoS), reducing computational costs and makespan in resource scheduling while ensuring that the resource load is balanced throughout the Grid. The makespan of a schedule is the time required for all jobs to be processed when no one job could be interrupted during its execution and each node can perform at most one operation at any time.<br/>This new approach has been successfully tested in a real large-scale scenario such as Grid5000. The results presented along this Thesis show that our general solution is a reliable, flexible and scalable architecture to deploy and manage Grid Services in large-scale Grid Infrastructures. Moreover, the substitution of the heuristic algorithm approach used into the Grid Resource Scheduling (GRS) phase by other non-heuristics selection algorithms could make our solution useful in smaller Grid Infrastructures.

Self-healing in workflow is becoming prominent because of the increasing software complexity. In order to support robust workflow execution, in this paper, we propose a self-healing mechanism to replace the faulty service dynamically. It is based on our previous research work -workflow execution system (WES). The OWLS2BPEL Mapper in WES is further enhanced to embed the self-healing mechanism into BPEL workflow by exploiting Semantic Web service technology. In order to realize service substitution, business rules is utilized to model the essential business logic behind the service interface (local business rules) and the service selection policy of a composite service (global business rules) in a domain-dependent and semantic-rich manner. A concrete scenario in PC manufacturing collaborative virtual enterprise is presented to test our self-healing capable workflow execution system.

The advent of Cloud computing has paved the way to envision hybrid computational infrastructures based on powerful Grid resources combined with dynamic and elastic on-demand virtual infrastructures on top of Cloud deployments. However, the combination of Grid and Cloud resources for executing computationally intensive scientific applications introduces new challenges and opportunities in areas such as resource provisioning and management, meta-scheduling and elasticity. This paper describes different approaches to integrate the usage of Grid and Cloud-based resources for the execution of High Throughput Computing scientific applications. A reference architecture is proposed and the the opportunities and challenges of such hybrid computational scenarios are addressed. Finally, a prototype implementation is described and a case study that involves a protein design application is employed to outsource job executions to the Cloud when Grid resources become exhausted.

Data-intensive applications produce huge amounts of data that need to be stored, analyzed, and interpreted. A data grid serves as a cost-effective infrastructure for solving these data-intensive applications. Existing scheduling strategies are best suited for handling compute-intensive applications, although they lack in performance while handling data-intensive applications. In this work, a novel mechanism of incorporating cognitive science in a data grid is proposed for scheduling data-intensive workflows. A unique model is derived in which a cognitive engine (CE) is built into the middleware of the data grid. The intelligent agents present in the CE handle the request for data sets and use the LTP algorithm (learning, thinking, and perception) to effectively schedule the tasks using three phases. The CE also~finds a unique solution for placing data sets dynamically nearer to the execution site based on network resource considerations by reducing the~waiting time and data availability time for I/O-intensive jobs. The performance of the CE is validated by simulation and compared with that of existing scheduling strategies. The results of the simulation show that CE optimizes the data availability time, waiting time, data transfer time, and makespan.

New technological advancements in synchrotron light sources enable data acquisitions at unprecedented levels. This emergent trend affects not only the size of the generated data but also the need for larger computational resources. Although beamline scientists and users have access to local computational resources, these are typically limited and can result in extended execution times. Applications that are based on iterative processing as in tomographic reconstruction methods require high-performance compute clusters for timely analysis of data. Here, time-sensitive analysis and processing of Advanced Photon Source data on geographically distributed resources are focused on. Two main challenges are considered: (i) modeling of the performance of tomographic reconstruction workflows and (ii) transparent execution of these workflows on distributed resources. For the former, three main stages are considered: (i) data transfer between storage and computational resources, (i) wait/queue time of reconstruction jobs at compute resources, and (iii) computation of reconstruction tasks. These performance models allow evaluation and estimation of the execution time of any given iterative tomographic reconstruction workflow that runs on geographically distributed resources. For the latter challenge, a workflow management system is built, which can automate the execution of workflows and minimize the user interaction with the underlying infrastructure. The system utilizes Globus to perform secure and efficient data transfer operations. The proposed models and the workflow management system are evaluated by using three high-performance computing and two storage resources, all of which are geographically distributed. Workflows were created with different computational requirements using two compute-intensive tomographic reconstruction algorithms. Experimental evaluation shows that the proposed models and system can be used for selecting the optimum resources, which in turn can provide up to 3.13× speedup (on experimented resources). Moreover, the error rates of the models range between 2.1 and 23.3% (considering workflow execution times), where the accuracy of the model estimations increases with higher computational demands in reconstruction tasks.

Characterizing workflow-based activity on a production e-infrastructure using provenance data

Advances on multicore technologies lead to processors with tens and soon hundreds of cores in a single socket, resulting in an ever growing gap between computing power and available memory and I/O bandwidths for data handling. It would be beneficial if some of the computing power can be transformed into gains of I/O efficiency, thereby reducing this speed disparity between computing and I/O. In this paper, we design and implement a NEarline data COmpression and DECompression (neCODEC) scheme for data-intensive parallel applications. Several salient techniques are introduced in neCODEC, including asynchronous compression threads, elastic file representation, distributed metadata handling, and balanced subfile distribution. Our performance evaluation indicates that neCODEC can improve the performance of a variety of data-intensive microbenchmarks and scientific applications. Particularly, neCODEC is capable of increasing the effective bandwidth of S3D, a combustion simulation code, by more than 5 times.

BP plans world-scale PTA plant in Belgium

Grid services provide an important abstract layer on top of heterogeneous components (hardware and software) that take part into a grid environment. We are developing a data grid service prototype that aims at providing transparent use of grid resources to data intensive scientific applications. Our prototype was designed having as target three main issues: (1) dynamic scheduling and allocation of query execution engine modules into grid nodes; (2)adaptability of query execution to variations on environment conditions and (3) support to special scientific operations. We propose a new node scheduling algorithm and show how it can be integrated into a simple distributed and parallel query optimization strategy. Our implementation demonstrates a speedup of 16.6 with 18 scheduled nodes and a steady throughput rate, obtained applying a dynamic adaptive strategy.

Recently, distributed computing infrastructures such as grid computing, have witnessed huge developments which have a very important impact on computationally intensive scientific, industrial and engineering problems. Collaborative computing, resources access and sharing facilitated by grid computing systems amplifies concerns about cyber attacks and misbehaviors of grid resources (resources provider). In many cases, an organization may send out jobs for remote execution on untrustworthy machines. Since resource sharing and cooperation among different administrative domains are among the central goals of grid computing, it is necessary to guarantee that, among the shared resources, there are no malicious resources interested in invalidating or corrupting the job results. In order to guarantee reliable jobs execution in grid systems, we have developed a new approach for malicious grid resources tolerance using reputation to improve the efficiency of majority voting mechanisms. The grid broker service investigates the trustworthiness of the grid resources used in the voting procedure; this investigation is based on distributed and hierarchical reputation management system which maintains a reputation value for each grid resource based on its historical behavior. The validation of the result is then decided based on the grid resources reputation level.