Using multiple grid resources for bioinformatics applications in GADU

Grid Computing provides sharing of geographically distributed resources among large scale complex applications. Due to dynamic nature of resources in grid, there is a need of highly efficient job scheduling and resource management policies in grid. In this paper, a novel Grid Resource Scheduler (GRS) is proposed to effectively utilize the available resources in Grid. Proposed GRS contributes, an optimal job scheduling algorithm on Job Rank-Backfilling policy and a resource matching algorithm based on ranking of resources with best fit allocation model. Performance of GRS is measured by considering a web based BLAST algorithm -- a bioinformatics application -- . GRS aims in reducing; Make span of the job workflow and execution time of varied size jobs. The experimental results prove that proposed job scheduling and resource matching algorithms performs better when evaluated against widely used FCFS job scheduling and biggest free resource matching strategies.

This paper presents a mathematical model to evaluate the performance of grid resources when availability of the resources is taken into account. The proposed model uses continuous time Markov chains (CTMCs) to model the failure-repair behavior of a grid resource. In grid computing environment, a resource not only may fail during task execution, but also it can cancel its membership at any time. Hence, the proposed CTMC considers the availability of a grid resource to a grid user in both failure and membership refusal situations. After modeling the availability of the resource, the mean sojourn time of grid tasks in each of the availability states is estimated. Assigning the mean sojourn times of the tasks as performance levels to each of the CTMC's states, a Markov reward model (MRM) representing the combined performance and availability measures is obtained. Computing the cumulative state probability of the CTMC and multiplying reward rates of the MRM's states to each of the corresponding state probabilities, the expected accumulated sojourn time of grid tasks in each of the grid resources is achieved. An illustrative example is presented and the results obtained from the proposed model are reported in cases where various scheduling disciplines are considered inside the grid resource to simultaneously service grid and local tasks.

A Distributed Virtual Environment (DVE) system offers a computer-generated virtual world in which individuals located at different places in the physical world can interact with one another. In order to achieve real-time response for a large user base, DVE systems need to have a scalable architecture. In this paper, we present the design of a grid-enabled service oriented framework for facilitating the construction of scalable DVE systems on computing grids. A service component called "gamelet" is proposed, whose distinctive mark is its high mobility for supporting dynamic load sharing. We propose a gamelet migration protocol which can ensure the transparency and efficiency of gamelet migration, and an adaptive gamelet load-balancing (AGLB) algorithm for making gamelet redistribution decisions at runtime. The algorithm considers both the synchronization costs of the DVE system and network latencies inherent in the grid nodes. The activities of the users and the heterogeneity of grid resources are also considered in order to carry out load sharing more effectively. We evaluate the performance of the proposed mechanisms through a multiplayer online game prototype implemented using the Globus toolkit. The results show that our approach can achieve faster response times and higher throughputs than some existing approaches.

The computational grid provides a promising platform for the efficient execution of parallel coarse grain tasks computing over very large sample space. Scheduling such applications is challenging for the heterogeneity, autonomy, and dynamic adaptability of grid resources. Assuming resource owners have a preemptive priority, we propose an adaptive algorithm of jobs scheduling based on time balancing strategy, which solves the parallel computing tasks by using the idle resources in computational grid. A mathematical model is developed to predict performance, which also considers systems with heterogeneous machine utilization and heterogeneous service distribution. The model separates the influence of machine utilization, job service rate and parallel task allocation on the completion time. According to the time balancing policy, a task is partitioned into several subtasks and scheduled, and the costs of communication are reduced. The expected value of parallel task completion time is predicted with performance model. To get better parallel computing performance, an optimal subset of heterogeneous resources with the shortest parallel executing time of tasks can be selected with the efficient algorithm. Remapping strategy is applied during scheduling, which is more suitable for the dynamic adaptability and domain autonomy in the grid.

Grids are emerging as a promising solution for resource and computation demanding applications. However, the heterogeneity of resources in Grid computing, complicates resource management and scheduling of applications. In addition, the commercialization of the Grid requires policies that can take into account user requirements, and budget considerations in particular. This paper considers a basic model for workflow applications modelled as Directed Acyclic Graphs (DAGs) and investigates heuristics that allow to schedule the nodes of the DAG (or tasks of a workflow) onto resources in a way that satisfies a budget constraint and is still optimized for overall time. Two different approaches are implemented, evaluated and presented using four different types of basic DAGs.

Data analysis using grid resources is one of the fundamental challenges to be addressed before the start of LHC data taking. The ATLAS detector will produce petabytes of data per year, and roughly one thousand users will need to run physics analyses on this data. Appropriate user interfaces and helper applications have been made available to ensure that the grid resources can be used without requiring expertise in grid technology. These tools enlarge the number of grid users from a few production administrators to potentially all participating physicists. ATLAS makes use of three grid infrastructures for the distributed analysis: the EGEE sites, the Open Science Grid, and Nordu Grid. These grids are managed by the gLite workload management system, the PanDA workload management system, and ARC middleware; many sites can be accessed via both the gLite WMS and PanDA. Users can choose between two front-end tools to access the distributed resources. Ganga is a tool co-developed with LHCb to provide a common interface to the multitude of execution backends (local, batch, and grid). The PanDA workload management system provides a set of utilities called PanDA Client; with these tools users can easily submit Athena analysis jobs to the PanDA-managed resources. Distributed data is managed by Don Quixote 2, a system developed by ATLAS; DQ2 is used to replicate datasets according to the data distribution policies and maintains a central catalog of file locations. The operation of the grid resources is continually monitored by the Ganga Robot functional testing system, and infrequent site stress tests are performed using the Hammer Cloud system. In addition, the DAST shift team is a group of power users who take shifts to provide distributed analysis user support; this team has effectively relieved the burden of support from the developers.

The Open Science Grid (OSG) is a large, robust computing grid that started primarily as a collection of sites associated with large HEP experiments such as ATLAS, CDF, CMS, and DZero, but has evolved in recent years to a much larger user and resource platform. In addition to meeting the US LHC community’s computational needs, the OSG continues to be one of the largest providers of distributed high-throughput computing (DHTC) to researchers from a wide variety of disciplines via the OSG Open Facility. The Open Facility consists of OSG resources that are available opportunistically to users other than resource owners and their collaborators. In the past two years, the Open Facility has doubled its annual throughput to over 200 million wall hours. More than half of these resources are used by over 100 individual researchers from over 60 institutions in fields such as biology, medicine, math, economics, and many others. Over 10% of these individual users utilized in excess of 1 million computational hours each in the past year. The largest source of these cycles is temporary unused capacity at institutions affiliated with US LHC computational sites. An increasing fraction, however, comes from university HPC clusters and large national infrastructure supercomputers offering unused capacity. Such expansions have allowed the OSG to provide ample computational resources to both individual researchers and small groups as well as sizable international science collaborations such as LIGO, AMS, IceCube, and sPHENIX. Opening up access to the Fermilab FabrIc for Frontier Experiments (FIFE) project has also allowed experiments such as mu2e and NOvA to make substantial use of Open Facility resources, the former with over 40 million wall hours in a year. We present how this expansion was accomplished as well as future plans for keeping the OSG Open Facility at the forefront of enabling scientific research by way of DHTC.

High level QoS-driven model for Grid applications in a simulated environment

We study and develop a kernel of tools that allow MetaSchedulers to take advantage of a consistent set of shared information across Resource Management Systems, in both local and/or Grid scheduler systems. We demonstrate the usefulness of such tools within the MonALISA monitoring framework and the STAR Unified Meta-Scheduler. We will present and define the requirements and schema by which one can consistently provide queue attributes for the most common batch systems. We evaluate the best scalable and lightweight approach to access the monitored parameters from a client perspective and, in particular, the feasibility of accessing real-time and aggregate information. Client programs are envisioned to function in a non-centralized, fault tolerant fashion. Inherent delays as well as scalability issues of each approach (implementing it at a large number of sites) will be discussed. We believe that such developments could highly benefit Grid laboratory efforts such as the Grid3+ and the OpenScience Grid (OSG).

The advent of Cloud computing has paved the way to envision hybrid computational infrastructures based on powerful Grid resources combined with dynamic and elastic on-demand virtual infrastructures on top of Cloud deployments. However, the combination of Grid and Cloud resources for executing computationally intensive scientific applications introduces new challenges and opportunities in areas such as resource provisioning and management, meta-scheduling and elasticity. This paper describes different approaches to integrate the usage of Grid and Cloud-based resources for the execution of High Throughput Computing scientific applications. A reference architecture is proposed and the the opportunities and challenges of such hybrid computational scenarios are addressed. Finally, a prototype implementation is described and a case study that involves a protein design application is employed to outsource job executions to the Cloud when Grid resources become exhausted.

Grid Computing is a newly emerging technology that enables users to share a large number of computing resources distributed over a network. Due to that the grid computational resources are not storable, the advance reservation of these resources are necessary for users to request resources from multiple scheduling systems at a specific time. In this paper, we use auction-based scheduling method for resource reservation. We propose a variant version of traditional ascending auction for automatic resource reservation and introduce two novel heuristic strategies to guide agents on participating actions. We also compare our bidding strategies with several other different bidding strategies for the computational resource reservation in different scenarios. The results of experiments show that our heuristic bidding strategies outperforms those methods in those cases.

The data grid is one of the infrastructures for data and storage resource management. Data replication can improve data access performance, fault tolerance, and reduce network transmission bandwidth. As the site storage space is limited in the grid, how to effectively use grid resources become an important challenge. This paper introduces a dynamic grid replication algorithm based on popularity support and confidence (BPSC). Through the algorithm, data and its associated copy can be placed on a suitable site, with reduced access latency. Through Optorsim, simulation results show that the algorithm can provide better performance compared with other algorithms Grid computing is an important branch of distributed computing. Grid computing constructs a virtual computer system through using a different location of the large number of heterogeneous resources (CPU and storage), provides the model to solve the problem of large scale computing and storage. The data grid is mainly the grid for data storage management. The data grid is defined as an effective combination of data and computing resources of distributed systems by Data Grid Group. The large-scale data- intensive applications, such as bioinformatics, high energy physics, space information generated vast amounts of data, these data may be TB scale, even up to the scale of PB. It is not practical to maintain large scale data at each site, and provide computing resources for visitors. At the same time, it will lead to increased access latency through a centralized way to storing and accessing these data from each site. Data replication technology is an effective method to reduce access latency in the data grid system. Data replication is one of the basic methods of distributed systems to improve the efficiency of system access, maintenance, system high availability. Data replication is to store data in the appropriate site for users to effective access to massive data. Data replication can reduce data access latency, reduce network bandwidth consumption, avoid congestion, adjust the load balancing in server-side, improve system reliability. Many data replication algorithm can reduce data access latency, improve system reliability, but which data will be copied, when the data will be copied at, where the data will be copied are the basic means of determining data replication algorithm. In this paper, we introduced a real-time dynamic data replication algorithm. The algorithm is based on the popularity of file dynamic to replicate data, and consider the storage space limitations of the site. Through optorsim simulation verified the algorithm. This paper is organized as follows: Section 2 is about related works on replication in data grids. In Section 3, system model and component in system are presented. The file

We have designed and deployed the New York State Grid (NYS Grid), which consists of an integrated computational and data grid. NYS Grid is used in a ubiquitous fashion, where the users have virtual access to their data sets and applications, allowing the user to perform tasks without knowledge of the physical hosts for data storage or compute systems. A wide variety of applications have been ported to NYS Grid, including critical programs in a variety of fields that are ideally suited to a multiprocessor computing environment with distributed datasets. Two applications from structural biology are presented as exemplars, including our Grid portal version of the SnB program, which has been run simultaneously on all computational resources on NYS Grid, as well as on the majority of the tens of thousands of processors available through the Open Science Grid. This paper also discusses previous grids that we developed, including the Buffalo-based (ACDC) experimental grid and the Western New York Grid, as well as a wide variety of advances that we have made in terms of grid monitoring, predictive scheduling, grid portal design, and grid-enabling application templates, to name a few.

Grids represent an emerging technology that allows geographically- and organizationally-distributed resources (e.g., compute systems, data repositories, sensors, imaging systems, and so forth) to be linked in a fashion that is transparent to the user. The New York State Grid (NYS Grid) is an integrated computational and data grid that provides access to a wide variety of resources to users from around the world. NYS Grid can be accessed via a Web portal, where the users have access to their data sets and applications, but do not need to be made aware of the details of the data storage or computational devices that are specifically employed in solving their problems. Grid-enabled versions of the SnB and BnP programs, which implement the Shake-and-Bake method of molecular structure (SnB) and substructure (BnP) determination, respectively, have been deployed on NYS Grid. Further, through the Grid Portal, SnB has been run simultaneously on all computational resources on NYS Grid as well as on more than 1100 of the over 3000 processors available through the Open Science Grid.

It is common at research institutions to maintain multiple clusters that represent different owners or generations of hardware, or that fulfill different needs and policies. Many of these clusters are consistently under utilized while researchers on campus could greatly benefit from these unused capabilities. By leveraging principles from the Open Science Grid it is now possible to utilize these resources by forming a lightweight campus grid. The campus grids framework enables jobs that are submitted to one cluster to overflow, when necessary, to other clusters within the campus using whatever authentication mechanisms are available on campus. This framework is currently being used on several campuses to run HEP and other science jobs. Further, the framework has in some cases been expanded beyond the campus boundary by bridging campus grids into a regional grid, and can even be used to integrate resources from a national cyberinfrastructure such as the Open Science Grid. This paper will highlight 18 months of operational experiences creating campus grids in the US, and the different campus configurations that have successfully utilized the campus grid infrastructure.