Implement the Grid Workflow Scheduling for Data Intensive Applications with CSF4

Currently the research on the grid workflow scheduling algorithms commonly suffer by one or several of the following drawbacks: grid workflow model of directed acyclic graph (DAG), not enough considerations of multi-dimensional Quality of Service (QoS) parameters and single objective optimal scheduling. Based on abstract grid workflow language (AGWL) model, this paper presents an algorithm for multi-objective optimal grid workflow scheduling with QoS constraints, which make uses of an improved NSGA-II algorithm. Theoretical analysis and experimental results indicate the feasibility and efficiency of the algorithm.KeywordsQuality of service (QoS)grid workflow schedulingNSGA-II

Workflow technology has been adopted in scientific domains to orchestrate and automate scientific processes in order to facilitate experimentation. Such scientific workflows often involve large data sets and intensive computation that necessitate the use of the Grid, which offers supercomputing power through shared distributed resources. To execute a scientific workflow in the Grid, tasks within the workflow that represent steps in the scientific process are assigned to Grid resources for execution. To ensure efficient execution of the workflow, Grid workflow scheduling is required to manage the allocation of Grid resources. Although many Grid workflow scheduling techniques exist, they are mainly designed for the execution of a single workflow. This is not the case with parameter sweep workflows which focus on parametric study and parameter optimisation. A parameter sweep workflow is executed numerous times with different input parameters in order to determine the effect of each parameter combination on the experiment. While executing multiple instances of a parameter sweep workflow in parallel can reduce the time required for the overall execution, this execution introduces new challenges to Grid workflow scheduling. Not only is a scheduling algorithm that is able to manage multiple workflow instances required, but this algorithm also needs the ability to schedule tasks across multiple workflow instances judiciously, as tasks may require the same set of Grid resources. Without appropriate resource allocation, resource competition problem could arise. In the thesis, we propose a new Grid workflow scheduling technique for parameter sweep workflow called the Besom scheduling algorithm. The scheduling decision of our algorithm is based on the resource dependencies of tasks in the workflow, as well as conventional Grid resource-performance metrics. In addition, the proposed technique is extended to handle loop structures in scientific workflows without using existing loop-unrolling techniques. We evaluate the Besom algorithm under a variety of conditions. A comparison between the simulation results of the Besom algorithm and of the three existing Grid workflow scheduling algorithms shows that the Besom algorithm is able to perform better than the existing algorithms for workflows that have complex structures and that involve overlapping resource dependencies of tasks. The Besom scheduling algorithm advances the ability to schedule parallel execution of parameter sweep workflows in the Grid. The outcomes of this thesis justify continuing research in this area to increase our understanding of scheduling multiple Grid workflow instances and to provide support to those involved in parametric study and scientific workflow management.

Grid workflow systems provide mechanisms to execute complex tasks which consist of related sub tasks. Due to the intensive computing and data transferring in Grid workflows, the locations of tasks and data have great impact to the execution performance of Grid workflows. In this paper, we present a novel approach to search for optimal Grid workflow scheduling effectively. We model workflow execution with fetching input data and running tasks, and present a optimized scheduling searching algorithm based on simulated annealing, which can find neighborhood scheduling fast. The experimental results show that our approach is effective and scalable.KeywordsExecution TimeSimulated AnnealingGrid ResourceExecution PerformanceGrid TaskThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

A data replication algorithm for groups of files in data grids

A Scheduler based on Resource Competition for Parameter Sweep Workflow

The data grid is one of the infrastructures for data and storage resource management. Data replication can improve data access performance, fault tolerance, and reduce network transmission bandwidth. As the site storage space is limited in the grid, how to effectively use grid resources become an important challenge. This paper introduces a dynamic grid replication algorithm based on popularity support and confidence (BPSC). Through the algorithm, data and its associated copy can be placed on a suitable site, with reduced access latency. Through Optorsim, simulation results show that the algorithm can provide better performance compared with other algorithms Grid computing is an important branch of distributed computing. Grid computing constructs a virtual computer system through using a different location of the large number of heterogeneous resources (CPU and storage), provides the model to solve the problem of large scale computing and storage. The data grid is mainly the grid for data storage management. The data grid is defined as an effective combination of data and computing resources of distributed systems by Data Grid Group. The large-scale data- intensive applications, such as bioinformatics, high energy physics, space information generated vast amounts of data, these data may be TB scale, even up to the scale of PB. It is not practical to maintain large scale data at each site, and provide computing resources for visitors. At the same time, it will lead to increased access latency through a centralized way to storing and accessing these data from each site. Data replication technology is an effective method to reduce access latency in the data grid system. Data replication is one of the basic methods of distributed systems to improve the efficiency of system access, maintenance, system high availability. Data replication is to store data in the appropriate site for users to effective access to massive data. Data replication can reduce data access latency, reduce network bandwidth consumption, avoid congestion, adjust the load balancing in server-side, improve system reliability. Many data replication algorithm can reduce data access latency, improve system reliability, but which data will be copied, when the data will be copied at, where the data will be copied are the basic means of determining data replication algorithm. In this paper, we introduced a real-time dynamic data replication algorithm. The algorithm is based on the popularity of file dynamic to replicate data, and consider the storage space limitations of the site. Through optorsim simulation verified the algorithm. This paper is organized as follows: Section 2 is about related works on replication in data grids. In Section 3, system model and component in system are presented. The file

The data connection of electric power marketing and distribution network is an integrated platform of operation and distribution data based on geographic information platform, which integrates the data of marketing business application system, production management information system and grid geospatial information service platform, and shares the whole face of production data and marketing customer data, so as to achieve the connection of operation and distribution data and business, and realize fault location and electric failure Scope analysis, line loss statistics, business expansion and installation, etc. Graph database has the technical advantages of high efficiency of topological data organization, low redundancy of topological data storage, strong expansibility of topological data model and high efficiency of topological analysis and calculation, which can meet the storage, query and retrieval requirements of large-scale operation and distribution data. After application verification, the results show that the graph database can provide data storage, query and retrieval technology scheme for the operation and distribution of power grid. With the business development of operation and distribution of power grid, there are performance bottlenecks in the existing data storage mode, which seriously restricts the deepening of application. It is necessary to introduce advanced computing technology to achieve breakthrough, and improve the data storage efficiency of the existing data platform The parallel and concurrent ability of data retrieval and query can meet the management requirements of geometric growth of power grid equipment, the analysis requirements of real-time data of power grid, and the concurrent access requirements of largescale growth of users.

In recent years data grids have been deployed and grown in many scientific experiments and data centers. The deployment of such environments has allowed grid users to gain access to a large number of distributed data. Data replication is a key issue in a data grid and should be applied intelligently because it reduces data access time and bandwidth consumption for each grid site. Therefore this area will be very challenging as well as providing much scope for improvement. In this paper, we introduce a new dynamic data replication algorithm named Popular File Group Replication, PFGR which is based on three assumptions: first, users in a grid site (Virtual Organization) have similar interests in files and second, they have the temporal locality of file accesses and third, all files are read-only. Based on file access history and first assumption, PFGR builds a connectivity graph for a group of dependent files in each grid site and replicates the most popular group files to the requester grid site. After that, when a user of that grid site needs some files, they are available locally. The simulation results show that our algorithm increases performance by minimizing the mean job execution time and bandwidth consumption and avoids unnecessary replication.

Extracting minimal functional dependencies (MFDs) from relational databases is an import database analysis technique. With the advent of big data era, it is challenging to discover MFDs from big data, especially large-scale distributed data stored in many different sites. The key to discovering MFDs as fast as possible is pruning the useless candidate MFDs. And in most existed algorithms, it usually prunes candidate MFDs from top to bottom or from bottom to top. We present a new algorithms FastMFDs for discovering all MFDs from large-scale distributed data both from top to bottom and from bottom to top in parallel. We experimented our algorithm in real-life datasets, and our algorithm is more efficient and faster than the existed discovering algorithms.

Recently, the computational requirements for large-scale data-intensive analysis of scientific data have grown significantly. In High Energy Physics (HEP) for example, the Large Hadron Collider (LHC) produced 13 petabytes of data in 2010. This huge amount of data are processed on more than 140 computing centers distributed across 34 countries. The MapReduce paradigm has emerged as a highly successful programming model for large-scale data-intensive computing applications. However, current MapReduce implementations are developed to operate on single cluster environments and cannot be leveraged for large-scale distributed data processing across multiple clusters. On the other hand, workflow systems are used for distributed data processing across data centers. It has been reported that the workflow paradigm has some limitations for distributed data processing, such as reliability and efficiency. In this paper, we present the design and implementation of G-Hadoop, a MapReduce framework that aims to enable large-scale distributed computing across multiple clusters. G-Hadoop uses the Gfarm file system as an underlying file system and executes MapReduce tasks across distributed clusters.

Understanding the main determinants of species coexistence across space and time is a central question in ecology. However, ecologists still know little about the scales and conditions at which biotic interactions matter and how these interact with the environment to structure species assemblages. Here we use recent theory developments to analyze plant distribution and trait data across Europe and find that plant height clustering is related to both evapotranspiration and gross primary productivity. This clustering is a signal of interspecies competition between plants, which is most evident in mid-latitude ecoregions, where conditions for growth (reflected in actual evapotranspiration rates and gross primary productivities) are optimal. Away from this optimum, climate severity likely overrides the effect of competition, or other interactions become increasingly important. Our approach bridges the gap between species-rich competition theories and large-scale species distribution data analysis.

A long-standing objective of ecology has been to explain the basis for diversity patterns. Empirical evidence suggests that regional variation in richness of both animals and plants depends strongly on energy availability. The generality of the richness-energy hypothesis is limited by the paucity of analyses of invertebrates, which are much more diverse than the more thoroughly investigated vertebrate taxa. In this study, we consider two groups of North American Lepidoptera for which large-scale distribution data are available: the Papilionidae (swallowtail butterflies) and forest lepidopterans (moths based on the Canadian Forest Insect Survey). Energy, as measured by potential evapotranspiration (PET), statistically explains between 61 and 72% of the variability in the richness patterns of the Lepidoptera we have examined. It is the single best predictor of the richness of these groups, and the relationships have a very similar form to richness-PET relationships observed earlier in vertebrate taxa. After PET, Papilionidae richness is related to topographical heterogeneity. These patterns are true both within and among biomes. These results suggest that the richness-energy hypothesis applies generally to both vertebrates and insects in cold and temperate regions.

The computational grid provides the global computing infrastructure for users to access the services over a network. However, grid service providers charge users for the services based on their usage and QoS level specified. Therefore, in order to optimize the grid workflow execution, a robust multi-objective scheduling algorithm is needed considering economic cost along with execution performance. Generally, in multi-objective problems, simulations rely on running large number of evaluations to obtain the accurate results. However, algorithms that consider the preferences of decision maker, convergence to optimal tradeoff solutions is faster. Thus, in this chapter, the author proposed the preference-based guided search mechanism into MOEAs. To obtain solutions near the pre-specified regions of interest, the author has considered two MOEAs, namely R-NSGA-II and R-ε-MOEA. Further, to improve the diversity of solutions, a modified form called M-R-NSGA-II is used. Finally, the experimental settings and performance metrics are presented for the evaluation of the algorithms.

Replica creation strategy based on quantum evolutionary algorithm in data gird

Drug discovery is a lengthy, expensive and difficult process. Indentifying and understanding the hidden relationships among drugs, genes, proteins, and diseases will expedite the process of drug discovery. In this paper, we propose an effective methodology to discover drug-related semantic relationships over large-scale distributed web data in medicine, pharmacology and biotechnology. By utilizing semantic web and distributed system technologies, we developed a novel hierarchical knowledge abstraction and an efficient relation discovery protocol. Our approach effectively facilitates the realization of the full potential of harnessing the collective power and utilization of the drug-related knowledge scattered over the Internet.