P2P-based resource discovery in dynamic grids allowing multi-attribute and range queries

Resource discovery technologies for grids and peer-to-peer (P2P) systems share some characteristics. However, some P2P systems such as skip graph cannot simply be applied to grid resource discovery because complex grid resources need to be searched by using multi-attribute queries. This paper proposes a new multi-attribute P2P resource discovery approach (SkipMard) that extends skip graph structure to support multi-attribute queries. SkipMard provides a prefix matching resource routing algorithm to resolve multi-attribute queries, and introduces the concepts of and crossing layer nearest neighbor into the data structure. To decrease message passing numbers, an approximate closest-point method is addressed that can help routing a searching key to a node with a key value that has the minimum distance between two keys. Each node has O(m*l) neighbors for total m layers and I levels in SkipMard. The expected time for a multi-attribute query is O(log N) and the message passing number is O(logN)+O(k).

RCT: A distributed tree for supporting efficient range and multi-attribute queries in grid computing

The aim of this thesis is to design and implement a new Grid Resource Management methodology, where non-massive resources owners would be able to share their resources and integrate human collaboration across multiple domains regardless of network technology, operative platform or administrative domain.<br/>This thesis proposes a distributed and heuristic policy-based resource management architecture for large-scale Grids. The resource management architecture proposed herein is composed of four main building blocs: services management, resource discovery and monitoring, resource scheduling and jobs allocation and activation. The Grid Services Management (GSM) and Jobs Allocation and Activation (JAA) are supported by means of a Policy-based Grid Resource Management Architecture (PbGRMA). This architecture is able to identify service needs arising from diverse sources during the deployment and management of Grid Services, such as requirements demanded by customers, applications and network conditions. Afterwards, the PbGRMA merges these requirements into deployment policies for the corresponding Grid Services. The Grid Resource Discovery and Monitoring (GRDM) is supported by the introduction of the SNMP-based Balanced Load Monitoring Agents for Resource Scheduling (SBLOMARS), in which network and computational resources are monitored by distributed agents. This allows for a flexible, heterogeneous and scalable monitoring system. The Grid Resource Scheduling (GRS) is based on the Balanced Load Multi-Constrained Resource Scheduler (BLOMERS). This heuristic scheduler represents an alternate way of solving the inherent NP-hard problem for resource scheduling in large-scale distributed networks by means of the implementation of a Genetic Algorithm.<br/>Finally, based on the outcome of both the GRDM and GRS, the PbGRMA allocates the corresponding Grid Services by means of its interfaces with Globus ToolKit Middleware and Unix-based CLI commands along of any large-scale Grid Infrastructure. <br/>The synergy obtained by these components allows Grid administrators to exploit the available resources with predetermined levels of Quality of Service (QoS), reducing computational costs and makespan in resource scheduling while ensuring that the resource load is balanced throughout the Grid. The makespan of a schedule is the time required for all jobs to be processed when no one job could be interrupted during its execution and each node can perform at most one operation at any time.<br/>This new approach has been successfully tested in a real large-scale scenario such as Grid5000. The results presented along this Thesis show that our general solution is a reliable, flexible and scalable architecture to deploy and manage Grid Services in large-scale Grid Infrastructures. Moreover, the substitution of the heuristic algorithm approach used into the Grid Resource Scheduling (GRS) phase by other non-heuristics selection algorithms could make our solution useful in smaller Grid Infrastructures.

Recently, the resource sharing systems apply the P2P technique to provide scalable multi-attribute range queries. However, due to the heterogeneity of resources and the variation of sharing policies in different providers, current P2P-based resource discovery systems may suffer the load imbalance problem in a large scale distributed system. In this paper, we propose a self-adaptive resource index and discovery system (SARIDS) to achieve load balancing. SARIDS adopts a two-tier architecture based on the structured P2P overlay. The intra-overlay is constructed by normal peers with the same attribute via the locality preserving hash function; and, the inter-overlay is constructed by super-peers with classified attributes in different intra-overlays. SARIDS supports not only the multi-attribute range queries but also the self-adaptive mechanisms for load balancing in the intra-overlay and among the intra-overlays. The simulation results show that SARIDS is scalable and efficient for load balancing even in the non-uniform peer range environment.

In grid environment where resources are generally owned by different people, communities or organizations with varied administration policies and capabilities, managing the grid resources is not a simple task. Resource brokers simplify this process by providing an abstraction layer for users to access heterogeneous resources transparently. However, discovery of grid resources that suits to the user's job requirements is always a difficult job as the probability of grid resources satisfying the user's requirements is very less. Conventionally, in order to run a job on the grid a user has to identify a set of platforms capable of running that job by the virtue of having the required installation of operating system, libraries, tools, and the configuration of environment variables, etc. In practice, the availability of such software environments will either be limited to a very narrow set, or the job has to be made compatible with an environment supported by a large resource provider, such as TeraGrid. Further, if we could identify such an environment, it is hard to guarantee that the resource will be available when needed, for as long as needed, and that user will get his or her fair share of that resource. This difficulty can be overcome by incorporating the concept of Virtualization in Grid environment that enables the creation of dynamic user defined execution environment in a grid resource.

Grids enable users to share and access large collections and various types of resources in wide areas, and how to locate resources in such dynamic, heterogeneous and autonomous distributed environments is a key and challenging issue. In this paper, a three-level decentralized and dynamic VEGA Infrastructure for Resource Discovery (VIRD) is proposed. In this architecture, every Border Grid Resource Name Server (BGRNS) or Grid Resource Name Server (GRNS) has its own local policies, governing information organization, management and searching. Changes in resource information are propagated dynamically among GRNS servers according to a link-state-like algorithm. A client can query its designated GRNS either recursively or iteratively. Optimizing techniques, such as shortcut, are adopted to make the dynamic framework more flexible and efficient. A simulator called SimVIRD is developed to verify the proposed architecture and algorithms. Experiment results indicate that this architecture could deliver good scalability and performance for grid resource discovery.

Collaborative peer-to-peer (P2P), grid, and cloud computing rely on resource discovery (RD) solutions to aggregate groups of multi-attribute, dynamic, and distributed resources. However, specific characteristics of real-world resources and queries, and their impact on P2P-based RD, are largely unknown. We analyze the characteristics of resources and queries using data from four real-world systems. These characteristics are then used to qualitatively and quantitatively evaluate the fundamental design choices for P2P-based multi-attribute RD. The datasets exhibit several noteworthy features that affect the performance. For example, compared to uniform queries, real-world queries are relatively easier to resolve using unstructured, superpeer, and single-attribute-dominated query-based structured P2P solutions, as queries mostly specify only a small subset of the available attributes and large ranges of attribute values. However, all the solutions are prone to significant load balancing issues, as the resources and queries are highly skewed and correlated. The implications of our findings for improving RD solutions are also discussed.

Grid computing is the next generation of distributed system. Its goal is to create a huge, independent and powerful virtual machine, and it has been created by gathering different machines with the aim of sharing them. Resource discovery is the most fundamental phase of resource management which has been considered in our work. Considering the advantages and disadvantages of P2P and grid and their architectural similarity; using some of P2P approaches as a solution in Grid can improve the performance of Grid. Semantic technology and fuzzy logic are the two intelligent approaches, which recently apply on P2P network and Grid environment. In P2P networks, nodes are grouping based on their interests. Moreover semantic can be very useful for grouping nodes. This paper follows the identification of resource discovery in grid environment with the help of semantic and fuzzy theory and seeking to present a resource discovery algorithm with respect to exiting obstacles. The proposed approach is assessed in simulated grid environment and the results are compared with other approaches in same conditions. The goals of our approach are to obtain precious matching by clustering nodes which have services semantically related, improving search expressiveness by considering the distances between nodes (delay) in clustering phase, and decreasing response time and processing time by considering the nodes computational capabilities and free space of each node. We evaluate the performance of our approach with some distributed semantic based grid resource discovery models. The results of the experiments show the efficiency of the proposed approach in term of scalability, precision, search expressiveness and response time.

Peer-to-Peer resource discovery in Grids: Models and systems

Resource discovery is critical to the usability and accessibility of grid computing systems. Distributed Hash Table (DHT) has been applied to grid systems as a distributed mechanism for providing scalable range-query and multi-attribute resource discovery. Multi-DHT-based approaches depend on multiple DHT networks with each network responsible for a single attribute. Single-DHT-based approaches keep the resource information of all attributes in a single node. Both classes of approaches lead to high overhead. In this paper, we propose a Heuristic Low-Overhead Range-query Multi-attribute (H-LORM) DHT-based resource discovery approach. Unlike other DHT-based approaches, H-LORM relies on a single compound graph-based DHT network and distributes resource information among nodes in balance by taking advantage of the compound graph structure. Moreover, it has high capability to handle the large-scale and dynamic characteristics of resources in grids. Experimental results demonstrate the efficiency of H-LORM in comparison with other resource discovery approaches. LORM dramatically reduces maintenance and resource discovery overhead. In addition, it yields significant improvements in resource location efficiency. We also analyze the performance of the LORM approach rigorously by comparing it with other multi-DHT-based and single-DHT-based approaches with respect to their overhead and efficiency. The analytical results are consistent with experimental results, and prove the superiority of the H-LORM approach in theory.

Serving large number of users without compromising service availability and performance is key to the success of the cloud. A fundamental challenge in building such services is incorporating scalability and fail safe techniques for discovering and provisioning of resources. As peer-to-peer (P2P) architectures are invincible to these setbacks, the work proposes a P2P-based resource discovery and provisioning method for the cloud. It first addresses the multi attribute data publishing and the range querying inability of existing distributed hash table (DHT)-based P2P schemes and proposes an attribute hub-based discovery of provisioning information. Focused on that, a decentralised resource provisioning model is proposed using multi-attribute utility theory methods. The simulation shows that the proposed approach is 44.24% and 45.81% faster than the centralised and DHT-based approaches respectively in case of multidimensional range querying. It also shows the lesser number of service level objective (SLO) violations and migrations which are about 24.11% and 33.43% respectively.

A hybrid mechanism for resource/service discovery in ad-hoc grids

Ad hoc network is an infra structure less network, which is formed by mobile devices like laptops, PDAs, cell phones etc. Each device has different computational capability, power, hardware and software, which forms a heterogeneous network. These devices can be integrated to form an infrastructure known as grid. A grid integrates and coordinates resources and users that are within the same network with different capabilities. Hence we can visualize a grid over an ad hoc network that effectively utilizes the heterogeneity in the mobile devices. The major challenge in forming a grid over an ad hoc network is the mobility of the nodes. In this paper, we address the challenges due to mobility by considering a trace model for the movement of the nodes. Next, we demonstrate the feasibility of forming a grid over a mobile ad hoc network by proposing lightweight algorithms for grid formation, resource discovery, negotiation, job scheduling, and resource sharing. We have analyzed the performance of mobile ad hoc grid both by using a theoretical model and by simulation. The results point to a promising approach to form a mobile ad hoc grid.

In this article, we illustrate that the boundary of a general-purpose node can be extended into the network by extracting information from network traffic generated by that general-purpose node to infer the state of its hardware components. This information is represented in a delay signature latent within the network traffic. In contrast, the traditional approach to determine the internal state of a node’s resources meant that a software application with internal processes had to be resident on the node. The aforementioned delay signature is the keystone that provides a correlation between network traffic and the internal state of the source node. We characterize this delay signature by (1) identifying the different types of assembly language instructions that source this delay and (2) describing how architectural techniques, such as instruction pipelining and caching, give rise to this delay signature. In theory, highly utilized nodes (due to multiple threads) will contain excessive context switching and contention for shared resources. One important shared resource is main memory, and excessive use of this resource by applications and internal processes eventually leads to a decrease in cache efficiency that eventually stalls the instruction pipeline. Our results support this theory; specifically, we have observed that excessive context switching in active applications increases the effective memory access time and wastes precious CPU cycles, thus adding additional delay to the execution of load, store, and other instructions. Because the operating system (OS) kernel accesses memory to send network packets, the delay signature is induced into network traffic in situations where user-level utilization is high. We demonstrate this theory in two case studies: (1) resource discovery in cluster grids and (2) network-based detection of bitcoin mining on compromised nodes.

In economic resource management systems, the objective is to bring about an efficient allocation of resource while allowing market participants, resource consumers and owners, to act on their individual valuations of jobs and resources. In this paper we investigate the pricing of resources in dynamic grids. Particularly, we investigate grid resource management based on a computational commodity market of CPU resources, where resources are determined by supply-and-demand equilibrium. In particular introduce several categories of CPU's characterized by execution speed. These differ in cost and performance but may be used interchangeably in executing jobs and thus represent so-called substitutable resources. We investigate, using simulation, whether the algorithms for computing the supply-and-demand equilibrium, function in this context.