Group Aware Semantic Matchmaker for Grid Resource Discovery

Resource discovery is a crucial component in high-performance computing (HPC) systems. This paper presents a multi-agent model for resource discovery in distributed exascale systems. Agents are categorized based on resource types and behavior-specific characteristics. The model enables efficient identification and acquisition of memory, process, file, and IO resources. Through a comprehensive exploration, we highlight the potential of our approach in addressing resource discovery challenges in exascale computing systems, paving the way for optimized resource utilization and enhanced system performance.

We present a general framework for resource discovery, composition and substitution in mobile ad-hoc networks, exploiting knowledge representation techniques. Key points of the proposed approach are: (1) reuse of discovery information at network layer in order to build a fully unified semantic-based discovery and routing framework; (2) use of semantic annotations in order to perform the orchestration of elementary resources for building personalized services adopting a concept covering procedure, and to allow the automatic substitution of no more suitable/available components. Using ns-2 simulator, we evaluated performances of the proposed framework with reference to a disaster recovery scenario. In particular, the impact of the number of available services and active clients has been investigated in various mobility conditions and for several service covering threshold levels. Obtained results show that: (1) the proposed framework is highly scalable, given that its overall performance is improved by increasing the number of active clients; (2) the traffic load due to clients is negligible; (3) also for a very small number of available service providers very high hit ratios can be reached; (4) increasing the number of providers can lead to hit ratios very close to 100% at the expense of an increased traffic load. Finally, the effectiveness of cross-layer interaction between routing and resource discovery protocols has been also evaluated and discussed.

In large-scale computational or P2P Grids, discovery of heterogeneous resources as a working group is crucial to achieving scalable performance. This paper presents a hierarchical cycloid overlay (HCO) architecture with resource clustering and discovery algorithms for efficient and robust resource discovery in wide-area distributed Grid systems. We establish program/data locality by clustering resources based on their physical proximity and functional matching with user applications. We further develop randomized probing and cluster-token forwarding algorithms. The novelty of the HCO scheme lies in low overhead, fast speed and dynamism resilience in multi-resource discovery. The paper presents the HCO framework, new performance metrics, and simulation experimental results. This HCO scheme compares favorably with other resource management methods in static and dynamic Grid applications. In particular, it supports efficient resource clustering, reduces communications cost, and enhances resource discovery success rate in promoting large-scale distributed supercomputing applications.

As grid computing continues to gain popularity in the industry and research community, it also attracts more attention from the customer level. The large number of users and high frequency of job requests in the consumer market make this scenario a challenge for grid computing. Clearly, all the current client/server(C/S)-based architectures will become unfeasible for supporting large-scale grid applications due to their poor scalability and poor fault-tolerance. In order to address this issue, a novel C/S and peer-to-peer hybrid architecture to realise a highly scalable and flexible platform for optical grid is proposed here. The anycast algorithms, which are used to search suitable resources for job requests in this hybrid architecture, are also investigated in detail. Simulation and experimental results show that the proposed architecture is suitable and efficient for grid applications, and it outperforms C/S-based architecture in a large-scale grid environment.

Since ten years, the Grid systems are hot research topics. Recently, the Grid systems open towards the management of heterogeneous and distributed data on a large-scale environment. The Grid data management raises new problems and presents real challenges: (i) resource discovery and allocation, (ii) query processing and optimization, (iii) monitoring services, (iv) replication and caching, (v) cost models, (vi) autonomic data management, and (vii) security. The main characteristics offered by these systems are: large scale (e.g. high numbers of data sources, and computing resources) and dynamicity of nodes (unstable system). The synergy and convergence of interests between Grid systems and agent systems have been clearly pointed out. To address some of the above problems, a complementary approach (with respect to web services and P2P techniques) based on mobile agents deserves to be explored. In this talk, we draw up a synthetic state of the art of using mobile agents to solve some fundamental problems of data management in grid systems (e.g. resource discovery and allocation, distributed query and optimization, monitoring services for query optimization, cost models). We point out the advantages of mobile agents and how they can help for decentralized control, and scaling.

Grids enable users to share and access large collections and various types of resources in wide areas, and how to locate resources in such dynamic, heterogeneous and autonomous distributed environments is a key and challenging issue. In this paper, a three-level decentralized and dynamic VEGA Infrastructure for Resource Discovery (VIRD) is proposed. In this architecture, every Border Grid Resource Name Server (BGRNS) or Grid Resource Name Server (GRNS) has its own local policies, governing information organization, management and searching. Changes in resource information are propagated dynamically among GRNS servers according to a link-state-like algorithm. A client can query its designated GRNS either recursively or iteratively. Optimizing techniques, such as shortcut, are adopted to make the dynamic framework more flexible and efficient. A simulator called SimVIRD is developed to verify the proposed architecture and algorithms. Experiment results indicate that this architecture could deliver good scalability and performance for grid resource discovery.

Most of the semantic matchmaking algorithms have been developed for the search of Services, since there is a huge need for tools that help taking advantages of services that already exist, but which most of the times are under exploited because their existence remains unknown; thus to promote their usage, matchmaking algorithms are build. However, similar types of solutions for resource discovery in organizations are quasi-inexistent mainly due to the difficulty that represents the management of all the resources available in organizations where human interaction and their implications must be considered. In this paper, we explore an approach to the search of shared resources in collaborative settings by profiting from several principles of the typical semantic match- making algorithms. The main contribution of our proposal is to consider people's collaborative context, preferences, and social relationships in order to evaluate their current restrictions when they make a request regarding shared resources. In this way, the users will receive a highly personalized match. Keywords-ubiquitous collaborative environments, resource discovery, semantic matchmaking. I. INTRODUCTION Services have shown to be useful for most of the organizations since they are designed as modular applica- tions that that can be published, located, and invoked through the Web (1). In addition, the majority of Services are accessible to the public. Thanks to these features, not only their growth was notorious, but also the need of match- making algorithms capable of obtaining the most adequate resources for the user arose. The most common algorithms to satisfy this requirement are semantic since, unlike the syntactic ones, they are able to find services considering their description meaning. When the matchmaking of Services is carried out, the underlying algorithm should be in charge of verifying whether the announced service is similar enough to the asked service, meaning that a service descrip- tion could be close enough to the user's request for them to be useful, even if both descriptions are not identical. To achieve this flexibility in matchmaking algorithms, similarity degrees have been used. According to Paolucci et al. (2), the engineering of matchmaking algorithms should satisfy the following requirements:

To realize the vision of Internet of Things, there must be mechanisms to discover resources and their capabilities. Thus resource discovery becomes a fundamental requirement of any IoT platform. This paper provides a comprehensive categorization of the current technology landscape for discovery while pointing out their advantages and limitations. Then, a novel search engine based resource discovery framework is proposed. The framework comprises of a proxy layer which includes drivers for various communication technologies and protocols. There is a central registry to store the configurations of the resources and index them based on the configuration parameters. A “lifetime” attribute is introduced which denotes the period through which a resource is discoverable. The search engine ranks the resulting resources of a discovery request and returns an URI to directly access each resource. The functionalities of the proposed framework are exposed using RESTful web services. The framework allows discovery of both smart and legacy devices. We have performed gap analysis of current IoT standards for discovery mechanisms and provide suggestions to improve them to maintain interoperability. The outcomes are communicated to Standard Development Organizations like W3C (in Web of Things Interest Group) and oneM2M.

In traditional job scheduling systems for the grid, a single or a few machines handle information of all computing resources and scheduling tasks. This centralized approach is not scalable, since it introduces single point of failure and bottleneck. Some decentralized scheduling systems have been proposed to improve scalability. They avoid concentration of scheduling costs by broadcasting job execution requests in a peer-to-peer style. However, resource utilization tends to be low, because most of them are not aware of the dynamic states of computing resources before throwing execution requests. This paper introduces a decentralized scheduling system that improves resource utilization by using a Gossip-based multicast protocol. With this protocol, peers can gather information of each other efficiently and schedule jobs individually. The simulation shows that our system is scalable and it handles many jobs efficiently in large scale grid environments

A survey on resource discovery mechanisms, peer-to-peer and service discovery frameworks

The Internet of Things (IoT) is defined as interconnected digital and mechanical devices with intelligent and interactive data transmission features over a defined network. The ability of the IoT to collect, analyze and mine data into information and knowledge motivates the integration of IoT with grid and cloud computing. New job scheduling techniques are crucial for the effective integration and management of IoT with grid computing as they provide optimal computational solutions. The computational grid is a modern technology that enables distributed computing to take advantage of a organization’s resources in order to handle complex computational problems. However, the scheduling process is considered an NP-hard problem due to the heterogeneity of resources and management systems in the IoT grid. This paper proposed a Greedy Firefly Algorithm (GFA) for jobs scheduling in the grid environment. In the proposed greedy firefly algorithm, a greedy method is utilized as a local search mechanism to enhance the rate of convergence and efficiency of schedules produced by the standard firefly algorithm. Several experiments were conducted using the GridSim toolkit to evaluate the proposed greedy firefly algorithm’s performance. The study measured several sizes of real grid computing workload traces, starting with lightweight traces with only 500 jobs, then typical with 3000 to 7000 jobs, and finally heavy load containing 8000 to 10,000 jobs. The experiment results revealed that the greedy firefly algorithm could insignificantly reduce the makespan makespan and execution times of the IoT grid scheduling process as compared to other evaluated scheduling methods. Furthermore, the proposed greedy firefly algorithm converges on large search spacefaster , making it suitable for large-scale IoT grid environments.

In this paper we present an innovative decentralized dynamic resource scheduling solution of large scale application workflows onto distributed, heterogeneous grid environments. The presented resource allocation solution aims to optimize large-scale scientific application workflows by efficiently and effectively mapping and scheduling them onto grid resources. The proposed solution is part of a framework that aims to help scientists easily deploy large-scale scientific workflow applications from a wide-range of research fields. The decentralized architecture, together with the scheduling policies and fault management being proposed in this paper, are specifically designed to optimize the scheduling management for workflows in case of a wide range of grid environment. In order to demonstrate the validity and performance of the presented scheduling solution we propose using modeling and simulation techniques. For that we present the characteristics of a generic simulator that includes all the necessary components to help in evaluating a wide-range of scheduling procedures, such as the ones presented in this, in the context of large-scale heterogeneous grid environment.

With the evolution of smart grids, accurate and secure predictions of the electricity load become crucial for efficient energy management and reliability. In this paper, a scalable and cyber-resilient methodology for electricity consumption forecasting on individual smart meter level based on machine learning and anomaly detection schemes is proposed. The proposed technique utilizes K-MEANS Clustering and Neural Networks (KMEANS–NN) to enhance Individual Load Forecasting (ILF) with reduced computational complexity and high prediction accuracy. A Principal Component Analysis based One-Class Support Vector Machine (PCA–OCSVM) model is employed as an Anomaly Detection Scheme (ADS) to identify the false data injection attacks in smart meter telemetry. The system uses five months of real-world data from :text{2,089} smart meters gathered under the supervision of Electrical Distribution Sector (EDS) of Suez Canal Authority (SCA) in Egypt. KMEANS–NN strategy reduces significantly MAAPE by up to :25.6% and cuts computational time from days to minutes. It improves forecasting accuracy across four proposed models: ARIMA, CTREE, MLP and NNETAR. To assess the cyber-security profile, :50% of the dataset is orchestrated with scaling, ramping and random cyber-attack simulation. Proposed ADS achieves :99.3% overall accuracy, :100% sensitivity, :98.62% precision, :98.6% specificity and F1-score of::0.9896, whereas it’s :100% accurate on clean data. This integrated model offers accurate, efficient, and secure load forecasting presenting good potential for its deployment in large-scale smart grid environments.

The aim of this thesis is to design and implement a new Grid Resource Management methodology, where non-massive resources owners would be able to share their resources and integrate human collaboration across multiple domains regardless of network technology, operative platform or administrative domain.<br/>This thesis proposes a distributed and heuristic policy-based resource management architecture for large-scale Grids. The resource management architecture proposed herein is composed of four main building blocs: services management, resource discovery and monitoring, resource scheduling and jobs allocation and activation. The Grid Services Management (GSM) and Jobs Allocation and Activation (JAA) are supported by means of a Policy-based Grid Resource Management Architecture (PbGRMA). This architecture is able to identify service needs arising from diverse sources during the deployment and management of Grid Services, such as requirements demanded by customers, applications and network conditions. Afterwards, the PbGRMA merges these requirements into deployment policies for the corresponding Grid Services. The Grid Resource Discovery and Monitoring (GRDM) is supported by the introduction of the SNMP-based Balanced Load Monitoring Agents for Resource Scheduling (SBLOMARS), in which network and computational resources are monitored by distributed agents. This allows for a flexible, heterogeneous and scalable monitoring system. The Grid Resource Scheduling (GRS) is based on the Balanced Load Multi-Constrained Resource Scheduler (BLOMERS). This heuristic scheduler represents an alternate way of solving the inherent NP-hard problem for resource scheduling in large-scale distributed networks by means of the implementation of a Genetic Algorithm.<br/>Finally, based on the outcome of both the GRDM and GRS, the PbGRMA allocates the corresponding Grid Services by means of its interfaces with Globus ToolKit Middleware and Unix-based CLI commands along of any large-scale Grid Infrastructure. <br/>The synergy obtained by these components allows Grid administrators to exploit the available resources with predetermined levels of Quality of Service (QoS), reducing computational costs and makespan in resource scheduling while ensuring that the resource load is balanced throughout the Grid. The makespan of a schedule is the time required for all jobs to be processed when no one job could be interrupted during its execution and each node can perform at most one operation at any time.<br/>This new approach has been successfully tested in a real large-scale scenario such as Grid5000. The results presented along this Thesis show that our general solution is a reliable, flexible and scalable architecture to deploy and manage Grid Services in large-scale Grid Infrastructures. Moreover, the substitution of the heuristic algorithm approach used into the Grid Resource Scheduling (GRS) phase by other non-heuristics selection algorithms could make our solution useful in smaller Grid Infrastructures.

This paper is concerned with modelling the process of service and resource discovery and allocation with specific emphasis on the management of that process. Its scope ranges from local low level discrete resources, e.g. finding and using network elements, through network services, e.g. establishing communications facilities, on to the highest level global service offerings, e.g. discovering suitable VPNs and subsequent negotiation with them. The discovery and allocation of both services and resources may be treated symmetrically.