Using join operations as reducers in distributed query processing

In the environment of wide-area networks such as the Internet, distributed query processing is becoming problematic due to changes of the underlying networks and user requirements. In this new context, conventional query processing strategies with the homogeneous assumption will not work well, because they are unable to adapt to unexpected changes in the performance of the communication networks. In this paper, we address the issue of how to make distributed query processing aware of these changes. We introduce the idea of integrating distributed query processing with quality of service (QoS) management and accordingly illustrate our view of QoS-aware distributed query processing.

Fault-tolerance has long been a feature of database systems, with transactions supporting the structuring of applications so as to ensure continuation of updating applications in spite of machine failures. For read-only queries the perceived wisdom has been that support for fault-tolerance is too expensive to be worthwhile. Distributed query processing is coming to be seen as a promising way of implementing applications that combine structured data and analysis operations in dynamic distributed settings such as computational grids. Such a query may be long-running and having to redo the whole query after a failure may cause problems (e.g. if the result may trigger business or safety critical activities). This work describes and evaluates a new scheme for adding fault-tolerance to distributed query processing through a rollback-recovery mechanism. The high level expression of user requests in a physical algebra offers opportunities for tuning the fault-tolerance provision so as to reduce the cost, and give better performance than employment of generic fault-tolerance mechanisms at the lowest level of query processing. This paper outlines how the publicly-available OGSA-DQP computational grid-based distributed query processing system can be modified to include support for fault-tolerance and presents a performance evaluation which includes measurements of the cost of both protocol overheads and rollback-recovery, for a set of example distributed queries.

An integer programming model is given to deal with the problem that an online store wants to arrange the display time of goods pictures within the given scheduling horizon and space in order to maximize its profits. In this model, two characters that goods pictures have effect on demands and there are no inventories in many online stores are considered. It is explained that the problem discussed is equivalent to a special multiple-choice knapsack problem, so a dynamic programming algorithm with a complexity of pseudo polynomial time is provided; the properties of the objective function are studied, a greedy-like heuristic algorithm based on them with a complexity of polynomial time is proposed, and an example is given to demonstrate that the greedy-like heuristic method is effective.

Query processing in Web search engines today is mainly performed within a single site or data center, which is required to scale as the Web grows and users require fast answers to their queries. Constraints in the size and cost of data centers, however, may limit the scalability of search engines. Multi-site search engines that perform distributed query processing represent one way to overcome such constraints. Each site processes locally as many queries as possible, keeping latency low without contacting remote sites. Forwarding a query to remote sites depends on the document collection of remote sites. Multi-site search engines pose several new challenges. When a site updates its index, it has to inform other sites. The updates, however, are not instantaneous due to the volume of data exchanged or possible network failures. During the period of time that there are index inconsistencies across sites, queries may not be forwarded optimally. In this work, we investigate the impact of index inconsistencies on a distributed query processing algorithm, when there are index updates, and we observe that delayed index information propagation reduces the effectiveness of query processing, because queries are less likely to be routed optimally.

A Theory of Complexity for Continuous Time Systems

The article describe how to configure the InfoSphere Federation Server associated with the ORACLE database and distributed query processing in this environment. It presents how to create objects associated with distributed database. It presents how to define data sources and the wrappers used for the mapping of usersconnected to the remote server, the definition of nicknames for a remote data source object, the associated tables and stored procedures remotely.

This work aims at building secure multi-party database systems, for emerging federation networks in healthcare, finance, and other marketplaces. The technical challenges come from the interface gap between existing distributed query processing (DQP) and multi-party computation software. We propose compositional MPC for modular DQP and MPC-aware query optimizations.

Distributed data processing is becoming a reality. Businesses want to do it for many reasons, and they often must do it in order to stay competitive. While much of the infrastructure for distributed data processing is already there (e.g., modern network technology), a number of issues make distributed data processing still a complex undertaking: (1) distributed systems can become very large, involving thousands of heterogeneous sites including PCs and mainframe server machines; (2) the state of a distributed system changes rapidly because the load of sites varies over time and new sites are added to the system; (3) legacy systems need to be integrated—such legacy systems usually have not been designed for distributed data processing and now need to interact with other (modern) systems in a distributed environment. This paper presents the state of the art of query processing for distributed database and information systems. The paper presents the “textbook” architecture for distributed query processing and a series of techniques that are particularly useful for distributed database systems. These techniques include special join techniques, techniques to exploit intraquery paralleli sm, techniques to reduce communication costs, and techniques to exploit caching and replication of data. Furthermore, the paper discusses different kinds of distributed systems such as client-server, middleware (multitier), and heterogeneous database systems, and shows how query processing works in these systems.

Query Processing is a key determinant in the overall performance of distributed databases. It requires processing of data at their respective sites and transmission of the same between them. These together constitute a distributed query processing strategy (DQP). DQP aims to arrive at an efficient query processing strategy for a given query. This strategy involves generation of efficient query plans for a distributed query. In case of distributed relational queries, the number of possible query plans grows exponentially with an increase in the number of relations accessed by the query. This number increases further when the relations, accessed by the query, have replicas at different sites. Such a large search space renders it infeasible to find optimal query plans. This paper presents a query plan generation algorithm that attempts to generate optimal query plans, for a given query, using genetic algorithm. The query plans so generated involve fewer sites, thus leading to efficient query processing. Further, experimental results show that the proposed algorithm converges quickly towards optimal query plans for an observed crossover and mutation probability.

(i) (ii) (is) In the integrated strategy, one component decides which relation should remain fragmented at different sites. The other component decides which local operations, selections and projections, should be performed before the join operations. Our experimental results reveal that the choices made by the integrated algorithm in deciding which relation should remain fragmented and which local operations to be performed are valid. More precisely , the response times of queries processed by the algorithm are lower than those of the same queries processed using other strategies. These experimental results agree with the analytic cost model which we have previously proposed [YGC87]. In addition, our experiments provide insight on the relative cost VO, network, and processing cost in a real environment. This enables us to identify timeconsuming operations in distributed query processing, and suggest ways to improve perforniance.

Graph Analytics (GA) is a rapidly growing field that holds significant importance in numerous applications that focuses on analysing the complex relations in a network. One of the crucial concepts in GA is Query Reachability which refers to the ability to find a path connecting two nodes (u, v) in graph G. The multiple attributes in reachability query processing play a significant role to discover some interesting patterns. Existing studies did not deal with processing complex queries in attributed graphs. To fulfill some of the challenges, We contributed to this paper with three ideas for solving complex queries, finding paths, and processing multiple queries in parallel and distributed paradigms. Therefore, 1) Proposed Distributed Set Reachability Query in Attributed Networks (DSRQ_AN) algorithm using distributed computing through the Neo4j graph data platform and Cypher Query Language, 2) Finding the path to the given complex queries, and 3) Proposed a Multiple Query Processing in Parallel and Distributed (MQP_PD) algorithm to execute in a parallel environment. This paper assessed the performance of the proposed approaches through a parallel paradigm on Neo4j slaves. Furthermore, the demonstrated results of parallel execution outperform sequential execution for multiple queries on attributed graphs.

This paper explores secure data processing in hybrid clouds wherein local computing capability is exploited alongside public cloud services to deliver an efficient and secure data management solution. Hybrid clouds offer numerous advantages including the ability to selectively outsource data and computations based on sensitivity/confidentiality. Data processing in hybrid clouds must address two interrelated challenges: (i) data distribution: how is data distributed across public and private machines, and (ii) distributed query processing: how are queries executed efficiently without leaking sensitive data to untrusted public machines. This paper addresses these challenges and incorporates the respective solutions into an add-on tool for a Hadoop, Spark, and Hive based cloud computing infrastructure. Our results show performance advantages in using our strategy as compared to other secure alternatives, even when the percentage of sensitive data is as high as 50%.

In distributed query processing systems where caching infrastructure is distributed and scales with the number of servers, it is becoming more important to orchestrate and leverage a large number of cached objects in distributed caching systems seamlessly as the present trend is to build large scalable distributed systems by connecting small heterogeneous machines. With a large scale distributed caching system, a scheduling policy must consider both cache hit ratio and system load balance to optimize multiple queries. A scheduling policy that considers system load but not cache hit ratio often fails to reuse cached data by not assigning a query to the sever that has data objects the query needs. On the contrary, a scheduling policy that considers cache hit ratio but not system load balance may suffer from system load imbalance. To maximize the overall system throughput and to reduce query response time, a multiple query scheduling policy must balance system load and also leverage cached objects. In this paper, we present a distributed query processing framework that exhibits high cache hit ratio while achieving good system load balance. In order to seamlessly manage our distributed scalable caching system, our framework performs autonomic cached data migrations to improve cache hit ratio. Our experiments show that our proposed query scheduling policy and data migration policy significantly improve system throughput by achieving high cache hit ratio while avoiding system load imbalance.

With the growing popularity of Internet of Things (IoT) and sensing technologies, a large number of data streams are being generated at a very rapid pace. To explore the potentials of the integration of IoT and semantic technologies, a few RDF Stream Processing (RSP) query engines are made available which are capable of processing, analyzing and reasoning over semantic data streams in real-time. This way, RSP mitigates data interoperability issues and promotes knowledge discovery and smart decision making for time-sensitive applications. However, a major hurdle in the wide adoption of RSP systems is their query performance. Particularly, the ability of RSP engines to handle a large number of concurrent queries is very limited which refrains large scale stream processing applications (e.g. smart city applications) to adopt RSP. In this paper, we propose a shared-join based approach to improve the performance of an RSP engine for concurrent queries. We also leverage query federation mechanisms to allow distributed query processing over multiple RSP engine instances in order to gain performance for concurrent and distributed queries. We apply load balancing strategies to distribute queries and further optimize the concurrent query performance. We provide a proof of concept implementation by extending CQELS RSP engine and evaluate our approach using existing benchmark datasets for RSP. We also compare the performance of our proposed approach with the state of the art implementation of CQELS RSP engine.

The previous schemes of key establishment in the wireless sensor networks may not be employed for the mission-critical application due to several limitations: lightweightness and scalability from the point of performance, vulnerabilities against node compromise and various existing attacks from the point of security. In this paper, after identifying security requirements of mission-critical applications over sensor networks, we propose a scalable and robust hierarchical key establishment scheme which enhances resilience against node capture, traffic analysis attack and acknowledgment spoofing attack. In addition, our scheme provides periodic key updates without communication costs for key transport. We verified that our scheme requires less storage, computation and communication cost compared with the previous scheme in the open literature. When AES-256 is used for symmetric encryption and one cluster consists of 50 sensor nodes, we can reduce 93.4% storage requirement and 17.2%~51.3% communication cost of the authentication request for the cluster. Since the reduced communication and computation costs enable the time of authentication process to be short, our scheme can support relatively fast initialization and fault recovery. Moreover, our scheme prolongs the lifetime of the wireless sensor networks.