Fault-Tolerance in Distributed Query Processing

Query Processing is a key determinant in the overall performance of distributed databases. It requires processing of data at their respective sites and transmission of the same between them. These together constitute a distributed query processing strategy (DQP). DQP aims to arrive at an efficient query processing strategy for a given query. This strategy involves generation of efficient query plans for a distributed query. In case of distributed relational queries, the number of possible query plans grows exponentially with an increase in the number of relations accessed by the query. This number increases further when the relations, accessed by the query, have replicas at different sites. Such a large search space renders it infeasible to find optimal query plans. This paper presents a query plan generation algorithm that attempts to generate optimal query plans, for a given query, using genetic algorithm. The query plans so generated involve fewer sites, thus leading to efficient query processing. Further, experimental results show that the proposed algorithm converges quickly towards optimal query plans for an observed crossover and mutation probability.

In the environment of wide-area networks such as the Internet, distributed query processing is becoming problematic due to changes of the underlying networks and user requirements. In this new context, conventional query processing strategies with the homogeneous assumption will not work well, because they are unable to adapt to unexpected changes in the performance of the communication networks. In this paper, we address the issue of how to make distributed query processing aware of these changes. We introduce the idea of integrating distributed query processing with quality of service (QoS) management and accordingly illustrate our view of QoS-aware distributed query processing.

Query processing in Web search engines today is mainly performed within a single site or data center, which is required to scale as the Web grows and users require fast answers to their queries. Constraints in the size and cost of data centers, however, may limit the scalability of search engines. Multi-site search engines that perform distributed query processing represent one way to overcome such constraints. Each site processes locally as many queries as possible, keeping latency low without contacting remote sites. Forwarding a query to remote sites depends on the document collection of remote sites. Multi-site search engines pose several new challenges. When a site updates its index, it has to inform other sites. The updates, however, are not instantaneous due to the volume of data exchanged or possible network failures. During the period of time that there are index inconsistencies across sites, queries may not be forwarded optimally. In this work, we investigate the impact of index inconsistencies on a distributed query processing algorithm, when there are index updates, and we observe that delayed index information propagation reduces the effectiveness of query processing, because queries are less likely to be routed optimally.

In distributed query processing systems where caching infrastructure is distributed and scales with the number of servers, it is becoming more important to orchestrate and leverage a large number of cached objects in distributed caching systems seamlessly as the present trend is to build large scalable distributed systems by connecting small heterogeneous machines. With a large scale distributed caching system, a scheduling policy must consider both cache hit ratio and system load balance to optimize multiple queries. A scheduling policy that considers system load but not cache hit ratio often fails to reuse cached data by not assigning a query to the sever that has data objects the query needs. On the contrary, a scheduling policy that considers cache hit ratio but not system load balance may suffer from system load imbalance. To maximize the overall system throughput and to reduce query response time, a multiple query scheduling policy must balance system load and also leverage cached objects. In this paper, we present a distributed query processing framework that exhibits high cache hit ratio while achieving good system load balance. In order to seamlessly manage our distributed scalable caching system, our framework performs autonomic cached data migrations to improve cache hit ratio. Our experiments show that our proposed query scheduling policy and data migration policy significantly improve system throughput by achieving high cache hit ratio while avoiding system load imbalance.

Multiple query scheduling for distributed semantic caches

Graphs are increasingly being used as the data structure of choice to represent interactions between heterogeneous entities. Graph path querying is a primary operation in the network graph space, for both real time querying and inferential analysis. The rate and volume of interconnected data being generated warrants efficient distributed solutions to manage and query network graphs in a scalable fashion. Existing distributed solutions have proposed several optimization techniques, including intelligent joins and partial evaluations to process path queries. However, the former relies on comprehensive indices while the latter involves extensive driver-side processing to combine the partial results, neither of which is efficient for processing large graphs. In this paper, we propose a novel distributed graph path query processing system using the Apache Spark framework.

(i) (ii) (is) In the integrated strategy, one component decides which relation should remain fragmented at different sites. The other component decides which local operations, selections and projections, should be performed before the join operations. Our experimental results reveal that the choices made by the integrated algorithm in deciding which relation should remain fragmented and which local operations to be performed are valid. More precisely , the response times of queries processed by the algorithm are lower than those of the same queries processed using other strategies. These experimental results agree with the analytic cost model which we have previously proposed [YGC87]. In addition, our experiments provide insight on the relative cost VO, network, and processing cost in a real environment. This enables us to identify timeconsuming operations in distributed query processing, and suggest ways to improve perforniance.

Graph Analytics (GA) is a rapidly growing field that holds significant importance in numerous applications that focuses on analysing the complex relations in a network. One of the crucial concepts in GA is Query Reachability which refers to the ability to find a path connecting two nodes (u, v) in graph G. The multiple attributes in reachability query processing play a significant role to discover some interesting patterns. Existing studies did not deal with processing complex queries in attributed graphs. To fulfill some of the challenges, We contributed to this paper with three ideas for solving complex queries, finding paths, and processing multiple queries in parallel and distributed paradigms. Therefore, 1) Proposed Distributed Set Reachability Query in Attributed Networks (DSRQ_AN) algorithm using distributed computing through the Neo4j graph data platform and Cypher Query Language, 2) Finding the path to the given complex queries, and 3) Proposed a Multiple Query Processing in Parallel and Distributed (MQP_PD) algorithm to execute in a parallel environment. This paper assessed the performance of the proposed approaches through a parallel paradigm on Neo4j slaves. Furthermore, the demonstrated results of parallel execution outperform sequential execution for multiple queries on attributed graphs.

Distributed data processing is becoming a reality. Businesses want to do it for many reasons, and they often must do it in order to stay competitive. While much of the infrastructure for distributed data processing is already there (e.g., modern network technology), a number of issues make distributed data processing still a complex undertaking: (1) distributed systems can become very large, involving thousands of heterogeneous sites including PCs and mainframe server machines; (2) the state of a distributed system changes rapidly because the load of sites varies over time and new sites are added to the system; (3) legacy systems need to be integrated—such legacy systems usually have not been designed for distributed data processing and now need to interact with other (modern) systems in a distributed environment. This paper presents the state of the art of query processing for distributed database and information systems. The paper presents the “textbook” architecture for distributed query processing and a series of techniques that are particularly useful for distributed database systems. These techniques include special join techniques, techniques to exploit intraquery paralleli sm, techniques to reduce communication costs, and techniques to exploit caching and replication of data. Furthermore, the paper discusses different kinds of distributed systems such as client-server, middleware (multitier), and heterogeneous database systems, and shows how query processing works in these systems.

Semijoin has traditionally been relied upon for reducing the communication cost required for distributed query processing. However, judiciously applying join operations as reducers can lead to further reduction in the communication cost. In view of this fact, the approach of using join operations, in addition to semijoins, as reducers in distributed query processing is explored. It is shown that the problem of determining a sequence of join operations for a query graph can be transformed to that of finding a set of cuts to that graph, where a cut to a graph is a partition of the nodes in that graph. In light of the mapping, an efficient heuristic algorithm to determine an effective sequence of join reducers for a query is developed. The algorithm, using the concept of divide-and-conquer, is shown to have polynomial time complexity. Examples are given to illustrate the results.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

The article describe how to configure the InfoSphere Federation Server associated with the ORACLE database and distributed query processing in this environment. It presents how to create objects associated with distributed database. It presents how to define data sources and the wrappers used for the mapping of usersconnected to the remote server, the definition of nicknames for a remote data source object, the associated tables and stored procedures remotely.

This paper explores secure data processing in hybrid clouds wherein local computing capability is exploited alongside public cloud services to deliver an efficient and secure data management solution. Hybrid clouds offer numerous advantages including the ability to selectively outsource data and computations based on sensitivity/confidentiality. Data processing in hybrid clouds must address two interrelated challenges: (i) data distribution: how is data distributed across public and private machines, and (ii) distributed query processing: how are queries executed efficiently without leaking sensitive data to untrusted public machines. This paper addresses these challenges and incorporates the respective solutions into an add-on tool for a Hadoop, Spark, and Hive based cloud computing infrastructure. Our results show performance advantages in using our strategy as compared to other secure alternatives, even when the percentage of sensitive data is as high as 50%.

With the growing popularity of Internet of Things (IoT) and sensing technologies, a large number of data streams are being generated at a very rapid pace. To explore the potentials of the integration of IoT and semantic technologies, a few RDF Stream Processing (RSP) query engines are made available which are capable of processing, analyzing and reasoning over semantic data streams in real-time. This way, RSP mitigates data interoperability issues and promotes knowledge discovery and smart decision making for time-sensitive applications. However, a major hurdle in the wide adoption of RSP systems is their query performance. Particularly, the ability of RSP engines to handle a large number of concurrent queries is very limited which refrains large scale stream processing applications (e.g. smart city applications) to adopt RSP. In this paper, we propose a shared-join based approach to improve the performance of an RSP engine for concurrent queries. We also leverage query federation mechanisms to allow distributed query processing over multiple RSP engine instances in order to gain performance for concurrent and distributed queries. We apply load balancing strategies to distribute queries and further optimize the concurrent query performance. We provide a proof of concept implementation by extending CQELS RSP engine and evaluate our approach using existing benchmark datasets for RSP. We also compare the performance of our proposed approach with the state of the art implementation of CQELS RSP engine.

This work aims at building secure multi-party database systems, for emerging federation networks in healthcare, finance, and other marketplaces. The technical challenges come from the interface gap between existing distributed query processing (DQP) and multi-party computation software. We propose compositional MPC for modular DQP and MPC-aware query optimizations.

In distributed query processing, data shuffle is one of the most costly operations. We examined scaling limitations to data shuffle that current systems and the research literature do not solve. As the number of input and output partitions increases, naïve shuffling will result in high fan-out and fan-in. There are practical limits to fan-out, as a consequence of limits on memory buffers, network ports and I/O handles. There are practical limits to fan-in because it multiplies the communication errors due to faults in commodity clusters impeding progress. Existing solutions that limit fan-out and fan-in do so at the cost of scaling quadratically in the number of nodes in the data flow graph. This dominates the costs of shuffling large datasets. We propose a novel algorithm called Hyper Dimension Shuffle that we have introduced in production in SCOPE, Microsoft's internal big data analytics system. Hyper Dimension Shuffle is inspired by the divide and conquer concept, and utilizes a recursive partitioner with intermediate aggregations. It yields quasilinear complexity of the shuffling graph with tight guarantees on fan-out and fan-in. We demonstrate how it avoids the shuffling graph blow-up of previous algorithms to shuffle at petabyte-scale efficiently on both synthetic benchmarks and real applications.