S2D: Shared Distributed Datasets, Storing Shared Data for Multiple and Massive Queries Optimization in a Distributed Data Warehouse

Database researchers have made significant progress on several research issues related to multidimensional data analysis, including the development of fast cubing algorithms, efficient schemes for creating and maintaining precomputed group-bys, and the design of efficient storage structures for multidimensional data. However, to date there has been little or no work on multidimensional query optimization. Recently, Microsoft has proposed “OLE DB for OLAP” as a standard multidimensional interface for databases. OLE DB for OLAP defines Multi-Dimensional Expressions (MDX), which have the interesting and challenging feature of allowing clients to ask several related dimensional queries in a single MDX expression. In this paper, we present three algorithms to optimize multiple related dimensional queries. Two of the algorithms focus on how to generate a global plan from several related local plans. The third algorithm focuses on generating a good global plan without first generating local plans. We also present three new query evaluation primitives that allow related query plans to share portions of their evaluation. Our initial performance results suggest that the exploitation of common subtask evaluation and global optimization can yield substantial performance improvements when relational database systems are used as data sources for multidimensional analysis.

In order to answer efficiently range queries in 2-d R-trees, first we sort queries by means of a space filling curve, then we group them together, and finally pass them for processing. Initially, we consider grouping of pairs of requests only, and give two algorithms with exponential and linear complexity. Then, we generalize the linear method, grouping more than two requests per group. We evaluate these methods under different LRU buffer sizes, measuring the cache misses per query. We present experimental results based on real and synthetic data. The results show that careful query scheduling can improve substantially the overall performance of multiple range query processing.

An effective query processing plays an important role in the uncertain data streams. Specifically, multiple top-k queries processing on uncertain data streams obtained from large applications of several fields such as sensor network monitoring and internet traffic control requires periodic execution of queries and sharing results among them. The system that monitors uncertain events in such data streams manipulates the top-k queries. Here the problem is that systems were not designed to allow query results sharing which in turn leads to high computation cost and inaccurate response from the system. To overcome these issues (Queries results sharing), the system using a sampling algorithm for sample the top possible worlds from well-known possible worlds based on their high probability. System uses an optimal dynamic programming approach that split the multiple queries into number of groups. Then the query groups are scheduled and planned for sharing results to yield minimum computation cost. A faster greedy algorithm is used to reduce the time and storage space of the top-k queries based on the greedy rule. Thus the proposed approach allows sharing computation among multiple top-k queries and generates best plan of query execution.

MQO is a distributed multiple query processing middleware that can use resources available on the Grid to optimize query processing for data analysis and visualization applications. It does so by introducing one or more proxies that act as front-ends to a collection of backend servers. The basic idea behind this architecture is active semantic caching, whereby queries can leverage available cached results in the proxy either directly or through transformations. While this approach has been shown to speed up query evaluation under multi-client workloads, the caching infrastructure in the backend servers is not used well for query processing. Because this collective caching infrastructure scales with the number of servers, it is an important asset. In this paper, we describe a distributed multidimensional indexing scheme that enables the proxy to directly consider the cache contents available at the backend servers for query planning and scheduling. This approach is shown to produce better query plans and faster query response times as we experimentally demonstrate.

With the explosion of data in the past decade, big data is becoming a research hotspot in the information field. Many cloud-based distributed data processing platforms have been proposed to provide efficient and cost effective solutions for big data query processing, such as Hadoop, Hive, Pig, etc. However, most of the current research works are focus on improving the performance of query processing based on the view of systematics while without considering the characteristics of queries themselves, such as the query similarity, which will cause large numbers of redundant computation, effect query execution efficiency, thus having an adverse impact on promotion of the multi-queries processing performance. To solve this problem, in this paper, we propose a Multi-queries optimization framework based on MapReduce-oriented cloud environment (Multi-Q), which utilizes the dependence between multiple queries to realize query results reuse. Firstly, a cluster-based partition algorithm called CPA has been exploited to conduct the logic partition of the search range of query workload. Secondly, a multi-queries reuse dependence graph (MRDG) construction method on the basis of the cluster-based partition results has been presented to depict the dependence between the multiple queries. Finally, a Multi-Q processing algorithm based on Multi-Q Reuse Dependence Graph has been put forward to achieve the query results reuse and improve the overall query processing performance. We evaluate our approach by deploying Multi-Q based on Hadoop in a real cloud environment, called SEU-Cloud, and conducting extensive experiments based on the standard TPC-H. The result verifies that compared with Hive, the performance of improvement is approximately 39.3% by using our Multi-Q.

This paper presents a Graphics Processing Unit (GPU) based solution for classical geometric hashing and its variation with transformation functions on Speeded Up Robust Features (SURF) space of images. GPU based classical geometric hashing provides speed-up of 14.0x to 61.0x for offline indexing and 1.08x to 10.06x for online searching compared to sequential one for different partitioning sizes. GPU based transformation by mean invariancy and principal component based alignment with geometric hashing provides speed-up of 12.12x to 63.13x for offline indexing and 1.02x to 5.82x for online searching. This paper also proposes solution to execute multiple query simultaneously. It proves to be better than the serial execution of multiple queries. GPU based implementation of multi-query provide speed-up of 1.68x to 460.45x than the sequential one for online searching for multiple queries between 1 to 10 simultaneously. Experimentation is done using standard CASIA Palm-print based images.

Stream processing engines need to process multiple queries over streams simultaneously, and continuous window aggregation plays a critical role in various applications as a part of data analysis pipelines. However, the system suffers from scalability issues when dealing with massive queries with different window and slide sizes over data streams with high input rates. To address this problem, we propose LSiX (longest-shortest-window-based indexing) to aggregate multiple queries over data streams efficiently. More precisely, we employ two arrays based on the longest and shortest windows among all registered queries, and all query results are computed by using the shared partial aggregations in the two arrays using only two operations at most for each query, enabling efficient aggregation computation. We have conducted extensive experiments, and the results show that LSiX can be at least 3 times faster than the comparative methods, including the state-of-the-art method, MCQA.

Multi-route query processing and optimization

We study the problem of optimizing one-time and continuous subgraph queries using the new worst-case optimal join plans. Worst-case optimal plans evaluate queries by matching one query vertex at a time using multiway intersections. The core problem in optimizing worst-case optimal plans is to pick an ordering of the query vertices to match. We make two main contributions: 1. A cost-based dynamic programming optimizer for one-time queries that (i) picks efficient query vertex orderings for worst-case optimal plans and (ii) generates hybrid plans that mix traditional binary joins with worst-case optimal style multiway intersections. In addition to our optimizer, we describe an adaptive technique that changes the query vertex orderings of the worst-case optimal subplans during query execution for more efficient query evaluation. The plan space of our one-time optimizer contains plans that are not in the plan spaces based on tree decompositions from prior work. 2. A cost-based greedy optimizer for continuous queries that builds on the delta subgraph query framework. Given a set of continuous queries, our optimizer decomposes these queries into multiple delta subgraph queries, picks a plan for each delta query, and generates a single combined plan that evaluates all of the queries. Our combined plans share computations across operators of the plans for the delta queries if the operators perform the same intersections. To increase the amount of computation shared, we describe an additional optimization that shares partial intersections across operators. Our optimizers use a new cost metric for worst-case optimal plans called intersection-cost . When generating hybrid plans, our dynamic programming optimizer for one-time queries combines intersection-cost with the cost of binary joins. We demonstrate the effectiveness of our plans, adaptive technique, and partial intersection sharing optimization through extensive experiments. Our optimizers are integrated into GraphflowDB.

We present an scalable algorithm for answering multiple conjunctive queries using views. This is an important problem in query optimization, data integration and ontology-based data access. Since rewriting one conjunctive query using views is an NP-hard problem, we develop an approach where answering n queries takes less than n times the cost of answering one query, by compactly representing and indexing common patterns in the input queries and the views. Our initial experimental results show a promising speed up.

Deco is a comprehensive system for answering declarative queries posed over stored relational data together with data obtained on-demand from the crowd. In this paper we describe Deco's cost-based query optimizer, building on Deco's data model, query language, and query execution engine presented earlier. Deco's objective in query optimization is to find the best query plan to answer a query, in terms of estimated monetary cost. Deco's query semantics and plan execution strategies require several fundamental changes to traditional query optimization. Novel techniques incorporated into Deco's query optimizer include a cost model distinguishing between "free" existing data versus paid new data, a cardinality estimation algorithm coping with changes to the database state during query execution, and a plan enumeration algorithm maximizing reuse of common subplans in a setting that makes reuse challenging. We experimentally evaluate Deco's query optimizer, focusing on the accuracy of cost estimation and the efficiency of plan enumeration.

For two different types of data stream - a relational stream and an XML stream, this paper contains an adaptive query optimization strategy called a selection-early. The evaluation order of selection constructs can significantly influence the overall performance of multiple query evaluation. Consequently, based on the filtering capability of the current evaluation sequence of selection constructs dynamically captured at run-time, a selection-early strategy establishes the efficient evaluation sequence adaptively. For this purpose, the overall filtering capability of the current evaluation sequence is periodically monitored and the sequence is rearranged when its filtering capability is varied higher than a specific threshold. Accordingly, this strategy keeps the current evaluation sequence to be as efficient as possible by coping with a dynamic variation of its filtering capability. The experimental studies of the proposed strategy show that it is practically more scalable and stable than other approaches.

Query scheduling plays an important role when systems are faced with limited resources and high workloads. It becomes even more relevant for servers applying multiple query optimization techniques to batches of queries, in which portions of datasets as well as intermediate results are maintained in memory to speed up query evaluation. We present a dynamic query scheduling model based on a priority queue implementation using a directed graph and a strategy for ranking queries. We examine the relative performance of several ranking strategies on a shared-memory machine using two different versions of an application, called the Virtual Microscope, for browsing digitized microscopy images.

We present a relational schema that stores the computations of a shared query evaluation plan, and tools that search the common computations between new queries and the schema, which are the two essential parts of the Incremental Multiple Query Optimization (IMQO) framework we proposed to allow the efficient construction of the optimal evaluation plan for multiple continuous queries.

The increase in database amount, number of tables, blocks in the database and the size of query make Multi Join Query Optimization (MJQO) garnered considerable attention in Database Management System research. Many applications often involve complex multiple queries which share a lot of common subexpressions (CSEs) to Identifying and exploiting the CSEs to improve query performance is essential in these applications. MJQO aimed to find the optimal Query Execution Plan (QEP) in lower cost and minimum query execution time. The first contribution of this paper we examine the optimal join order (OJO) problem, which is a fundamental query optimization task for SQL-like queries in RDBMS, second contribution we propose a Swarm Intelligent approach to solve the MJQO problem. Two phase Artificial Bee Colony Algorithm (ABC) is used to solve the MJQO problems by simulating and exploiting the foraging behavior of honey bees. Results from the experiments show that the performance of two phase ABC when compared to Particle Swarm Optimization (PSO), Ant colony optimization (ACO) in terms of computational time is very promising. This indicates that the two phase ABC can solve MJQO problems in less amount of time, lower cost and more efficient than that of PSO and ACO.