Multiple range query optimization in spatial databases

With the explosion of data in the past decade, big data is becoming a research hotspot in the information field. Many cloud-based distributed data processing platforms have been proposed to provide efficient and cost effective solutions for big data query processing, such as Hadoop, Hive, Pig, etc. However, most of the current research works are focus on improving the performance of query processing based on the view of systematics while without considering the characteristics of queries themselves, such as the query similarity, which will cause large numbers of redundant computation, effect query execution efficiency, thus having an adverse impact on promotion of the multi-queries processing performance. To solve this problem, in this paper, we propose a Multi-queries optimization framework based on MapReduce-oriented cloud environment (Multi-Q), which utilizes the dependence between multiple queries to realize query results reuse. Firstly, a cluster-based partition algorithm called CPA has been exploited to conduct the logic partition of the search range of query workload. Secondly, a multi-queries reuse dependence graph (MRDG) construction method on the basis of the cluster-based partition results has been presented to depict the dependence between the multiple queries. Finally, a Multi-Q processing algorithm based on Multi-Q Reuse Dependence Graph has been put forward to achieve the query results reuse and improve the overall query processing performance. We evaluate our approach by deploying Multi-Q based on Hadoop in a real cloud environment, called SEU-Cloud, and conducting extensive experiments based on the standard TPC-H. The result verifies that compared with Hive, the performance of improvement is approximately 39.3% by using our Multi-Q.

A Reverse Top-k Geo-Social Keyword Query (RkGSKQ) aims to find all the users who have a given geo-social object in their top-k geo-social keyword query results. This query is practical in detecting prospective customers for online business in social networks. Existing work on RkGSKQ only explored efficient approaches in answering a single query per time, which could not be efficient in processing multiple queries in a query batch. In many real-life applications, multiple RkGSKQs for multiple query objects can be issued at the same time. To this end, in this paper, we focus on the efficient batch processing algorithm for multiple RkGSKQs. To reduce the overall cost and find concurrently results of multiple queries, we present a group processing framework based on the current state-of-the-art indexing and group pruning strategies to answer multiple RkGSKQs by sharing common CPU and I/O costs. Extensive experiments on three data sets demonstrate the effectiveness and efficiency of our proposed methods.

Wireless sensor networks (WSNs) have been proposed for a large variety of applications. As the number of applications of sensor networks continue to grow, the number of users in sensor networks increases as well. Consequently, it is not uncommon that base station often needs to process multiple queries simultaneously. Furthermore, these queries often need to collect data from some particular sets of sensors such as the sensors in a hot spot.To reduce the communication cost of multiple query processing in WSNs, this paper proposes a new optimization technique based on similarities among multiple queries. Given a set of queries, Q, the proposed scheme constructs a set of shared intermediate views (SIVs) from Q. Each SIV identifies a set of shared data among queries in Q. The SIVs, are processed only once, but reused by at least two queries in Q. The queries in Q, are rewritten into a different set of queries, Q ′. The collected sensor data from Q ′ and SIVs, are aggregated and returned as the processing results for the original set of queries in Q. The simulation results show that the proposed technique can effectively reduce the communication cost of multiple query processing in WSNs.KeywordsSensor NetworkSensor NodeWireless Sensor NetworkData TransmissionQuery ProcessingThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Nowadays, with the constantly increasing amount of data, we are facing a growing number of users, who are characterized by a frequent and a massively concurrent data access. The large number of users pose multiple query optimization problems. In a distributed data warehousing system such as Hadoop/Hive, queries are evaluated one at a time and processed with the MapReduce paradigm. The massive query execution usually overloads and slows down the entire distributed environment mainly due to multiple data scan tasks. In this paper we aim to optimize the multiple query execution performance on Hive. We propose Shared Distributed Datasets (\( S2D \)), a method that dynamically looks for and shares common data among queries. The evaluation shows that, compared to Hive, \( S2D \) consumes on average 20% less memory in the Map-scan task and it is 12% faster regarding the execution time of interactive and reporting queries from TPC-DS.

The existing key aggregate searchable encryption (KASE) schemes allow searches on the encrypted dataset using a single query trapdoor, with a feature to delegate the search rights of multiple files using a constant-size key. However, the limitation of the existing KASE schemes is they only support the exact keyword match $$ (Keyword = Value)$$ search query for a single query keyword. In addition, to perform a conjunctive keyword search using the existing KASE schemes, the user requires to submit different trapdoors for each individual keyword to the server. Considering the existence of numeric keywords, the KASE should support different types of query such as range query$$ (Value_1 \le Keyword \le Value_2) $$, comparison query($$ (Keyword \ge Value)$$ or $$ (Keyword \le Value)$$), and sort query. Therefore, a novel KASE scheme is proposed in this paper that supports the conjunctive range and sort query (i.e., CRSQ-KASE) on the encrypted dataset and enhances the query expressiveness of the existing KASE schemes. The proposed CRSQ-KASE scheme supports search among different keyword fields considering both numeric and non-numeric keywords. The user can search over shared encrypted dataset by giving arbitrary conjunctive queries $$ (Keyword_1 = Value_1\wedge Keyword_2 \le Value_2 \wedge \cdots \wedge Value_{n_1} \le Keyword_n \le Value_{n_2}) $$ using a single trapdoor. Furthermore, in the multi-owner setting, the user can search over multiple users’ documents set using a single query trapdoor. The theoretical, empirical, and security analyses show that the proposed CRSQ-KASE scheme performs better than the existing KASE schemes. To the best of our knowledge, the CRSQ-KASE is the first attempt that efficiently supports multi-dimensional, conjunctive keyword searches on the multi-owner encrypted dataset and gives the sorted search result.

Infrared (IR) images are essential to improve the visibility of dark or camouflaged objects. Object recognition and segmentation based on a neural network using IR images provide more accuracy and insight than color visible images. But the bottleneck is the amount of relevant IR images for training. It is difficult to collect real-world IR images for special purposes, including space exploration, military and fire-fighting applications. To solve this problem, we created color visible and IR images using a Unity-based 3D game editor. These synthetically generated color visible and IR images were used to train cycle consistent adversarial networks (CycleGAN) to convert visible images to IR images. CycleGAN has the advantage that it does not require precisely matching visible and IR pairs for transformation training. In this study, we discovered that additional synthetic data can help improve CycleGAN performance. Neural network training using real data (N = 20) performed more accurate transformations than training using real (N = 10) and synthetic (N = 10) data combinations. The result indicates that the synthetic data cannot exceed the quality of the real data. Neural network training using real (N = 10) and synthetic (N = 100) data combinations showed almost the same performance as training using real data (N = 20). At least 10 times more synthetic data than real data is required to achieve the same performance. In summary, CycleGAN is used with synthetic data to improve the IR image conversion performance of visible images.

MQO is a distributed multiple query processing middleware that can use resources available on the Grid to optimize query processing for data analysis and visualization applications. It does so by introducing one or more proxies that act as front-ends to a collection of backend servers. The basic idea behind this architecture is active semantic caching, whereby queries can leverage available cached results in the proxy either directly or through transformations. While this approach has been shown to speed up query evaluation under multi-client workloads, the caching infrastructure in the backend servers is not used well for query processing. Because this collective caching infrastructure scales with the number of servers, it is an important asset. In this paper, we describe a distributed multidimensional indexing scheme that enables the proxy to directly consider the cache contents available at the backend servers for query planning and scheduling. This approach is shown to produce better query plans and faster query response times as we experimentally demonstrate.

Fast estimates for aggregate queries are useful in database query optimization, approximate query answering and online query processing. Hence, there has been a lot of focus on “selectivity estimation”, that is, computing summary statistics on the underlying data and using that to answer aggregate queries fast and to a reasonable approximation. We present two sets of results for range aggregate queries, which are amongst the most common queries.First, we focus on a histogram as summary statistics and present algorithms for constructing histograms that are provably optimal (or provably approximate) for range queries; these algorithms take (pseudo-) polynomial time. These are the first known optimality or approximation results for arbitrary range queries; previously known results were optimal only for restricted range queries (such as equality queries, hierarchical or prefix range queries).Second, we focus on wavelet-based representations as summary statistics and present fast algorithms for picking wavelet statistics that are provably optimal for range queries. No previously-known wavelet-based methods have this property.We perform an experimental study of the various summary representations show the benefits of our algorithms over the known methods.

An effective query processing plays an important role in the uncertain data streams. Specifically, multiple top-k queries processing on uncertain data streams obtained from large applications of several fields such as sensor network monitoring and internet traffic control requires periodic execution of queries and sharing results among them. The system that monitors uncertain events in such data streams manipulates the top-k queries. Here the problem is that systems were not designed to allow query results sharing which in turn leads to high computation cost and inaccurate response from the system. To overcome these issues (Queries results sharing), the system using a sampling algorithm for sample the top possible worlds from well-known possible worlds based on their high probability. System uses an optimal dynamic programming approach that split the multiple queries into number of groups. Then the query groups are scheduled and planned for sharing results to yield minimum computation cost. A faster greedy algorithm is used to reduce the time and storage space of the top-k queries based on the greedy rule. Thus the proposed approach allows sharing computation among multiple top-k queries and generates best plan of query execution.

There is more and more commercial and research interest in nearest neighbor objects for location-based search from spatial database. Specially, a spatial keyword query takes a user location and user-supplied keywords as arguments and returns objects that is nearest k objects from user current location and textually relevant to the user required keyword. In these systems, user can type one or more word for required keyword. To find the result that contains at least one required keyword, it is important to support Boolean OR semantic keyword search on spatial database. In this paper we study how to answer such queries efficiently for both kNN query and range query. This paper proposes new index structure that combines K-d tree and inverted file. K-d tree efficiently support for both nearest neighbor and range queries. We also discuss Boolean OR Semantic keyword search for the user’s required keywords.

Graph Analytics (GA) is a rapidly growing field that holds significant importance in numerous applications that focuses on analysing the complex relations in a network. One of the crucial concepts in GA is Query Reachability which refers to the ability to find a path connecting two nodes (u, v) in graph G. The multiple attributes in reachability query processing play a significant role to discover some interesting patterns. Existing studies did not deal with processing complex queries in attributed graphs. To fulfill some of the challenges, We contributed to this paper with three ideas for solving complex queries, finding paths, and processing multiple queries in parallel and distributed paradigms. Therefore, 1) Proposed Distributed Set Reachability Query in Attributed Networks (DSRQ_AN) algorithm using distributed computing through the Neo4j graph data platform and Cypher Query Language, 2) Finding the path to the given complex queries, and 3) Proposed a Multiple Query Processing in Parallel and Distributed (MQP_PD) algorithm to execute in a parallel environment. This paper assessed the performance of the proposed approaches through a parallel paradigm on Neo4j slaves. Furthermore, the demonstrated results of parallel execution outperform sequential execution for multiple queries on attributed graphs.

Accurately modelled computer-generated data can be used in place of real-world signals for the design, test and validation of signal processing techniques in situations where real data is difficult to obtain. Bio-signal processing researchers interested in working with fNIRS data are restricted due to the lack of freely available fNIRS data and by the prohibitively expensive cost of fNIRS systems. We present a simplified mathematical description and associated MATLAB implementation of model-based synthetic fNIRS data which could be used by researchers to develop fNIRS signal processing techniques. The software, which is freely available, allows users to generate fNIRS data with control over a wide range of parameters and allows for fine-tuning of the synthetic data. We demonstrate how the model can be used to generate raw fNIRS data similar to recorded fNIRS signals. Signal processing steps were then applied to both the real and synthetic data. Visual comparisons between the temporal and spectral properties of the real and synthetic data show similarity. This paper demonstrates that our model for generating synthetic fNIRS data can replicate real fNIRS recordings.

As Geographic Information Systems (GIS) technologies have evolved, more and more GIS applications and geospatial data are available on the web. Spatial objects in a given query range can be retrieved using spatial range query ? one of the most widely used query types in GIS and spatial databases. However, it can be challenging to retrieve these data from various web applications where access to the data is only possible through restrictive web interfaces that support certain types of queries. A typical scenario is the existence of numerous business web sites that provide their branch locations through a limited "nearest location" web interface. For example, a chain restaurant's web site such as McDonalds can be queried to find some of the closest locations of its branches to the user's home address. However, even though the site has the location data of all restaurants in, for example, the state of California, it is difficult to retrieve the entire data set efficiently due to its restrictive web interface. Considering that k-Nearest Neighbor (k-NN) search is one of the most popular web interfaces in accessing spatial data on the web, this paper investigates the problem of retrieving geospatial data from the web for a given spatial range query using only k-NN searches. Based on the classification of k-NN interfaces on the web, we propose a set of range query algorithms to completely cover the rectangular shape of the query range (completeness) while minimizing the number of k-NN searches as possible (efficiency). We evaluated the efficiency of the proposed algorithms through statistical analysis and empirical experiments using both synthetic and real data sets.

Communication-efficient processing of multiple continuous aggregate queries

We propose strategies to efficiently execute a query workload, which consists of multiple related queries submitted against a scientific dataset, on a distributed-memory system in the presence of partial dataset replicas. Partial replication re-organizes and re-distributes one or more subsets of a dataset across the storage system to reduce I/O overheads and increase I/O parallelism. Our work targets a class of queries, called range queries, in which the query predicate specifies lower and upper bounds on the values of all or a subset of attributes of a dataset. Data elements whose attribute values fall into the specified bounds are retrieved from the dataset. If we think of the attributes of a dataset forming multi-dimensional space, where each attribute corresponds to one of the dimensions, a range query defines a bounding box in this multidimensional space. We evaluate our strategies in two scenarios involving range queries. The first scenario represents the case in which queries have overlapping regions of interest, such as those arising from an exploratory analysis of the dataset by multiple users. In the second scenario, queries represent adjacent rectilinear sections that capture an irregular subregion in the multi-dimensional space. This scenario corresponds to a case where the user wants to query and retrieve a spatial feature from the dataset. We propose cost models and an algorithm for optimizing such queries. Our results using queries for subsetting and analysis of medical image datasets show that effective use of partial replicas can result in reduction in query execution times.