Optimal and approximate computation of summary statistics for range aggregates

The in--network aggregation paradigm in sensor networks provides a versatile approach for evaluating aggregate queries. Traditional approaches need a separate aggregate to be computed and communicated for each query and hence do not scale well with the number of queries. Since approximate query results are sufficient for many applications, we use an alternate approach based on summary data--structures. We consider two kinds of aggregate queries: value range queries that compute the number of sensors that report values in the given range, and location range queries that compute the sum of values reported by sensors in a given location range. We construct summary data--structures called linear sketches, over the sensor data using in--network aggregation and use them to answer aggregate queries in an approximate manner at the base--station. There is a trade--off between accuracy of the query results and lifetime of the sensor network that can be exploited to achieve increased lifetimes for a small loss in accuracy. Experimental results show that linear sketching achieves significant improvements in lifetime of sensor networks for only a small loss in accuracy of the queries. Further, our approach achieves more accurate query results than the other classical techniques using Discrete Fourier Transform and Discrete Wavelet Transform.

Data encryption is necessary to keep user information secure and private on the cloud. However, adversaries can still learn valuable information about the encrypted data by observing data access patterns. To solve this issue, Oblivious RAMs (ORAMs) are proposed to hide access patterns. However, ORAMs are expensive and not suitable for deployment in a large database. In this work, we propose a range query algorithm while providing data access pattern protection based on uniform access frequency. In the preprocessing, multiple key-value pairs in the database are grouped and stored in each storage module, and we make copies for frequently accessed key-value pairs and also add some dummy key-value pairs on each storage module. In the online query processing, according to the range query length of the received query access request, we visit the specific storage module for the query and obtain the query result. Based on the techniques above, our method makes the uniform distribution of access frequency of data blocks in the database and achieves a security guarantee as strong as the state-of-the-art method. Compared with data queries that do not provide data access pattern protection, the ratio of network communication overhead is constant rather than logarithmic in ORAMs.

The existing privacy-preserving aggregation query processing methods in sensor networks rely on pre-established network topology and require all nodes in the network to participate in query processing. Maintaining the topology results in a large amount of energy overhead, and in many cases, the user is interested only in the aggregated query results of some areas in the network, and thus, the participation of the entire network node is not necessary. Aiming to solve this problem, this article proposes a spatial range aggregation query algorithm for a dynamic sensor network with privacy protection (energy-efficient privacy-preserving data aggregation). The algorithm does not rely on the pre-established topology but considers only the query area that the user is interested in, abandoning all nodes to participate in distributing the query messages while gathering the sensory data in the query range. To protect node data privacy, Shamir’s secret sharing technology is used to prevent internal attackers from stealing the sensitive data of the surrounding nodes. The analysis and experimental results show that the proposed algorithm outperforms the existing algorithms in terms of energy and privacy protection.

With the increasing popularity of the peer-to-peer (P2P) computing paradigm, many general range query schemes for distributed hash table (DHT)-based P2P systems have been proposed in recent years. Although those schemes can provide range query capability without modifying the underlying DHTs, they have the query delay depending on both the scale of the system and the size of the query space or the specific query, and thus cannot guarantee to return the query results in a bounded delay. In this paper, we propose Armada, an efficient range query processing scheme to support delay-bounded single-attribute and multiple-attribute range queries. It is the first delay-bounded general range query scheme on constant-degree DHTs, and can return the results for any range query within 2logN hops in a P2P system with N peers. Results of analysis and simulations show that the average delay in Armada is less than logN, and the average message cost of single-attribute range queries is about logN+2n 2 (n is the number of peers that intersect with the query). These results are very close to the lower bounds on delay and message cost of range queries over constant-degree DHTs.

Range queries (RQ) are crucial for analytical workloads, with indexing support being essential to minimize storage accesses. However, indexing support for RQ faces several challenges. Existing tree-based indexes have suboptimal RQ performance and memory consumption when long-running RQs and short-lived updates coexist. Bitmap indexes show promise in overcoming these challenges because of their small size and their succinct and readily available query result; however, they have inherent limitations: they primarily target read-only, low-cardinality attributes. In this paper, we propose Ra nge Queries with Bit map Indexing (RABIT), a solution that addresses these shortcomings. Our design relies on three principles. First, we propose Group Encoding (GE), a novel encoding scheme that provides fast RQs and real-time updates while maintaining high compressibility. Second, we propose an efficient bitvector merging mechanism for GE. Depending on the bit density of each bitvector, we merge it in either its compressed or decompressed form, leveraging SIMD instructions when beneficial. Third, we propose a multi-layer update framework that enables lightweight multi-versioning and native index-only scans, while retaining single-versioned bitvectors, significantly reducing memory usage. Putting everything together, RABIT provides efficient point and range queries on attributes with any cardinality in tables ranging from read-only to frequently updated, unlocking the use of bitmap indexing as a general-purpose secondary index. We demonstrate that RABIT accelerates key DBMS operators (Scan, Join, and Aggregation), achieving substantial performance gains. In a row-store DBMS under HTAP workloads, RABIT offers up to 2.2x faster RQs, 530x faster updates, and 118x smaller footprint than tree indexes. In columnar DuckDB, RABIT accelerates TPC-H queries by up to 14.8x.

In decision support applications, the ability to provide fast approximate answers to aggregation queries is desirable. One commonly-used technique for approximate query answering is sampling. For many aggregation queries, appropriately constructed biased (non-uniform) samples can provide more accurate approximations than a uniform sample. The optimal type of bias, however, varies from query to query. In this paper, we describe an approximate query processing technique that dynamically constructs an appropriately biased sample for each query by combining samples selected from a family of non-uniform samples that are constructed during a pre-processing phase. We show that dynamic selection of appropriate portions of previously constructed samples can provide more accurate approximate answers than static, non-adaptive usage of uniform or non-uniform samples.

As more and more data become accessible, ranking query semantics such as ranked retrieval, possibly combined with Boolean query conditions, has gained a lot of attention lately. As the formulation of such queries is known to be difficult, we aim at providing quick approximate answers as cues for interactive query refinements. Toward the goal, we study approximate answering techniques for Boolean+ranking queries. While approximate query answering has been studied for Boolean-only queries, we observed that a straightforward extension of this work for advanced queries incurs prohibitive overheads. We thus propose a systematic framework which significantly outperforms such a naive extension. We also empirically validate the effectiveness and efficiency of our framework.

We are addressing the problem of query processing in large sensor networks. Each sensor produces a large amount of streaming data and it may not be possible to store all the data. By caching some of the data in aggregated format, we will be able to answer queries referring to past data. We are investigating a hierarchical caching model where summarized data is cached. The granularity of aggregation becomes coarser as we move up the levels, starting from the actual data of the immediate past stored at the lowest level. We categorize queries into two types: queries that can be answered exactly and those that can be answered approximately. We have provided an analysis of the conditions under which the exact and approximate answers can be provided for a given query. When the query answer is approximate, an estimation of the error is provided.

In the Database-As-a-Service (DAS) model, clients store their database contents at servers belonging to potentially untrusted service providers. To maintain data confidentiality, clients need to outsource their data to servers in encrypted form. At the same time, clients must still be able to execute queries over encrypted data. One prominent and fairly effective technique for executing SQL-style range queries over encrypted data involves partitioning (or bucketization) of encrypted attributes.However, executing aggregation-type queries over encrypted data is a notoriously difficult problem. One well-known cryptographic tool often utilized to support encrypted aggregation is homomorphic encryption; it enables arithmetic operations over encrypted data. One technique based on a specific homomorphic encryption function was recently proposed in the context of the DAS model. Unfortunately, as shown in this paper, this technique is insecure against ciphertext-only attacks. We propose a simple alternative for handling encrypted aggregation queries and describe its implementation. We also consider a different flavor of the DAS model which involves mixed databases, where some attributes are encrypted and some are left in the clear. We show how range queries can be executed in this model.

Large Language Models (LLMs) are attracting an enormous amount of interest at the moment in many domains. Their general nature, and ability to "understand" natural language, has already stimulated multiple areas of research at our company. Here we successfully demonstrate a Natural Language querying system, which is able to search a large repository of unstructured exploration data. The system supports follow up querying on the returned results, plus automatic summarization of content. The system is integrated into our novel end-to-end data-mining platform, which continuously mines our unstructured exploration data for new changes and indexes the results. Important in our method are the enrichment processes that occur prior to use of the LLM. Our approach avoids usual "chunking" techniques, which in our experience results in inferior results, especially in the multiple domain areas of Exploration. By integrating our novel ontology-model AI in the enrichment of the initial Index, we drastically boost the performance of search resulting from the LLM steps. In order to perform the search, key parts of our unstructured data, plus the query itself, need to be transformed into a vector form. This is performed using the embedding feature of the LLM. For this work, we had around 500,000 embeddings to calculate. To improve performance these were indexed in a leading Analytics Engine as a vector object, allowing fast search via cosine or Euclidian similarity. A custom dashboard was made to allow fresh searches of the vector datastore to be returned for further analysis. Our current search time across 500,000 embeddings is under 20 milli-seconds. Our custom dashboard returns the top matches for further interrogation and analysis. This includes follow-up Natural Language question support on the returned matches for summarization tasks and other customised querying. Since our exploration-specific, ontology model is able to tag each piece of data with over 40 exploration-specific labels, we are able to cross-examine the LLM returned results with the tags. Agreement on a range of queries - ranging from targeted, highly specific questions to general, open-ended queries - was surprisingly good. Natural Language based querying of our unstructured data is opening a whole new approach to data discovery in our company. Tailoring it to the exploration domain has required specific domain expertise and a novel ontology-model be used to ensure relevant prompts and query results. Obtaining search results quickly has also required expertise and fine-tuning. Future directions include ingesting more data, scaling the support infrastructure and further capability enhancement.

Given an array A of n elements, we wish to support queries for the most frequent and least frequent element in a subrange [l, r] of A. We also wish to support updates that change a particular element at index i or insert/ delete an element at index i. For the range mode problem, our data structure supports all operations in O(n^{2/3}) deterministic time using only O(n) space. This improves two results by Chan et al. [Timothy M. Chan et al., 2014]: a linear space data structure supporting update and query operations in O~(n^{3/4}) time and an O(n^{4/3}) space data structure supporting update and query operations in O~(n^{2/3}) time. For the range least frequent problem, we address two variations. In the first, we are allowed to answer with an element of A that may not appear in the query range, and in the second, the returned element must be present in the query range. For the first variation, we develop a data structure that supports queries in O~(n^{2/3}) time, updates in O(n^{2/3}) time, and occupies O(n) space. For the second variation, we develop a Monte Carlo data structure that supports queries in O(n^{2/3}) time, updates in O~(n^{2/3}) time, and occupies O~(n) space, but requires that updates are made independently of the results of previous queries. The Monte Carlo data structure is also capable of answering k-frequency queries; that is, the problem of finding an element of given frequency in the specified query range. Previously, no dynamic data structures were known for least frequent element or k-frequency queries.

In order to answer efficiently range queries in 2-d R-trees, first we sort queries by means of a space filling curve, then we group them together, and finally pass them for processing. Initially, we consider grouping of pairs of requests only, and give two algorithms with exponential and linear complexity. Then, we generalize the linear method, grouping more than two requests per group. We evaluate these methods under different LRU buffer sizes, measuring the cache misses per query. We present experimental results based on real and synthetic data. The results show that careful query scheduling can improve substantially the overall performance of multiple range query processing.

This paper proposes a new application for data mining. It is discovery of user-interests from the user queries. Since queries themselves represent users' interests in nature without knowing the query results, we can discover user-interests from the users' queries. The user-interests plays an important role in improving the quality of information servers, and database performance tuning. In this paper, we focus on range queries on a continuous attribute. We propose an effective iterative algorithm to discover the most interest range, in the sense that the range is accessed by enough users, and is covered by the users' queries largestly on the average.

Representing the trajectories of mobile objects is a hot topic from the widespread use of smartphones and other GPS devices. However, few works have focused on representing trips over public transportation networks (buses, subway, and trains) where a user's trips can be seen as a sequence of stages performed within a vehicle shared with many other users. In this context, representing vehicle journeys reduces the redundancy because all the passengers inside a vehicle share the same arrival time for each stop. In addition, each vehicle journey follows exactly the sequence of stops corresponding to its line, which makes it unnecessary to represent that sequence for each journey. To solve data management for transportation systems, we designed a conceptual model that gave us a better insight into this data domain and allowed us the definition of relevant terms and the detection of redundancy sources among those data. Then, we designed two compact representations focused on users' trips (TTCTR) and on vehicle trips (AcumM), respectively. Each approach owns some strengths and is able to answer some queries efficiently. We include experimental results over synthetic trips generated from accurate schedules obtained from a real network description (from the bus transportation system of Madrid) to show the space/time trade-off of both approaches. We considered a wide range of different queries about the use of the transportation network such as counting-based or aggregate queries regarding the load of any line of the network at different times.

AMID: Approximation of MultI-measured Data using SVD