A method for automatic rule derivation to support semantic query optimization

Data stream management systems (DSMS) processing long-running queries over large volumes of stream data must typically deliver time-critical responses. We propose the first semantic query optimization (SQO) approach that utilizes dynamic substream metadata at runtime to find a more efficient query plan than the one selected at compilation time. We identify four SQO techniques guaranteed to result in performance gains. Based on classic satisfiability theory we then design a lightweight query optimization algorithm that efficiently detects SQO opportunities at runtime. At the logical level, our algorithm instantiates multiple concurrent SQO plans, each processing different partially overlapping substreams. Our novel execution paradigm employs multi-modal operators to support the execution of these concurrent SQO logical plans in a single physical plan. This highly agile execution strategy reduces resource utilization while supporting lightweight adaptivity. Our extensive experimental study in the CAPE stream processing system using both synthetic and real data confirms that our optimization techniques significantly reduce query execution times, up to 60%, compared to the traditional approach.

In this paper, semantic query optimization in distributed database systems is translated into a multilevel search process. The overall search process is decomposed in two main stages: (1) guided by the syntactic complexity of a query expression, search for an appropriate optimization strategy, (2) given this strategy, transform the query expression into an efficient distributed query evaluation plan. During the second stage, properties of the application being modeled are used to attack a number of problems: detecting inconsistent and redundant selection and join conditions, estimating intermediate and final results, defining and using fragmentation knowledge. An extensible knowledge-based architecture is described to accommodate a variety of existing and future optimization techniques.

Optimizing queries over XML streams has been an important and non-trivial issue with the emergence of complex XML stream applications such as monitoring sensor networks and online transaction processing. Our system, R-SOX, provides a platform for runtime query optimization based on dynamic schema knowledge embedded in the XML streams. Such information provides refined runtime schema knowledge thus dramatically enlarged the opportunity for schema-based query optimizations. In this demonstration, we focus on the following three aspects: (1) annotation of runtime schema knowledge; (2) incremental maintenance of run-time schema knowledge; (3) dynamic semantic query optimization techniques. The overall framework for runtime semantic query optimization, including several classes of dynamic optimization techniques, will be shown in this demonstration.

article Free Access Share on Containment of conjunctive queries: beyond relations as sets Authors: Yannis E. Ioannidis Univ. of Wisconsin, Madison Univ. of Wisconsin, MadisonView Profile , Raghu Ramakrishnan View Profile Authors Info & Claims ACM Transactions on Database SystemsVolume 20Issue 3Sept. 1995 pp 288–324https://doi.org/10.1145/211414.211419Published:01 September 1995Publication History 89citation759DownloadsMetricsTotal Citations89Total Downloads759Last 12 Months36Last 6 weeks6 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my Alerts New Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Semantic Query Optimisation makes use of the semantic knowledge of a database (rules) to perform query transformation. Rules are normally learned from former queries fired by the user. Over time, however, this can result in the rule set becoming very large thereby degrading the efficiency of the system as a whole. Such a problem is known as the utility problem. This paper seeks to provide a solution to the utility problem through the use of statistical techniques in selecting and maintaining an optimal rule set. Statistical methods have, in fact, been used widely in the field of Knowledge Discovery to identify and measure relationships between attributes. Here we extend the approach to Semantic Query Optimisation using the Chi-square statistical method which is integrated into a prototype query optimiser developed by the authors. We also present a new technique for calculating Chi-square, which is faster and more efficient than the traditional method in this situation.

A stream processing approach for temporal query processing and optimization is discussed. Given properly sorted data, the implementation of temporal joins and semijoins as stream processors can be very effective. Also discussed are the tradeoffs among sort orders, the amount of local workspace, and multiple scans over input streams; in particular, the authors are interested in the effect of sort ordering on the local workspace. They present stream processing algorithms for various temporal joins and semijoins, and their workspace requirements for various data sort orderings. Finally, it is shown how semantic query optimization can play an important and natural role in optimization of temporal queries. >

Integrity constraints are usually assumed to be permanent properties that must be satisfied by any database state. However, there are many situations requiring also temporary constraints, that is, constraints that must hold only for a single database state. In this paper we propose a schema to define both permanent and temporary constraints, that supports efficient constraint checking and semantic query optimization. The proposed schema associates integrity constraints to rules and queries, and it uses methods, that were originally defined for permanent constraints only, to perform constraint checking and semantic query optimization.

Semantic Query Optimization in an Automata-Algebra Combined XQuery Engine over XML Streams

Query optimizers nowadays draw upon many sources of information about the database to optimize queries. They employ runtime statistics in cost-based estimation of query plans. They employ integrity constraints in the query rewrite process. Primary and foreign key constraints have long played a role in the optimizer, both for rewrite opportunities and for providing more accurate cost predictions. More recently, other types of integrity constraints are being exploited by optimizers in commercial systems, for which certain semantic query optimization techniques have now been implemented. These new optimization strategies that exploit constraints hold the promise for good improvement. Their weakness, however, is that often the “constraints” that would be useful for optimization for a given database and workload are not explicitly available for the optimizer. Data mining tools can find such “constraints” that are true of the database, but then there is the question of how this information can be kept by the database system, and how to make this information available to, and effectively usable by, the optimizer. We present our work on soft constraints in DB2. A soft constraint is a syntactic statement equivalent to an integrity constraint declaration. A soft constraint is not really a constraint, per se, since future updates may undermine it. While a soft constraint is valid, however, it can be used by the optimizer in the same way integrity constraints are. We present two forms of soft constraint: absolute and statistical . An absolute soft constraint is consistent with respect to the current state of the database, just in the same way an integrity constraint must be. They can be used in rewrite, as well as in cost estimation. A statistical soft constraint differs in that it may have some degree of violation with respect to the state of the database. Thus, statistical soft constraints cannot be used in rewrite, but they can still be used in cost estimation. We are working long-term on absolute soft constraints. We discuss the issues involved in implementing a facility for absolute soft constraints in a database system (and in DB2), and the strategies that we are researching. The current DB2 optimizer is more amenable to adding facilities for statistical soft constraints. In the short-term, we have been implementing pathways in the optimizer for statistical soft constraints. We discuss this implementation.

The number of papers on database system education has been close to zero at recent conferences. As database systems are a fundamental part of computer science curricula this comes as a surprise. The apparent draught in new ideas in database system education is even more surprising given the fact that this area of computer science has arguably undergone its most significant changes for more than 30 years within the last 5 years. Many new database system paradigms (most of them appearing under the NoSQL label) have been introduced for various reasons. Some of these massively distributed database systems facilitate unprecedented data processing capabilities often labeled as BigData. In order to offer a sustainable and practical CS education, at least some of those new paradigms and methodologies should probably be included into database education. This BoF can be a starting point for a discussion about which of those deserve inclusion into a typical CS curriculum and (even more interestingly) at what time in the curriculum and by replacing what other topics. Depending on interest of the audience a decision could be made as to what is the correct technology support for continuing the discussion beyond the conference. The BoF will also offer participants a forum to share their opinions, ideas and experiences on recent and future changes in database system education in general.

Compared to their fundamental part of computer science curricula the small number of papers on database system education comes as a surprise. The apparent draught in new ideas in database system education is even more surprising given the fact that this area of computer science has arguably undergone its most significant changes for more than 30 years within the last 5 to 10 years. Many new database system paradigms (most of them appearing under the NoSQL label) have been introduced for various reasons. Some of these massively distributed database systems facilitate unprecedented data processing capabilities often labeled as BigData. In order to offer a sustainable and practical CS education, at least some of those new paradigms and methodologies should be included into database education. The BoF is meant to continue the discussion from the 2014 BoF about which of those deserve inclusion into a typical CS curriculum and (even more interestingly) at what time in the curriculum and by replacing what other topics. It shall offer participants a forum to share their opinions and ideas on recent and future changes in database system education as well as to present practical experiences with new concepts picked up at last year's BoF or elsewhere.

Interacting with Large Language Models (LLMs) via declarative queries is increasingly popular for tasks like question answering and data extraction, thanks to their ability to process vast unstructured data. However, LLMs often struggle with answering complex factual questions, exhibiting low precision and recall in the returned data. This challenge highlights that executing queries on LLMs remains a largely unexplored domain, where traditional data processing assumptions often fall short. Conventional query optimization, typically cost-driven, overlooks LLM-specific quality challenges such as contextual understanding. Just as new physical operators are designed to address the unique characteristics of LLMs, optimization must consider these quality challenges. Our results highlight that adhering strictly to conventional query optimization principles fails to generate the best plans in terms of result quality. To tackle this challenge, we present a novel approach to enhance SQL results by applying query optimization techniques specifically adapted for LLMs. We introduce a database system, GALOIS, that sits between the query and the LLM, effectively using the latter as a storage layer. We design alternative physical operators tailored for LLM-based query execution and adapt traditional optimization strategies to this novel context. For example, while pushing down operators in the query plan reduces execution cost (fewer calls to the model), it might complicate the call to the LLM and deteriorate result quality. Additionally, these models lack a traditional catalog for optimization, leading us to develop methods to dynamically gather such metadata during query execution. Our solution is compatible with any LLM and balances the trade-off between query result quality and execution cost. Experiments show up to 144% quality improvement over questions in Natural Language and 29% over direct SQL execution, highlighting the advantages of integrating database solutions with LLMs.

The role of a query optimizer in a database system is to find the best execution plan for a given SQL statement based on statistics about the objects related to the tables referenced in the statement. These statistics include the tables themselves, their indexes, and other derived objects. Statistics include the number of rows, space utilization on disk, distribution of column values, etc. Optimization also relies on system statistics, such as the I/O bandwidth of the storage sub-system. All of this information is fed into a cost model. The cost model is used to compute the cost whenever the query optimizer needs to make a decision for an access path, join method, join order, or query transformation. The optimizer picks the alternative that yields the lowest cost. The quality of the final execution plan depends primarily on the quality of the information fed into the cost model as well as the cost model itself. In this paper, we discuss two of the problems that affect the quality of execution plans generated by the query optimizer: the cardinality of intermediate results and host variable values. We will give details of the solutions we introduced in Oracle 11g. Our approach establishes a bridge from the SQL execution engine to the SQL compiler. The bridge brings valuable information to help the query optimizer assess the impact of its decisions and make better decisions for future executions of the SQL statement. We illustrate the merits of our solutions based on experiments using the Oracle E-Business Suite workload.

S.70-79

A multidatabase system (MDBS) is a database system which integrates pre-existing databases, called component local database systems (LDBSs), to support global applications accessing data at more than one LDBS. An important research issue in MDBS is query optimization. The query optimization problem in MDBS is quite different from the case of distributed database system (DDBS) since, due to schema heterogeneity and local autonomy of component LDBSs, is not possible to assume that the query optimizer has a complete information on the execution cost and database statistics. In this paper we present a distributed query optimization algorithm that works under very general assumptions for MDBSs with relational global data model. The algorithm is based on the idea of delegating the evaluation of the execution cost of the elementary steps in a query execution plan to the LDBS where the computation is performed. The optimization process is organized as a sequence of steps, in which at each step all LDBSs work in parallel to evaluate the cost of execution plans for partial queries of increasing size, and send their cost estimates to the other LDBS that need them for the next step. The computation is totally distributed, and organized in order to perform no duplicate computation, and to discard as soon as possible the execution plans that may not lead to an optimal solution.KeywordsExecution ModelExecution PlanQuery OptimizationQuery ExecutionExecution CostThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.