Framework for query optimization in distributed statistical databases

In epidemiology it is extremely useful to compare the observed trends of various phenomena with the expected trends in order to find out abnormal morbid phenomena: in statistical databases (SDBS) this means performing table manipulation on aggregated data (macro data). The table manipulation are often implemented in different ways. since different aggregate fimctions are used to generate different kinds of macro data (data type) from disaggregated data. Therefore, logically similar manipulations at macro data level need to be defined separately and ad-hoc by the user for each data type. We propose to model macro data using an object-oriented approach with an instsnceinheritance mechanism, which allows the user to manage an SDB without having to explicitly deal with the different data types (automatic data type management). In the paper we describe the static and dynamic properties of our object oriented-model; the metaschema of statistical database system embodying such concepts (ADAMS: Aggregated DAta Management System) and the advantages of our system are

Since the very beginning of query processing in database systems, cost-based query optimization has been the essential strategy for effectively answering complex queries on large documents. XML documents can be efficiently stored and processed using native XML database management systems. Even though such systems can choose from a huge repertoire of join operators (e. g., Structural Joins and Holistic Twig Joins) and various index access operators to efficiently evaluate queries on XML documents, the development of full-fledged XML query optimizers is still in its infancy. Especially the evaluation of complex XQuery expressions using these operators is not well understood and needs further research. The extensible, rule-based, and cost-based XML query optimization framework proposed in this chapter, serves as a testbed for exploring how and whether well-known concepts from relational query optimization (e. g., join reordering) can be reused and which new techniques can make a significant contribution to speed-up query execution. Using the best practices and an appropriate cost model that will be developed using this framework, it can be turned into a robust cost-based XML query optimizer in the future.

Optimizing queries over XML streams has been an important and non-trivial issue with the emergence of complex XML stream applications such as monitoring sensor networks and online transaction processing. Our system, R-SOX, provides a platform for runtime query optimization based on dynamic schema knowledge embedded in the XML streams. Such information provides refined runtime schema knowledge thus dramatically enlarged the opportunity for schema-based query optimizations. In this demonstration, we focus on the following three aspects: (1) annotation of runtime schema knowledge; (2) incremental maintenance of run-time schema knowledge; (3) dynamic semantic query optimization techniques. The overall framework for runtime semantic query optimization, including several classes of dynamic optimization techniques, will be shown in this demonstration.

The domain of statistical and scientific databases is targeted here, and the class of aggregate queries which are very often encountered in this domain is considered. Such a query is aimed at retrieving some aggregate characteristics of the raw data. The tree-based statistics access method (TBSAM), which provides support for the efficient processing of aggregate queries, is presented. It is related to the B/sup +/-tree and also processes the B/sup +/-tree's efficient update properties. Complementing TBSAM is the provision of a grouped update algorithm for minimizing expensive indexed database updates. >

Decision support and knowledge discovery systems often compute aggregate values of interesting attributes by processing a huge amount of data in very large databases and/or warehouses. In particular, iceberg query is a special type of aggregation query that computes aggregate values above a user-provided threshold. Usually, only a small number of results will satisfy the threshold constraint. Yet, the results often carry very important and valuable business insights. Because of the small result set, iceberg queries offer many opportunities for deep query optimization. However, most existing iceberg query processing algorithms do not take advantage of the small-result-set property and rely heavily on the tuple-scan-based approach. This incurs intensive disk accesses and computation, resulting in long processing time especially when data size is large. Bitmap index, which builds one bitmap vector for each attribute value, is gaining popularity in both column-oriented and row-oriented databases in recent years. It occupies less space than the raw data and gives opportunities for more efficient query processing. In this paper, we exploited the property of bitmap index and developed a very effective bitmap pruning strategy for processing iceberg queries. Our index-pruning-based approach eliminates the need of scanning and processing the entire data set (table) and thus speeds up the iceberg query processing significantly. Experiments show that our approach is much more efficient than existing algorithms commonly used in row-oriented and column-oriented databases.

In structural vector autoregressive (SVAR) modeling, sometimes the identifying restrictions are insufficient for a unique specification of all shocks. In this paper it is pointed out that specific distributional assumptions can help in identifying the structural shocks. In particular, a mixture of normal distributions is considered as a possible model that can be used in this context. Our model setup enables us to test restrictions which are just-identifying in a standard SVAR framework. The results are illustrated using a U.S. macro data set and a system of U.S. and European interest rates.

This study examines the impact of innovation on employment in the Turkish labor market between the years 1991-2021, using monthly patent grants and annual R&D expenditure statistics. As for empirical technique, ARDL (autoregressive distributed lag) approach is used. The reason for choosing this approach is that it separates the long-term and short-term results and gives better results in analysis with a lower number of observations than other methods. The difference between the results of this study from the literature is that the analysis was performed in two different time periods, with two different proxy variables, and they gave the same result as proof of the robustness of the results. When the long-run model and the short-run model are investigated separately, it is found that while the effect of innovation on employment is negative in the short-run, it turns out to be positive in the long-run. Thus, during the period 1991-2021 in the Turkish labor market, while innovation might negatively affect employment levels to some extent in the short run, innovation could exert a more structural and sustainable positive impact on employment levels in the long run. In the short-run, the negative effect of innovation on employment can be seen as a kind of creative destruction, but in the long-run, the positive effect of innovation supports the hypothesis that the increase in the education and training levels of workers along with the profit and productivity provided by innovation increases employment by adapting workers to innovation. The aim of this study is to make an inference with macro data sets but, using micro-level, firm data may provide significant results on the effect of innovation on employment.

Despite the proliferation of Web services, the field of service research is still in its infancy, where there has been little foundational work to date. In this chapter, we present the foundational service framework that helps address some fundamental research issues on Web services [44, 74]. The presented framework offers a systematic approach that enables users to achieve an op- timal Service Execution Plan (SEP) by submitting declarative service queries. Figure 3.1 shows the overall architecture of the service query optimization framework. At the bottom, we have the service space for a given application domain that consists of the actual service providers, a.k.a., service instances. The integration layer helps resolve the discrepancy of the service schemas and provides a uniformed view of the service space. It can achieve this by ap- plying the existing data integration techniques [40], which are not the focus of this work. The QoWS manager deals with the collection of quality related information from the service instances. The quality data will be used by the query optimizer to make selection from the competing service providers. The service model captures a set of key features of Web services that lay out a foundation for service query specification, processing, and optimization. The service calculus enables users to use declarative service queries to locate Web services, which is more precise and reliable than the keyword based search. The query processor generates SEPs which can be used by the users to invoke services.

Query optimization has been an active area of research ever since the first relational systems were implemented. In the last few years, research in the area has experienced renewed impulse, thanks to new development like data warehousing. In this article, we overview some of the recent advances made in complex query optimization. This article assumes knowledge of SQL and relational algebra, as well as the basics of query processing; in particular, the user is assumed to understand cost optimization of basic SQL blocks (Select-Project-Join queries). After explaining the basic unnesting approach to provide some background, we overview three complementary techniques: source and algebraic transformations (in particular, moving outerjoins and pushing down aggregates), query rewrite (materialized views), new indexing techniques (bitmap and join indices), and different methods to build the answer (online aggregation and sampling). Throughout this article, we will use subquery to refer to a nested SQL query and outer query to refer to an SQL query that contains a nested query. The TPCH benchmark database (TPC, n.d.) is used as a source of examples. This database includes (in ascending size order) tables Nation, Customer (with a foreign key for Nation), Order (with a foreign key for Customer), and Lineitem (with a foreign key for Order). All attributes from a table are prefixed with the initial of the table’s name (“c_” for Customer, and so on).

Managing the data in a composite materials data base, such as the data base required for MIL-HDBK-17, is a complex task. MIL-HDBK-17, “Polymer Matrix Composites,” is a fully coordinated military document currently supported by the Army, the Air Force, and the Federal Aviation Administration. The guidelines in Volume I of the handbook provide the requirements for a statistically based data base of polymer matrix composite materials properties. The MIL-HDBK-17 data base must be capable of handling property values for fibers, matrix materials, prepregs, laminae, and simple laminates. A prepreg is a ready-to-mold or -cure material in sheet form. It may be fiber, cloth, or a mat impregnated with resin and stored for use. The properties of each of these types of materials can be further divided into chemical, physical, and mechanical categories. While the material types and property categories readily provide an outline for a data base, a number of properties must overlap the data base boundaries to adequately describe a given material. Statistical analysis requirements for evaluating batch-to-batch variability and an interest in pooling data from various sources dictate the analysis and storage of all raw data in addition to the statistical parameters reported in MIL-HDBK-17. The capabilities for tracking these data use four types of files—raw data files, statistical parameter files, analysis files, and files that record the history of pooled data. In addition, a hard-copy record is retained for both analysis files and summary tables. The analysis files report raw data, the method of analysis as determined by batch-to-batch variability and the distribution of the data, and the resulting statistical parameters. The summary tables contain the statistical parameters in the MIL-HDBK-17 format tabulated by orientation, loading, temperature, and moisture conditions. These tabulated values can be readily transferred to a word processor for incorporation into the printed handbook. Data-base management on personal computers was selected for several reasons. PC availability is desirable for the statistical analysis program since each company or laboratory that submits data to MIL-HDBK-17 will want access to this program. Management of the data base itself is conducted through the use of METSEL2. METSEL2 is an interactive data-base program originally designed to store metals properties and currently being revised to include composite materials. Storage and manipulation of the statistical parameters are enhanced by using this program. The in-house data base is also considered a trial run for a data base that may be distributed to the members of the coordination group and other appropriate organizations. The establishment of this permanent, accessible data base depends on factors that have not yet been resolved, such as the export control of data.

The λ-DB project at the University of Texas at Arlington aims at developing frameworks and prototype systems that address the new query optimization challenges for object-oriented and object-relational databases, such as query nesting, multiple collection types, methods, and arbitrary nesting of collections. We have already developed a theoretical framework for query optimization based on an effective calculus, called the monoid comprehension calculus [4]. The system reported here is a fully operational ODMG 2.0 [2] OODB management system, based on this framework. Our system can handle most ODL declarations and can process most OQL query forms. λ-DB is not ODMG compliant. Instead it supports its own C++ binding that provides a seamless integration between OQL and C++ with low impedance mismatch. It allows C++ variables to be used in queries and results of queries to be passed back to C++ programs. Programs expressed in our C++ binding are compiled by a preprocessor that performs query optimization at compile time, rather than run-time, as it is proposed by ODMG. In addition to compiled queries, λ-DB provides an interpreter that evaluates ad-hoc OQL queries at run-time. The λ-DB system architecture is shown in Figure 1. The λ-DB evaluation engine is written in SDL (the SHORE Data Language) of the SHORE object management system [1], developed at the University of Wisconsin. ODL schemas are translated into SDL schemas in a straightforward way and are stored in the system catalog. The λ-DB OQL compiler is a C++ preprocessor that accepts a language called λ-OQL, which is C++ code with embedded DML commands to perform transactions, queries, updates, etc. The preprocessor translates λ-OQL programs into C++ code that contains calls to the λ-DB evaluation engine. We also provide a visual query formulation interface, called VOODOO, and a translator from visual queries to OQL text, which can be sent to the λ-DB OQL interpreter for evaluation. Even though a lot of effort has been made to make the implementation of our system simple enough for other database researchers to use and extend, our system is quite sophisticated since it employs current state-of-the-art query optimization technologies as well as new advanced experimental optimization techniques which we have developed through the years, such as query unnesting [3]. The λ-DB OODBMS is available as an open source software through the web at http://lambda.uta.edu/lambda-DB.html

In this paper, we present a translator from a relevant subset of SQL into relational algebra. The translation is syntax-directed, with translation rules associated with grammar productions; each production corresponds to a particular type of SQL subquery.

With the explosive growth of big data, data-intensive analysis services need a large data storage and high efficient parallel query processing techniques in cloud using lower-end machines, because most organizations adopt inexpensive low-end clusters. A key challenge is to optimize query processing in such a cloud environment. In this paper, we present a new hybrid data access architecture, called HyDB, for providing data-intensive analysis services, which is featured with a distributed data storage, parallel data access, and query optimization methods. First, we propose a new data partitioning method based on both workloads and co-located resources. The data partitioning method achieves higher consolidation and outperforms the existing approaches. Second, we provide a new parallel access method which includes parallel query processing, optimal query plan generation, and optimal path selection by using a plan tree pruning technique. We have implemented HyDB. Finally, we conduct extensive experimental studies and confirm the efficiency of our HyDB.

Most of the domains of science are facing an explosion in the data that they have to collect and process in order to conduct research. This is true for both: scientific domains dealing with experimental data, e.g. biology, sociology, astronomy etc, but also scientific domains dealing with simulation data, e.g. seismology, physics etc. To maximize the potential outcome of scientific data analysis, respective data management applications need to fulfil the following coarse tasks: fast on-demand data processing, and effective storage and consolidation of diverse data collections. These two tasks are in general hard to realize because of: (a) the big data size, (b) the diversity of data formats, (c) their conceptual dependencies, (d) disperse data locations, and (e) intensive and systematic nature of scientific queries. We present the characteristics of big scientific data collections and their necessities in terms of data management. Based on this discussion, we discuss the structure of a framework for the processing and consolidation of heterogeneous scientific data collections. Such a framework aims to mediate between the user and a set of available data management technologies, such as relational DBMSs, key-value stores and column stores, in order to efficiently direct data management operations (insertions, updates) and especially requests (queries) to the appropriate data management application. The framework aims to distribute, dissect, and schedule data management actions, as well as integrate results, in a way that reduces response time. This entails the accommodation of methods for the selective parallelism and serialization depending on partial results and response times. Also, this entails the accommodation of methods for the gradual alteration of data formats and storage, e.g. storage of semi-structured data or raw data in files into relational databases. Furthermore, we discuss the processing of scientific query bulks or workflows with the possibility to retrieve early partial results and calibrate query parameters.

The Relational Model first came into vogue in the early 1980's. It was based on the simplifying idea that all data could be modeled as mathematical relations (tables in "normal form"). Permissible operations on this table data structure were specified by the relation algebra and calculus. The Relational Model led to years of research in areas such as query languages, query optimization, transaction models, and database design methodologies. This research has dominated DBMS conferences for the last fifteen years and has also lead to major products offerings in wide-spread use in the computer industry today.While the Relational revolution was happening in the DBMS community there was a minority opinion emerging from the object community. This minority opinion surfaced in heated debates and panels at DBMS conferences where it was often pitted against "relational purists." The object proponents wanted to discuss storing and retrieving complex objects and relationships in databases while the relational purists insisted on maintaining "mathematical purity and simplicity". These panels were often some of the most acrimonious (and entertaining) at these conferences and many thought there was no way to bridge the chasm between the two schools of thought.However times changes and so do the realities of the commercial world. Object languages such as C++, Smalltalk, and Java have become de facto standards. The Internet and the PC have increased the demand for complex data types. And the relational model is evolving to accommodate these realities.All of the major RDBMS vendors have announced plans for, or are already shipping, Object-Relational DBMS (ORDBMS) products. A natural question for the object community is how well these new products will address the well-known "impedance mismatch" between a pure object model and the relational model. For example, how does one make Java, Smalltalk, or C++ objects persist using an ORDBMS? Can one search for these objects in the database using their methods?The vendors for ORDBMS are claiming well-known OO features in their implementations, such as extensible data types, inheritance, object identity, and object language bindings. This panel will allow the major vendors to explain how those features match up with the kinds of object models that OO developers are accustomed to. The panel will consist of representatives from three of the major ORDBMS vendors as well as a representative of "pure object think". The vendor representatives will each present a brief overview of the object-related features of their products and will explain why OO programmers are going to have an easier time with these ORDBMSs. This will be followed by counterpoint discussion from the pure object thinker.