Database patchwork on the Internet

Distributed data processing is becoming a reality. Businesses want to do it for many reasons, and they often must do it in order to stay competitive. While much of the infrastructure for distributed data processing is already there (e.g., modern network technology), a number of issues make distributed data processing still a complex undertaking: (1) distributed systems can become very large, involving thousands of heterogeneous sites including PCs and mainframe server machines; (2) the state of a distributed system changes rapidly because the load of sites varies over time and new sites are added to the system; (3) legacy systems need to be integrated—such legacy systems usually have not been designed for distributed data processing and now need to interact with other (modern) systems in a distributed environment. This paper presents the state of the art of query processing for distributed database and information systems. The paper presents the “textbook” architecture for distributed query processing and a series of techniques that are particularly useful for distributed database systems. These techniques include special join techniques, techniques to exploit intraquery paralleli sm, techniques to reduce communication costs, and techniques to exploit caching and replication of data. Furthermore, the paper discusses different kinds of distributed systems such as client-server, middleware (multitier), and heterogeneous database systems, and shows how query processing works in these systems.

A heterogeneous database management system combines multiple dissimilar models of data within a single integrated system. The objective is to allow a user to access data independently of how it is actually organized. For example, a user may access a database as though it were stored relationally (i.e., in tables) [CODD70], even though it is actually stored as a CODASYL/DBTG or network database [CODA71]. In addition, different subpieces of the database may be organized under different data models. The heterogeneous database system must present these to the user as an integrated whole. The user's model of his data may be different from any of the models chosen to implement it.Rather than construct a new database system from scratch, we are interested in constructing a heterogeneous system out of existing systems. The key difficulties with this approach are: (1) the formulation of database design methods that are applicable to a variety of different data models, and (2) the development of techniques to translate programs and data between dissimilar data models. In this paper, we briefly describe how high level abstraction has been applied to these problems. The use of abstraction in database systems is related to the application of abstraction techniques in programming languages and artificial intelligence research.

We describe an approach for controlling certain unauthorized inferences in a multilevel secure distributed database management system. In such a system, two or more multilevel secure database management systems are connected via a trusted network. Furthermore, the environment that we have considered is a limited heterogeneous one where not all of the nodes handle the same accreditation ranges. In our approach, security constraints, which are rules that assign security levels to the data, are processed during the distributed query, update, and database design operations in such a way that users do not acquire information to which they are not authorized via logical deductions. We describe the design and implementation of the distributed inference controller which functions during the query operation.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

In a previous JDM issue, one of the Associate Editors issued a challenge to database researchers to “...encourage us to think about innovation and point out areas that ... are not only important but also open to innovation.†(Pirkul, 1995, p.33). In this editorial, I address one area that Pirkul explicitly identified as a topic for innovation, namely, Heterogeneous Databases. Below I present a novel framework for understanding Heterogeneous Databases and Database Systems. This framework identifies three dimensions of heterogeneity: the “Why†?, “How†?, and “What?†of heterogeneous database systems. I claim that exploring answers to these three questions will help practitioners (1) identify heterogeneous application requirements and map them to appropriate database capabilities, (2) evaluate heterogeneous application techniques that fulfill these necessary requirements, and (3) select products that support the required techniques.

Database technology is of great variety, structure diversity, solving data integration and exchanges sharing of the heterogeneous database system have became the important problem in all fields. Because of XML is the very important data exchange technology to realize data exchange and transmission ability, this paper mainly explores the data exchange problem how to use XML in heterogeneous database system, , and .Net platform have good support to XML and heterogeneous database system, so the combination of XML and.net, can realize data exchange of heterogeneous database system.

The issues and solutions for the interoperability of a class of heterogeneous databases and their database systems are expounded in two parts. Part I presented the data-sharing issues in federated databases and systems (Hsiao, 1992). The present article explores resource-consolidation issues. Interoperability in this context refers to data sharing among heterogeneous databases, and to resource consolidation of computer hardware, system software, and support personnel. Resource consolidation requires the presence of a database system architecture which supports the heterogeneous system software, thereby eliminating the need for various computer hardware and support personnel. The class of heterogeneous databases and database systems expounded herein is termed federated, meaning that they are joined in order to meet certain organizational requirements and because they require their respective application specificities, integrity constraints, and security requirements to be upheld. Federated databases and systems are new. While there are no technological solutions, there has been considerable research towards their development. This tutorial is aimed at exposing the need for such solutions. A taxonomy is introduced in our review of existing research undertakings and exploratory developments. With this taxonomy, we contrast and compare various approaches to federating databases and systems.

The issues and solutions for the interoperability of a class of heterogeneous databases and their database systems are expounded in two parts. Part I presents the data-sharing issues in federated databases and systems. Part II, which will appear in a future issue, explores resource-consolidation issues. Interoperability in this context refers to data sharing among heterogeneous databases, and to resource consolidation of computer hardware, system software, and support personnel. Resource consolidation requires the presence of a database system architecture which supports the heterogeneous system software, thereby eliminating the need for various computer hardware and support personnel. The class of heterogeneous databases and database systems expounded herein is termed federated, meaning that they are joined in order to meet certain organizational requirements and because they require their respective application specificities, integrity constraints, and security requirements to be upheld. Federated databases and systems are new. While there are no technological solutions, there has been considerable research towards their development. This tutorial is aimed at exposing the need for such solutions. A taxonomy is introduced in our review of existing research undertakings and exploratory developments. With this taxonomy, we contrast and compare various approaches to federating databases and systems.

Development of a dynamic data interchange scheme to support product design in agile manufacturing

Several aspects are given of the Remote-Exchange project at USC, which focuses on the controlled sharing and exchange of information among autonomous, heterogeneous database systems. The spectrum of heterogeneity which may exist among the components in a federation of database systems is examined, and an approach to accommodating such heterogeneity is described. An overview of the Remote-Exchange experimental system is provided.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Cost parameters and database statistics are the basis of query optimization techniques. However, in distributed and heterogeneous database systems, acquiring and treating information in order to help the optimization process are often tasks of a global query processor, which adapts its functionalities to a specific system architecture. Moreover, this acquisition process involves a large number of parameters and requires customized methods to retrieve data from specific sources. DIG (Distributed Information Gatherer) is a provider of data statistics and query costs that, through an independent and flexible service, aims to support global query optimization processing in distributed and heterogeneous database systems over autonomous data sources. We have developed a DIG prototype and experimented it with specific wrappers for a query middleware on both semi-structured data sources and an object DBMS.

The SIRIUS-DELTA architecture: A framework for co-operating database systems

In this paper we advocate that it is time for a radical rethinking of database systems design. Developers should be able to leverage high-level programming languages without having to pay a price in efficiency. To realize our vision of abstraction without regret , we present LegoBase, a query engine written in the high-level programming language Scala. The key technique to regain efficiency is to apply generative programming: the Scala code that constitutes the query engine, despite its high-level appearance, is actually a program generator that emits specialized, low-level C code. We show how the combination of high-level and generative programming allows to easily implement a wide spectrum of optimizations that are difficult to achieve with existing low-level query compilers, and how it can continuously optimize the query engine. We evaluate our approach with the TPC-H benchmark and show that: (a) with all optimizations enabled, our architecture significantly outperforms a commercial in-memory database system as well as an existing query compiler, (b) these performance improvements require programming just a few hundred lines of high-level code instead of complicated low-level code that is required by existing query compilers and, finally, that (c) the compilation overhead is low compared to the overall execution time, thus making our approach usable in practice for efficiently compiling query engines.

Abstraction without regret refers to the vision of using high-level programming languages for systems development without experiencing a negative impact on performance. A database system designed according to this vision offers both increased productivity and high performance instead of sacrificing the former for the latter as is the case with existing, monolithic implementations that are hard to maintain and extend. In this article, we realize this vision in the domain of analytical query processing. We present LegoBase, a query engine written in the high-level programming language Scala. The key technique to regain efficiency is to apply generative programming: LegoBase performs source-to-source compilation and optimizes database systems code by converting the high-level Scala code to specialized, low-level C code. We show how generative programming allows to easily implement a wide spectrum of optimizations, such as introducing data partitioning or switching from a row to a column data layout, which are difficult to achieve with existing low-level query compilers that handle only queries. We demonstrate that sufficiently powerful abstractions are essential for dealing with the complexity of the optimization effort, shielding developers from compiler internals and decoupling individual optimizations from each other. We evaluate our approach with the TPC-H benchmark and show that (a) with all optimizations enabled, our architecture significantly outperforms a commercial in-memory database as well as an existing query compiler. (b) Programmers need to provide just a few hundred lines of high-level code for implementing the optimizations, instead of complicated low-level code that is required by existing query compilation approaches. (c) These optimizations may potentially come at the cost of using more system memory for improved performance. (d) The compilation overhead is low compared to the overall execution time, thus making our approach usable in practice for compiling query engines.

Query optimization in heterogeneous database systems is not always possible since the component DBMS may not have the ability to transmit necessary information. However, these systems need query optimization because the cost of transmitting large quantities of data across diverse databases is very high. We propose a query strategy which uses hypothesis testing to determine which of two sets of data are larger. Our experiments show that this strategy is very likely to select the smaller set when the sampling results fall outside a region of uncertainty we call the “grey zone.” This provides query optimization without transmission of database statistics.

As data collections become larger and larger, data loading evolves to a major bottleneck. Many applications already avoid using database systems, e.g., scientific data analysis and social networks, due to the complexity and the increased data-to-query time. For such applications data collections keep growing fast, even on a daily basis, and we are already in the era of data deluge where we have much more data than what we can move, store, let alone analyze.Our contribution in this paper is the design and roadmap of a new paradigm in database systems, called NoDB, which do not require data loading while still maintaining the whole feature set of a modern database system. In particular, we show how to make raw data files a first-class citizen, fully integrated with the query engine. Through our design and lessons learned by implementing the NoDB philosophy over a modern DBMS, we discuss the fundamental limitations as well as the strong opportunities that such a research path brings. We identify performance bottlenecks specific for in situ processing, namely the repeated parsing and tokenizing overhead and the expensive data type conversion costs. To address these problems, we introduce an adaptive indexing mechanism that maintains positional information to provide efficient access to raw data files, together with a flexible caching structure.Our implementation over PostgreSQL, called PostgresRaw, is able to avoid the loading cost completely, while matching the query performance of plain PostgreSQL and even outperforming it in many cases. We conclude that NoDB systems are feasible to design and implement over modern database architectures, bringing an unprecedented positive effect in usability and performance.