Building large ROLAP data cubes in parallel

The pre-computation of data cubes is critical to improving the response time of on-line analytical processing (OLAP) systems and can be instrumental in accelerating data mining tasks in large data warehouses. In order to meet the need for improved performance created by growing data sizes, parallel solutions for generating the data cube are becoming increasingly important. The paper presents a parallel method for generating data cubes on a shared-nothing multiprocessor. Since no (expensive) shared disk is required, our method can be used on low cost Beowulf style clusters consisting of standard PCs with local disks connected via a data switch. Our approach uses a ROLAP representation of the data cube where views are stored as relational tables. This allows for tight integration with current relational database technology. We have implemented our parallel shared-nothing data cube generation method and evaluated it on a PC cluster, exploring relative speedup, local vs. global schedule trees, data skew, cardinality of dimensions, data dimensionality, and balance tradeoffs. For an input data set of 2000000 rows (72 Megabytes), our parallel data cube generation method achieves close to optimal speedup; generating a full data cube of /spl ap/227 million rows (5.6 Gigabytes) on a 16 processors cluster in under 6 minutes. For an input data set of 10,000,000 rows (360 Megabytes), our parallel method, running on a 16 processor PC cluster, created a data cube consisting of /spl ap/846 million rows (21.7 Gigabytes) in under 47 minutes.

Data cube has been playing an essential role in fast OLAP (online analytical processing) in many data warehouses. The pre-computation of data cubes is critical for improving the OLAP response time of in large high-dimensional data warehouses. However, as the sizes of data warehouses grow, the time it takes to perform this pre-computation becomes a significant performance bottleneck. In a high dimensional data warehouse, it might not be practical to build all these cuboids and their indices. In this paper, we propose a hierarchical cubing algorithm to partition the high dimensional data cube into low dimensional cube segments. It permits a significant reduction of CPU and I/O overhead for many queries by restricting the number of cube segments to be processed for both the fact table and bitmap indices. Experimental results show that the proposed method is significantly more efficient than other existing cubing methods.

University of Edinburgh, School of Informatics, EH8 9AB, Edinburgh, Scotland, United Kingdom ; cristinutza0107@yahoo.com Abstract. This paper discusses the problem of multidimensional databases, with the purpose of demonstrating the efficient support offered by them for the management of OLAP processing and motivation of decisions, especially the ones related to environmental risks. The OLAP (On-Line Analytical Processing) systems that implement or use multidimensional databases are oriented towards decision makers and allow them to an easily process, sectioning and grouping of data, as well as the visualization of data from different perspectives, for the discovery of the factors that would influence a certain behavior. OLAP systems offer advanced statistical analysis facilities, such as the one based on time series or forecasts, which are based on predefined models. The calculation of the environmental risk offers an image of the degree of air pollution due to specific polluting factors, obtained especially as result of the industrial processes that have taken place in the Targoviste, in Dambovita area. An OLAP system, consisting of decision support components, was implemented as part of the PN2 three years research project: “Sustainable Management System of Resources Used for Monitoring and Evaluating the Environmental Risks in Order to Prevent the Negative Effects and to Manage Crises Situations - MEMDUR”, code D11-037/18.09.2007. Keywords: environmental risks, multidimensional databases, OLAP systems, SOLAP architecture, MEMDUR project.

Chapter 64 - Exploiting Versions for On-line Data Warehouse Maintenance in MOLAP Servers

On-line Analytical Processing (OLAP) has become one of the most powerful and prominent technologies for knowledge discovery in VLDB (Very Large Database) environments. Central to the OLAP paradigm is the data cube, a multi-dimensional hierarchy of aggregate values that provides a rich analytical model for decision support. Various sequential algorithms for the efficient generation of the data cube have appeared in the literature. However, given the size of contemporary data warehousing repositories, multi-processor solutions are crucial for the massive computational demands of current and future OLAP systems. In this paper we discuss the cgmCUBE Project, a multi-year effort to design and implement a multi-processor platform for data cube generation that targets the relational database model (ROLAP). More specifically, we discuss new algorithmic and system optimizations relating to (1) a thorough optimization of the underlying sequential cube construction method and (2) a detailed and carefully engineered cost model for improved parallel load balancing and faster sequential cube construction. These optimizations were key in allowing us to build a prototype that is able to produce data cube output at a rate of over one TeraByte per hour.

Data warehouses provide data for on-line analytical processing (OLAP) systems, which deal with analytical tasks in businesses. As these tasks do not depend on the latest updates by transactions, the input from operational databases is separated from the outputs to dialogue interfaces for OLAP. In this paper a layered formal specification for data warehouses and OLAP systems using Abstract State Machines (ASMs) is presented. The approach explicitly exploits the fundamental idea of separating input from operational databases and output to OLAP systems. Then it will be shown how this specification can be extended to distributed data warehouses.KeywordsData WarehouseOperational DatabaseAbstract State MachineStar SchemaTional DatabaseThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Business data collection is growing exponentially in recent years. A variety of industries and businesses have adopted new technologies of data storages such as data warehouses. On Line Analytical Processing (OLAP) has become an important tool for executives, managers, and analysts to explore, analyze, and extract interesting patterns from enormous amount of data stored in data warehouses and multidimensional databases. However, it is difficult for human analysts to interpret and extract meaningful information from large amount of data if the data is presented in textual form as relational tables. Visualization and interactive tools employ graphical display formats that help analysts to understand and extract useful information fast from huge data sets. This paper presents a new visual interactive exploration technique for an analysis of multidimensional databases. Users can gain both overviews and refine views on any particular region of interest of data cubes through the combination of interactive tools and navigational functions such as drilling down, rolling up, and slicing. Our technique allows users who are not experts in OLAP technology to explore and analyze OLAP data cubes and data warehouses without generating sophisticated queries. Furthermore, the visualization in our technique displays the exploration path enhancing the user's understanding of the exploration.

Currently, data classification is either performed on data stored in relational databases or performed on data stored in flat files. The problem with these approaches is that for large data sets, they often need multiple scans of the original data and thus are often infeasible in many applications. In this chapter we propose to deploy classification on top of OLAP (online analytical processing) and data cube systems. First, we compute the statistics in various combinations of the attributes known as data cubes. The statistics are then used to derive classification models. In this way, we only scan the original data once, which improves the performance of classification significantly. Furthermore, our new classifier will provide “free” classification by eliminating the dominating I/O overhead of scanning the massive original data. An architecture that integrates database, data cube, and data mining is given and three new cube-based classifiers are presented and evaluated.

In this paper we propose the use of the compact data structure k2-treap to process data cubes of Data Warehouses (DWs) into main memory. Compact data structures are data structures that allow compacting the data without losing the capacity of querying them in their compact form. A DW is a data repository to store historical data for decision support, and consists of dimensions and facts. The former are an abstract concept that groups data with a similar meaning, they are modelled as hierarchies of levels, which contain elements. The latter are quantitative data associated to dimensions. A data cube is a typical way to retrieve facts at different levels of granularity (through navigation on dimensions hierarchies). A DW can store terabytes of data, thus the efficient processing of data cubes is key in OLAP (On-line Analytical Processing). We show that by using a compact representation of data cubes and bitmaps to represent dimensions we are able to improve the use of space in main memory, and achieve better performance for query processing.

OnLine Analytical Processing (OLAP) research issues (Gray, Chaudhuri, Bosworth, Layman, Reichart & Venkatrao, 1997) such as data cube modeling, representation, indexing and management have traditionally attracted a lot of attention from the Data Warehousing research community. In this respect, as a fundamental issue, the problem of efficiently compressing the data cube plays a leading role, and it has involved a vibrant research activity in the academic as well as industrial world during the last fifteen years. Basically, this problem consists in dealing with massive-in-size data cubes that make access and query evaluation costs prohibitive. The widely accepted solution consists in generating a compressed representation of the target data cube, with the goal of reducing its size (given an input space bound) while admitting loss of information and approximation that are considered irrelevant for OLAP analysis goals (e.g., see (Cuzzocrea, 2005a)). Compressed representations are also referred-in-literature under the term "synopsis data structures", i.e. succinct representations of original data cubes introducing a limited loss of information. Benefits deriving from the data cube compression approach can be summarized in a relevant reduction of computational overheads needed to both represent the data cube and evaluate resource-intensive OLAP queries, which constitute a wide query class allowing us to extract useful knowledge from huge amounts of multidimensional data repositories in the vest of summarized information (e.g., aggregate information based on popular SQL aggregate operators such as SUM, COUNT, AVG etc), otherwise infeasible by means of traditional OLTP approaches. Among such queries, we recall: (i) range-queries, which extract a sub-cube bounded by a given range; (ii) top-k queries, which extract the k data cells (such that k is an input parameter) having the highest aggregate values; (iii) iceberg queries, which extract the data cells having aggregate values above a given threshold. This evidence has given raise to the proliferation of a number of Approximate Query Answering (AQA) techniques, which, based on data cube compression, aim at providing approximate answers to resource-intensive OLAP queries instead of computing exact answers, as decimal precision is usually negligible in OLAP query and report activities (e.g., see (Cuzzocrea, 2005a)). Starting from these considerations, in this article we first provide a comprehensive, rigorous survey of main data cube compression techniques, i.e. histograms, wavelets, and sampling. Then, we complete our analytical contribution with a detailed theoretical review focused on spatio-temporal complexities of these techniques. Request access from your librarian to read this chapter's full text.

This paper presents a general methodology for the efficient parallelization of existing data cube construction algorithms. We describe two different partitioning strategies, one for top-down and one for bottom-up cube algorithms. Both partitioning strategies assign subcubes to individual processors in such a way that the loads assigned to the processors are balanced. Our methods reduce inter processor communication overhead by partitioning the load in advance instead of computing each individual group-by in parallel. Our partitioning strategies create a small number of coarse tasks. This allows for sharing of prefixes and sort orders between different group-by computations. Our methods enable code reuse by permitting the use of existing sequential (external memory) data cube algorithms for the subcube computations on each processor. This supports the transfer of optimized sequential data cube code to a parallel setting. The bottom-up partitioning strategy balances the number of single attribute external memory sorts made by each processor. The top-down strategy partitions a weighted tree in which weights reflect algorithm specific cost measures like estimated group-by sizes. Both partitioning approaches can be implemented on any shared disk type parallel machine composed of p processors connected via an interconnection fabric and with access to a shared parallel disk array. We have implemented our parallel top-down data cube construction method in C++ with the MPI message passing library for communication and the LEDA library for the required graph algorithms. We tested our code on an eight processor cluster, using a variety of different data sets with a range of sizes, dimensions, density, and skew. Comparison tests were performed on a SunFire 6800. The tests show that our partitioning strategies generate a close to optimal load balance between processors. The actual run times observed show an optimal speedup of p.

The rapidly changing data requirements of today's dynamic business environments are not handled well by current online analytical processing (OLAP) systems. Physically integrating data from new sources into OLAP systems is a long and time-consuming process, making logical integration the better choice in many situations. The increasing use of Extended Markup Language (XML) suggests that the required external data will most often be available in XML format. We present a theoretically well-founded approach to the logical federation of OLAP and XML data sources. The approach allows external XML data to be presented along with dimensional data in OLAP query results and enables the use of external XML data for selection and grouping. Special care is taken to ensure that semantic problems do not occur in the integration process. This opens up many new application areas for OLAP. A number of effective optimization techniques for OLAP-XML federations are presented. Performance results from the prototype implementation show that the approach is an attractive alternative to physical integration. The approach is exemplified using a real-world case study from the B2B domain.

Data cube construction is a commonly used operation in data warehouses. Because of the volume of data that is stored and analyzed in a data warehouse and the amount of computation involved in data cube construction, it is natural to consider parallel machines for this operation. We have developed a set of parallel algorithms for data cube construction using a new data structure called aggregation tree. Our experience has shown that a number of performance trade-offs arise in developing a parallel data cube implementation. We focus on three important issues, which are: (1) data distribution, i.e., how the original array is distributed among the processors; (2) level of parallelism, i.e., what parts of the computation are parallelized and sequentialized; and (3) frequency of communication, i.e., does the implementation require frequent interprocessor communication (and less memory) or less frequent communication (and more memory). We present a detailed experimental study evaluating the above trade-offs. We consider parallel data cube construction with different cube sizes and sparsity levels. Our experimental results show the following: (1) In all cases, reducing the frequency of communication and using higher memory gave better performance, though the difference was relatively small. (2) Choosing data distribution to minimize communication volume made a substantial difference in the performance in most of the cases. (3) Finally, using parallelism at all levels gave better performance, even though it increases the total communication volume.

Defining and structuring management views is a fundamental task of the specification of data marts and reporting systems. Today such systems are based on data warehouses. A data warehouse stores materialized views on business processes to support the management’s information requirements (cf. Inmon 1996; Inmon, Hackathorn 1994; Inmon et al. 1997). It is located on a central layer of an idealized layer oriented architecture connecting online transaction processing (OLTP) systems and components enabling online analytical processing (OLAP) (cf. Becker, Holten 1998; Chaudhuri, Dayal 1997). The latter components are intended to support navigations adequate for management users through so called multi dimensional information spaces. OLTP systems directly support the business processes and are the sources of data used by OLAP systems. Typically, the integration of OLTP systems and the data warehouse is based on tools performing so called extraction, transformation and loading tasks (ETL) (cf. Inmon 1996; Widom 1995)

Logistic regression is an important technique for analyzing and predicting data with categorical attributes. In this paper, We consider supporting online analytical processing (OLAP) of logistic regression analysis for multi-dimensional data in a data cube where it is expensive in time and space to build logistic regression models for each cell from the raw data. We propose a novel scheme to compress the data in such a way that we can reconstruct logistic regression models to answer any OLAP query without accessing the raw data. Based on a first-order approximation to the maximum likelihood estimating equations, we develop a compression scheme that compresses each base cell into a small compressed data block with essential information to support the aggregation of logistic regression models. Aggregation formulae for deriving high-level logistic regression models from lower level component cells are given. We prove that the compression is nearly lossless in the sense that the aggregated estimator deviates from the true model by an error that is bounded and approaches to zero when the data size increases. The results show that the proposed compression and aggregation scheme can make feasible OLAP of logistic regression in a data cube. Further, it supports real-time logistic regression analysis of stream data, which can only be scanned once and cannot be permanently retained. Experimental results validate our theoretical analysis and demonstrate that our method can dramatically save time and space costs with almost no degradation of the modeling accuracy.