- Research Article
45
- 10.1016/s0169-023x(01)00007-6
Materialized view selection under the maintenance time constraint
- Apr 04, 2001
- Data & Knowledge Engineering
- Weifa Liang + 2 more +2
Materialized view selection under the maintenance time constraint
A Data warehouse is a structure that stores big amount of data. This data is exploited in the best possible ways in order to improve the efficiency of decision-making. The huge volume of data makes answering queries complex and time-consuming. Therefore, materialized views are used in order to reduce the query processing time. Since materializing all views is not possible, due to space and maintenance constraints, materialized view selection became one of the crucial decisions in designing a data warehouse for optimal efficiency. In this paper, we propose a discrete - quantum based - version of Differential Evolution DE algorithm named QDE algorithm to solve the materialized view selection (MVS) problem with space constraint. This algorithm is a merging of the original DE with Quantum Evolutionary (QEA) algorithm. The experimental results show the efficiency of the proposed algorithm compared to well known algorithms used to solve MVS problem with space constraint such as HRUA and GEA.
Materialized view selection under the maintenance time constraint
Materialized view selection under the maintenance time constraint
Selecting materialized views using random algorithm
The data warehouse is a repository of information collected from multiple possibly heterogeneous autonomous distributed databases. The information stored at the data warehouse is in form of views referred to as materialized views. The selection of the materialized views is one of the most important decisions in designing a data warehouse. Materialized views are stored in the data warehouse for the purpose of efficiently implementing on-line analytical processing queries. The first issue for the user to consider is query response time. So in this paper, we develop algorithms to select a set of views to materialize in data warehouse in order to minimize the total view maintenance cost under the constraint of a given query response time. We call it query_cost view_ selection problem. First, cost graph and cost model of query_cost view_ selection problem are presented. Second, the methods for selecting materialized views by using random algorithms are presented. The genetic algorithm is applied to the materialized views selection problem. But with the development of genetic process, the legal solution produced become more and more difficult, so a lot of solutions are eliminated and producing time of the solutions is lengthened in genetic algorithm. Therefore, improved algorithm has been presented in this paper, which is the combination of simulated annealing algorithm and genetic algorithm for the purpose of solving the query cost view selection problem. Finally, in order to test the function and efficiency of our algorithms experiment simulation is adopted. The experiments show that the given methods can provide near-optimal solutions in limited time and works better in practical cases. Randomized algorithms will become invaluable tools for data warehouse evolution.
Read moreMaterialised view selection using differential evolution
A data warehouse is designed for answering analytical queries. These queries are usually long, complex and ad hoc in nature and their response time is high when processed against an exponentially growing data warehouse. Materialising views has been found to be an effective tool for reducing this response time. All views cannot be materialised on account of resource constraints. Further, optimal view selection is shown to be an NP-complete problem. This necessitates selection of an appropriate set of views, from amongst all possible views that reduce the query response time. Most of the view selection algorithms are greedy or evolutionary. In this paper, a differential evolution view selection algorithm (DEVSA) that selects the Top-K views from a multi-dimensional lattice is proposed. Further, it is shown that DEVSA when compared to the greedy, genetic and memetic-based view selection algorithms selects comparatively better quality views for higher dimensional datasets.
Read moreMaterialized View Selection Using Set Based Particle Swarm Optimization
A data warehouse is a central repository of historical data designed primarily to support analytical processing. These analytical queries are exploratory, long and complex in nature. Further, the rapid and continuous growth in the size of data warehouse increases the response times of such queries. Query response times need to be reduced in order to speedup decision making. This problem, being an NP-Complete problem, can be appropriately dealt with by using swarm intelligence techniques. One such technique, i.e. the set-based particle swarm optimization (SPSO), has been proposed to address this problem. Accordingly, a SPSO based view selection algorithm (SPSOVSA), which selects the Top-K views from a multidimensional lattice, is proposed. Experimental based comparison of SPSOVSA with the most fundamental view selection algorithm shows that SPSOVSA is able to select comparatively better quality Top-K views for materialization. The materialization of these selected views would improve the performance of analytical queries and lead to efficient decision making.
Read moreMultiobjective genetic algorithms for materialized view selection in OLAP data warehouses
On-Line Analytical Processing (OLAP) tools are frequently used in business, science and health to extract useful knowledgefrom massive databases. An important and hard optimization problem in OLAP data warehouses is the view selection problem, consisting of selecting a set of aggregate views of the data for speeding up future query processing. A common variant of the view selection problem addressed in the literature minimizes the sum of maintenance cost and query time on the view set. Converting what is inherently an optimization problem with multiple conflicting objectives into one with a single objective ignores the need and value of a variety of solutions offering various levels of trade-off between the objectives. We apply two non-elitist multiobjective evolutionary algorithms (MOEAs) to view selection under a size constraint. Our emphasis is to determine the suitability of the combination of MOEAs with constraint handling to the view selection problem, compared to a widely used greedy algorithm. We observe that the evolutionary process mimics that of the greedy in terms of the convergence process in the population. The MOEAs are competitive with the greedy on a variety of problem instances, often finding solutions dominating it in a reasonable amount of time.
Read moreMaterialized view selection using evolutionary algorithm for speeding up big data query processing
For speeding up query processing on Big Data, frequent sub-queries or views may be materialized such that the query processing cost is minimized with optimum cost of maintaining the materialized views and/or queries. Materializing frequent sub-queries and views means that resultant data set of the views reside in the memory of one or more nodes in the cluster, so that it reduces the MapReduce cost, submission and scheduling cost of Distributed File System jobs for query processing. We have defined materialized views as resultant data of frequent sub-queries and aggregation functions of a set of Big Data warehousing queries that are saved for enhancing query performance. The problem is defined as a multi-objective optimization problem for minimizing the total query processing MapReduce cost, MapReduce cost for maintaining the materialized views and the number of views selected for materializing with maximized total size of the views selected. We applied Differential Evolution algorithm and NSGA-II to study their performances for developing a recommendation system for selecting views for materializing in Big Data warehousing.
Read moreTemplate-Based Bitmap View Selection for Optimizing Queries Over Tree Data
Developing and exploiting flexible techniques for optimizing the evaluation of queries over loosely structured data (e.g. tree or graph databases) is of crucial importance for modern database applications. In this context, we consider a new type of views which can be materialized as compressed bitmaps over tree data. We introduce the concept of view structural template to define classes of views. We then define and address a novel view selection problem (called view class selection (VCS) problem) where the goal is to select classes of bitmap views in order to optimize the overall evaluation cost of all tree pattern queries (TPQs) that can be issued against a database while satisfying a space constraint and ensuring that all the TPQs can be answered using exclusively the materialized views. We show that the VCS problem is NP-hard and we design two heuristic greedy algorithms which iteratively generate new batches of candidate view classes and make them available for selection. Each algorithm uses a different view class expansion technique to enable the systematic generation of candidate view classes from classes with smaller templates. We run extensive experiments to evaluate both the effectiveness of the algorithms and their efficiency on real, benchmark and synthetic datasets. Our algorithms are able to suggest high quality selections of view classes in a reasonable amount of time.
Read moreAQEA-QAS: An Adaptive Quantum Evolutionary Algorithm for Quantum Architecture Search.
Quantum neural networks (QNNs) represent an emerging technology that uses a quantum computer for neural network computations. The QNNs have demonstrated potential advantages over classical neural networks in certain tasks. As a core component of a QNN, the parameterized quantum circuit (PQC) plays a crucial role in determining the QNN's overall performance. However, quantum circuit architectures designed manually based on experience or using specific hardware structures can suffer from inefficiency due to the introduction of redundant quantum gates, which amplifies the impact of noise on system performance. Recent studies have suggested that the advantages of quantum evolutionary algorithms (QEAs) in terms of precision and convergence speed can provide an effective solution to quantum circuit architecture-related problems. Currently, most QEAs adopt a fixed rotation mode in the evolution process, and a lack of an adaptive updating mode can cause the QEAs to fall into a local optimum and make it difficult for them to converge. To address these problems, this study proposes an adaptive quantum evolution algorithm (AQEA). First, an adaptive mechanism is introduced to the evolution process, and the strategy of combining two dynamic rotation angles is adopted. Second, to prevent the fluctuations of the population's offspring, the elite retention of the parents is used to ensure the inheritance of good genes. Finally, when the population falls into a local optimum, a quantum catastrophe mechanism is employed to break the current population state. The experimental results show that compared with the QNN structure based on manual design and QEA search, the proposed AQEA can reduce the number of network parameters by up to 20% and increase the accuracy by 7.21%. Moreover, in noisy environments, the AQEA-optimized circuit outperforms traditional circuits in maintaining high fidelity, and its excellent noise resistance provides strong support for the reliability of quantum computing.
Read moreSelecting Materialized Views based on Genetic Algorithm
As the data sets increase in data warehouse day by day, the cost used for OLAP operations becomes extremely high. One of the most effective ways to solve this problem is to build appropriate materialized views in the data warehouse. In order to select the appropriate views to be materialized with the possible minimized cost, we propose a novel approach to the materialized view selection problem based on a genetic algorithm. Experimental results show that the optimal solution for selecting materialized views can be obtained effectively using the approach presented.
Read moreMaterialized view selection and maintenance using multi-query optimization
Materialized views have been found to be very effective at speeding up queries, and are increasingly being supported by commercial databases and data warehouse systems. However, whereas the amount of data entering a warehouse and the number of materialized views are rapidly increasing, the time window available for maintaining materialized views is shrinking. These trends necessitate efficient techniques for the maintenance of materialized views.In this paper, we show how to find an efficient plan for the maintenance of a set of materialized views, by exploiting common subexpressions between different view maintenance expressions. In particular, we show how to efficiently select (a) expressions and indices that can be effectively shared, by transient materialization; (b) additional expressions and indices for permanent materialization; and (c) the best maintenance plan — incremental or recomputation — for each view. These three decisions are highly interdependent, and the choice of one affects the choice of the others. We develop a framework that cleanly integrates the various choices in a systematic and efficient manner. Our evaluations show that many-fold improvement in view maintenance time can be achieved using our techniques. Our algorithms can also be used to efficiently select materialized views to speed up workloads containing queries and updates.
Read moreDynaMat
Pre-computation and materialization of views with aggregate functions is a common technique in Data Warehouses. Due to the complex structure of the warehouse and the different profiles of the users who submit queries, there is need for tools that will automate the selection and management of the materialized data. In this paper we present DynaMat, a system that dynamically materializes information at multiple levels of granularity in order to match the demand (workload) but also takes into account the maintenance restrictions for the warehouse, such as down time to update the views and space availability. DynaMat unifies the view selection and the view maintenance problems under a single framework using a novel “goodness” measure for the materialized views. DynaMat constantly monitors incoming queries and materializes the best set of views subject to the space constraints. During updates, DynaMat reconciles the current materialized view selection and refreshes the most beneficial subset of it within a given maintenance window. We compare DynaMat against a system that is given all queries in advance and the pre-computed optimal static view selection. The comparison is made based on a new metric, the Detailed Cost Savings Ratio introduced for quantifying the benefits of view materialization against incoming queries. These experiments show that DynaMat's dynamic view selection outperforms the optimal static view selection and thus, any sub-optimal static algorithm that has appeared in the literature.
Read moreBlended Rank Evolutionary Algorithm for the Constrained Multiobjective Crop Rotation Problem
In a constrained multiobjective problem, solutions can be mapped onto three spaces: decision variable space, objective space, and constraint space. Blended Rank Evolutionary Algorithm uses measures from all three spaces and dynamically blends them together into a final fitness score for use in an evolutionary algorithm. Results on the highly constrained, multiobjective "nonlinear crop rotation" problem show that BREA reliably finds better quality non-dominated fronts than the popular algorithm NSGA-II. The difficulty of the nonlinear crop rotation problem leaves room for improvement in both algorithms.
Read moreFuzzy quantum computation based thermal unit commitment strategy with solar-battery system injection
This article presents a strategy to solve thermal unit commitment (UC) integrated with an equivalent solar battery system using Fuzzy based Quantum inspired Evolutionary Algorithm (FQEA). As a renewable power source, solar power is injected stochastically with the model. To handle the uncertainty and intermittency involved while integrating solar power and load forecasting, the trivial crisp problem formulations are modified by fuzzification. An evolutionary algorithm based on the concept and principle of quantum computation is applied to solve the UC problem. The conventional Quantum Evolutionary Algorithm (QEA) is advanced by using several operators such as binary differential operator, mutation and crossover along with trivial rotation operator with a re-defined rotational angle look-up table. The QEA is further modified by introducing multi-population based scheme. The fitness function is formulated by combining the objective function, penalty function and the aggregated fuzzy membership function. The proposed FQEA is applied to UC problem in different scaled power systems up to 100 units. Provided simulation results will show the effectiveness of FQEA.
Read moreMaterialized View Selection in Data Warehousing: A Survey
Materialized View Selection in Data Warehousing: A Survey
Materialized View Selection Using Vector Evaluated Genetic Algorithm
Data Warehousing is an approach in which data from multiple heterogeneous and distributed operational systems (OLTP) are extracted, transformed and loaded into a central repository for the purpose of decision making. Since such database stores huge amounts of historical data, it is necessary to devise methods by which complex OLAP queries can be answered as fast as possible. OLAP is an approach which facilitates analytical queries accessing multidimensional databases. Using materialized views as pre-computed results for time-consuming queries is a common method for speeding up analytical queries. However, some constraints do not allow the systems to save all possible views. Therefore, one of the crucial decisions that data warehouse designers need to make is in the selection of the right set of views to be materialized. This paper focuses on solving the multi-objective view selection problem using a Vector Evaluated Genetic Algorithm (VEGA) approach subject to disk space constraint.
Read more