Materialized View Selection Using Set Based Particle Swarm Optimization

A data warehouse is designed for answering analytical queries. These queries are usually long, complex and ad hoc in nature and their response time is high when processed against an exponentially growing data warehouse. Materialising views has been found to be an effective tool for reducing this response time. All views cannot be materialised on account of resource constraints. Further, optimal view selection is shown to be an NP-complete problem. This necessitates selection of an appropriate set of views, from amongst all possible views that reduce the query response time. Most of the view selection algorithms are greedy or evolutionary. In this paper, a differential evolution view selection algorithm (DEVSA) that selects the Top-K views from a multi-dimensional lattice is proposed. Further, it is shown that DEVSA when compared to the greedy, genetic and memetic-based view selection algorithms selects comparatively better quality views for higher dimensional datasets.

Data warehouse is designed for answering analytical queries. Data warehouse saves historical data. In the data warehouse, the response time to analytical queries is long. So reducing the response time is a critical problem. There are a lot of algorithms to solve the problem. Some of them, materialize frequent views. The previously posed queries have important information that will be used in the future. This paper proposes an algorithm for view materialization. The proposed algorithm finds proper views using previous queries and materializes them. The views are able to answer future queries. The view selection algorithm has four steps. At first, it clusters previous queries by SOM method. Then frequent queries are found by Apriori algorithm. In the third step the problem is converted to 0/1 knapsack equations and finally, optimal queries are joined to create only one view for each cluster. This paper improves the first and third step. This paper uses the SOM algorithm for clustering previous queries in the first step and it solves the 0/1 knapsack equations according to shuffled frog leaping algorithm in the third step. Experimental results show that it improves the previous view selection algorithms according to response time and storage space factor.

Set based particle swarm optimization for the feature selection problem

Materialized view selection under the maintenance time constraint

Data warehouse store historical records on which analysis queries are executed. Data warehouse are populated from the transaction database through a process called as Extract Transform and Load (ETL). Extract is to identify the transaction database records to be moved to the data warehouse. Transform is to make the records normalized to suit the data warehouse environment. Load is to actually store the records in the data warehouse. Today business wants to perform real time analysis of data. Availability of updated records in the data warehouse is necessary for real time analysis. This could be achieved through the process of near real time (ETL), i.e., to improve existing ETL process. Improvement to existing ETL could be achieved in many ways such as to increase the frequency of loads, to identify relevant changes that are required for analysis and move only those changes. Only through increasing the frequency of load would not be useful in many cases. Hence we should identify the changes and then move if they are useful for analysis making it smart ETL. In this paper we study three such techniques. First one is to create a replica of dimension table in data warehouse and move the changes to the replica this reduces the query response time. Second is to load the data in parallel so that loading time could be reduced. Third is to identify changes and trigger the loading process. We have used the first approach to create replica and loaded the data in parallel and observed that by loading in parallel not only does the loading time reduces but also the query response time. Then in the first approach is the query response time reduces to a certain limit methods suggest to move the data from replica dimensions to the original dimensions. Here we bring in right time trigger to move instead of just the query response time. We found that when we have a combined approach to query response time and right time trigger the number of moves are reduced.

A view selection algorithm takes as input a fact table and computes a set of views to store in order to speed up queries. The performance of view selection algorithm is usually measured by three criteria: (1) the amount of memory to store the selected views, (2) the query response time and (3) the time complexity of this algorithm. The two first measurements deal with the output of the algorithm. No existing solutions give good trade-off between amount of memory and queries cost with a small time complexity. We propose in this paper an algorithm guaranteeing a constant approximation factor of queries response time with respect to the optimal solution. Moreover, the time complexity for a D-dimensional fact table is O (D * 2D) corresponding to the fastest known algorithm. We provide an experimental comparison with two other well known algorithms showing that our approach also gives good performance in terms of memory.

Particle swarm optimization algorithms have been successfully applied to discrete/valued optimization problems. However, in many cases the algorithms have been tailored specifically for the problem at hand. This paper proposes a generic set-based particle swarm optimization algorithm for use in discrete-valued optimization problems that can be formulated as set-based problems. A detailed sensitivity analysis of the parameters of the algorithm is conducted. The performance of the proposed algorithm is then compared against three other discrete particle swarm optimization algorithms from literature using the multidimensional knapsack problem and is shown to statistically outperform the existing algorithms.

Feature selection carries significance in the outcome of any classification or regression task. Exercising evolutionary computation algorithms in feature selection has led to the construction of efficient discrete optimization algorithms. In this paper, a modified backtracking search algorithm is employed to perform wrapper-based feature selection, where two modifications of the standard backtracking search algorithm are adopted. The first one concentrates on utilizing a particle ranking operator regarding the current population. The second one focuses on removing the case of using a single particle on the mutation process. Then, the implementation of the above algorithm in feature selection is carried out in terms of two general frameworks, which originally were developed for the particle swarm optimization. The first framework is based on the binary and the second on the set-based particle swarm optimization. The experimental analysis shows that the above variants of the backtracking search algorithm perform equally well on the classification of several datasets.

The data warehouse is a repository of information collected from multiple possibly heterogeneous autonomous distributed databases. The information stored at the data warehouse is in form of views referred to as materialized views. The selection of the materialized views is one of the most important decisions in designing a data warehouse. Materialized views are stored in the data warehouse for the purpose of efficiently implementing on-line analytical processing queries. The first issue for the user to consider is query response time. So in this paper, we develop algorithms to select a set of views to materialize in data warehouse in order to minimize the total view maintenance cost under the constraint of a given query response time. We call it query_cost view_ selection problem. First, cost graph and cost model of query_cost view_ selection problem are presented. Second, the methods for selecting materialized views by using random algorithms are presented. The genetic algorithm is applied to the materialized views selection problem. But with the development of genetic process, the legal solution produced become more and more difficult, so a lot of solutions are eliminated and producing time of the solutions is lengthened in genetic algorithm. Therefore, improved algorithm has been presented in this paper, which is the combination of simulated annealing algorithm and genetic algorithm for the purpose of solving the query cost view selection problem. Finally, in order to test the function and efficiency of our algorithms experiment simulation is adopted. The experiments show that the given methods can provide near-optimal solutions in limited time and works better in practical cases. Randomized algorithms will become invaluable tools for data warehouse evolution.

Virtual network embedding (VNE) is an important problem in network virtualization for the flexible sharing of network resources. While most existing studies focus on centralized embedding for VNE, distributed embedding is considered more scalable and suitable for large-scale scenarios, but how virtual resources can be mapped to substrate resources effectively and efficiently remains a challenging issue. In this paper, we devise a distributed VNE system with historical archives (HAs) and metaheuristic approaches. First, we introduce metaheuristic approaches to each delegation of the distributed embedding system as the optimizer for VNE. Compared to the heuristic-based greedy algorithms used in existing distributed embedding approaches, which are prone to be trapped in local optima, metaheuristic approaches can provide better embedding performance for these distributed delegations. Second, an archive-based strategy is also introduced in the distributed embedding system to assist the metaheuristic algorithms. The archives are used to record the up-to-date information of frequently repeated tasks. By utilizing such archives as historical memory, metaheuristic algorithms can further improve embedding performance for frequently repeated tasks. Following this idea, we incorporate the set-based particle swarm optimization (PSO) as the optimizer and propose the distributed VNE system with HAs and set-based PSO (HA-VNE-PSO) system to solve the VNE problem in a distributed way. HA-VNE-PSO is empirically validated in scenarios of different scales. The experimental results verify that HA-VNE-PSO can scale well with respect to substrate networks, and the HA strategy is indeed effective in different scenarios.

This paper presents a comparison of two approaches for solving the vehicle routing problem with time windows (VRPTW). Scheduling of vehicles for pickup and delivery is a common problem in logistics and may be expressed as VRPTW, for which both exact and approximate techniques are available. It is therefore interesting to compare such techniques to evaluate their performance and Figure what is the best option based on the instance features and size. In this work, we compared Mixed Integer Linear Programming (MILP) with Set-Based Particle Swarm optimization (S-PSO). Both algorithms are tested on the full 56 instances of the Solomon dataset. The results show that the two algorithms perform similarly for lower number of customers while there are significant differences for the cases with higher number of customers. For higher number of customers MILP consistently performs as good as or better than S-PSO for the clustered data, both with short and long scheduling horizons, while the S-PSO outperforms MILP in most cases with random and mixed random clustered data with long scheduling horizons. Furthermore when the algorithms perform the same with regards to the main objective (number of vehicles), MILP generally achieves a better result in the second objective (distance traveled).

A Data warehouse is a structure that stores big amount of data. This data is exploited in the best possible ways in order to improve the efficiency of decision-making. The huge volume of data makes answering queries complex and time-consuming. Therefore, materialized views are used in order to reduce the query processing time. Since materializing all views is not possible, due to space and maintenance constraints, materialized view selection became one of the crucial decisions in designing a data warehouse for optimal efficiency. In this paper, we propose a discrete - quantum based - version of Differential Evolution DE algorithm named QDE algorithm to solve the materialized view selection (MVS) problem with space constraint. This algorithm is a merging of the original DE with Quantum Evolutionary (QEA) algorithm. The experimental results show the efficiency of the proposed algorithm compared to well known algorithms used to solve MVS problem with space constraint such as HRUA and GEA.

Materialized view selection is one of the crucial decisions in designing a data warehouse for optimal efficiency. Static selection of views may materialize certain views that are not beneficial as the data and usage trends change over time. On the contrary, dynamic selection of views works better only for queries demanding a high degree of aggregation. These facts point to the need for a technique that combines the improved response time of the static approach and the automated tuning capability of the dynamic approach. In this article, we propose a hybrid approach for the selection of materialized views. The idea is to partition the collection of all views into a static and dynamic set such that views selected for materialization from the static set are persistent over multiple query (and maintenance) windows, whereas views selected from the dynamic set can be queried and/or replaced on the fly. Highly aggregated views are selected on the fly based on the query access patterns of users, whereas the more detailed static set of views plays a significant role in the efficient maintenance of the dynamic set of views and in answering certain detailed view queries. We prove that our proposed strategy satisfies the monotonicity requirements, which is essential in order for the greedy heuristic to deliver competitive solutions. Experimental results show that our approach outperforms Dynamat, a well-known dynamic view management system that is known to outperform optimal static view selection.

Particle Swarm Optimization (PSO), classified as a swarm intelligence technique, mimics the well-informed swarming behavior of social species. A simple and effective searching strategy declares PSO as a potential member for solving various optimization problems. The present study embeds the concept of chaos at different stages of PSO, intending to enhance the convergence speed while trying to avoid stagnation and maintaining the solution quality. The proposed PSO variant is termed as “Totally disturbed PSO (TDPSO)”. The algorithm starts with a disturbed (chaotic) population, generated by considered chaotic system. Thereafter, when a certain number of iterations have elapsed and the searching process approaches equilibrium state, a relative velocity index is calculated for each particle to evaluate its present state and to decide whether or not the particle needs perturbation. The efficacy of proposed algorithm is tested against a set of benchmark problems and results are compared with existing Chaotic PSO and a standard PSO variant. Numerical results manifest that TDPSO works better over considered existing variants by effectively enhancing the searching capability and precision as well.

Particle swarm optimization (PSO) is a Swarm Intelligence technique used for optimization motivated by the social behavior of individuals in large groups in nature. The damped mass-spring analogy known as the PSO continuous model allowed us to derive a whole family of particle swarm optimizers with different properties with regard to their exploitation/exploration balance. Using the theory of stochastic differential and difference equations, we fully characterize the stability behavior of these algorithms. PSO and RR-PSO are the most performant algorithms of this family in terms of rate of convergence. Other family members have better exploration capabilities. The so called four point algorithms use more information of previous iterations to update the particles positions and trajectories and seem to be more exploratory than most of the 3 points versions. Finally, based on the done analysis, we can affirm that the PSO optimizers are not heuristic algorithms since there exist mathematical results that can be used to explain their consistency/convergence.