Maintaining Materialized Views for Data Warehouses with Multiple Remote Sources

Big data represents a new technology for managing data with high velocity, volume, variety and contributes to creating value for companies. As quoted in1, capturing all queries made on the company website or from customer support calls, emails or chat lines, regardless of their outcome, may have significant value in identifying emerging trends. The Big Data Era has largely contributed in accelerating the development strategic plans issued from governments and research organisms, coving the management, exploitation and analysis of these data by taking into account the different V's of Big Data. Among these plans, we can cite for instance the development of: (a) large-scale platforms (ex. data-clusters, distributed data clusters), (b) Software Defined Environments (SDE) (ex. IBM SDE), (c) advanced programming paradigms (ex. map-reduce, Spark, etc.), Data Analytics Tools (Rapid Miner, Google Fusion Tables, Solver), (d) Visualization tools (Google Chart, Tableau, Oracle Visual Analyzer), and (h) high quality and valuable Knowledge Bases (KB), constructed either by academicians (e.g., Cvc, DBpedia, Freebase, and YAGO) and industrials (e.g., Google Knowledge Graph, Facebook Knowledge Graph, Amazon Knowledge Graph, Credit Rating Agencies, Enterprise Knowledge Base, etc.). In this talk, we would like to foster the creation of a think tank dedicated to getting the best from Big Data V's and the efforts related to it to revisit our research activities without compromising them. In this talk, we would like to share the experience conducted with our Model and Data Engineering Team of the LIAS Laboratory at ISAE-ENSMA, which aims at the design of data warehousing applications. Based on the literature, this design is based on two main approaches: (i) a supply-driven approach (also called data-driven) that starts with an analysis of operational data sources in order to identify all the available data and (ii) a user-driven approach (also known as requirement-driven or goal-orientated) which stems from the determination of the information requirements of different business users. Several studies and experiments show that resorting to these two approaches entails a high risk for companies, since some functional requirements cannot be satisfied. This is due to the lack of relevant data in sources. In parallel, reference studies have identified the crucial role of knowledge bases (KB) for analytical tasks, by offering analysts more entities (people, places, products, etc.). The availability of a huge, high quality valuable KB is an asset for data warehousing designers and decision-makers to construct/exploit a valuable data warehouse. So, faced with this situation, we here present a value-driven approach that revisits the traditional life cycle of the design of data warehouses, by considering KB as an external resource. These different phases are illustrated via the YAGO KB.

In data warehouse (DW) development, a series of mappings must be specified between user concepts and data source elements, in order to identify which sources must undergo an integration process. Until now, these mappings are either assumed to be implied by name matching or identified according to the designer's experience. Then, the result is implemented as Extraction/Transformation/Loading (ETL) processes. Since ETL processes relate elements at the logical level, designers cannot adequately analyze how a change in requirements or in the data sources affects the analysis capabilities. Furthermore, this approach makes it difficult to perform incremental changes in DW design, requiring in some cases to perform the whole analysis again. In this paper we present a set of semantic mappings that relate user concepts specified by requirements to those obtained from data sources. In turn, this allows us to accurately identify how any potential change affects the different structures and ETL processes. As a DW evolves over time, our approach easily allows us to incorporate new concepts, as well as any change introduced at requirements or data sources into the DW repository with no need to redesign the whole DW. In order to show the application of our proposal, we show a real case study focusing on the Digital library of the University of Alicante.

A company, majorly company that active in commercial (profit orientation) need to analyze their sales performance. By analyzing sales performance, company can increase their sales performance. One of method to analyze sales performance is by collecting historical data that relates to sales and then process that data so that produce information that show company sales performance. A data warehouse is a set of data that has characteristic subject oriented, time variant, integrated, and nonvolatile that help company management in processing of decision making. Design of data warehouse is started from collecting data that relate to sales such as product, customer, sales area, sales transaction, etc. After collecting the data, next is data extraction and transformation. Data extraction is a process f or selecting data that will be loaded into data warehouse. Data transformation is making some change to the data afte r extracted to be more consistent. After transformation processing, data are loaded into data warehouse. Data in data warehouse is processed by OLAP (On Line Analytical Processing) to produce information. Information that are produced from data processing by OLAP are chart and query reporting. Chart reporting are sales chart based on cement type, sales chart based on sales area, sales chart based on plant, monthly and year ly sales chart, and chart based on customer feedback. Query reporting are sales based on cement type, sales area, plant and customer. Keywords: Data warehouse; OLAP; Sales performance analysis; Ready mix market

The data warehouse is based on the development of relational databases, parallel processing and distributed processing technologies, as well as online analytical processing technologies. SAP BW is widely used in many enterprise data warehouses. It can issue reports of enterprises at the group level, manage the group's business operations, help find the laws and trends behind the performance of corporations and ultimately provide decision-making support for decision makers of enterprises. Therefore, how to use the SAP BW model architecture to better realize enterprise data warehouses is one of the hot topics in the data warehouse industry today. For a large corporation, its branches and subsidiaries have deployed multiple sets of enterprise data warehouses. At the same time, because there are multiple sets of data warehouse systems, when the data model needs to be modified at the group level, the data model of the branch also needs to be modified accordingly, and the flexibility of maintenance is poor. Finally, every time the corporation needs to upgrade the version of the data warehouse, it needs to maintain and upgrade the version of the data warehouse of each branch and the maintenance cost is high. Therefore, how to integrate multiple sets of data warehouses distributed in various subsidiaries into a corporation enterprise data warehouse by designing the model hierarchy, so as to ensure the reliability of data, enhance the flexibility of model modification and reduce the cost of data warehouse. This paper completes the design of an enterprise data warehouse of a large corporation, which can be used for the summary and collation of data from multiple branches such as finance and sales of a corporation, effectively improving the efficiency of data utilization and ensuring the authenticity of data. At the same time, through hierarchical design, the problems of poor flexibility in model modification and high system maintenance costs are solved.

Grants is a program launched by the government as a means to develop the potential of the region or contribute to society in the form of money or objects In submitting as Grant there are many sources of data stored in the system e -Grant, The amount of the proposal Grant is very much in each period so as to be reorganized, One solution for the storage of e-submission of the Grant is to build a data warehouse, Due to the technological development of data warehouse will be able to support the creation of a single access door managerial parties in obtaining information in the data warehouse, The design of the data warehouse were calculated using a four step Kimball, Methods Kimball 4 step has the advantages of a more compact, dynamic, and supports the creation of a data warehouse that is structured output of the data warehouse to the e-Grant in the form of Reports Grant and a recapitulation of Grant where reports Grant is being used as an attachment to a defense in response to the report Expenditure Provincial Bali whereas Summary of Grants used for List SKPD to evaluate the Grant recipient.

Data analysis has become a vital requirement in various fields. The results of data analysis can be used as a data warehouse design that is expected to help in solving problems, evaluating learning outcomes and supporting decision making. In the field of education, student achievement is a reference for achieving quality learning. This success can be seen from the achievement of student learning completeness that is the basis for evaluating and making decisions. The data warehouse design can be used as a basis in knowing student learning progress that can be seen from the value obtained. The diversity of data makes managing values difficult and delayed, data warehouse design using Pentaho can help and simplify data integration. The results of the analysis and design of the data warehouse will be presented in a multidimensional form that can be seen through dimension tables and fact tables.

In the past few years, several conceptual approaches have been proposed for the specification of the main multidimensional (MD) properties of the data warehouse (DW) repository. However, most of them deal with isolated aspects of the DW and do not provide designers with an integrated and standard method for designing the whole DW life cycle (ETL processes, data sources, DW repository and so on).Some approaches are depending on specific platform or neglecting important issues in DW design life cycle. Extraction-transformation-loading (ETL) processes play an important role in data warehouse architecture because they are responsible of integrating data from heterogeneous data sources into the DW repository. This paper proposes a conceptual model to refresh data warehouse by (insert, update, delete) data using ETL processes and considering DW security requirements. Firstly, the proposed ETL model is based on Unified Modeling Language (UML), which allows us to accomplish the conceptual modeling of ETL processes .secondly; this part focuses on how to integrate the proposed ETL model with the DW model. The proposed DW model depends on the Model Driven Architecture (MDA). MDA is a standard framework for software development that addresses the complete life cycle of designing, deploying, integrating, and managing applications by using models in software development. This paper proposes an integrated conceptual model for addressing temporal data warehouse security requirements (CMTDWS).

A novel three-level architecture for large data warehouses

This study is objected to analyze the current system to find the problems being faced by ABC Hospital and propose alternative solutions. Additionally, it aims to generate information needed by the hospital management to design a data warehouse according to the information needs. The research method used is the analysis and design of data warehouse by Kimball and Ross cited by Connolly and Begg, known as Nine-Step Methodology. The result obtained is a data warehouse application with a multidimensional historical data that can assist the hospital management in decision making. Designing data warehouse at ABC Hospital makes enterprise data into concise and can be viewed from several dimensions that help users analyze data for strategic decision quickly and accurately.

Data warehousing has recently emerged as a possible solution to enable end user access to the increasing volumes of business oriented data. The foundation of any modern decision support architecture is the data warehouse. In this paper we will give a framework for the use of data warehouses in the tourism domain. The framework will cover the whole life cycle of the design, specification and application of data warehouses and their use in tourist information systems (TIS) and strategic decision support systems in tourism. Since many tourist information systems offer WWW-interfaces based on HTML-files and Java applets, we consider to use internet/intranet as the end user interface for any analysis, report and retrieval operation on the data warehouse. Special efforts will be given on the potential of data mining techniques, which are used on top of modern data warehouses

The Semantic Web (SW) has drawn the attention of data enthusiasts, and also inspired the exploitation and design of multidimensional data warehouses, in an unconventional way. Traditional data warehouses (DW) operate over static data. However multidimensional (MD) data modeling approach can be dynamically extended by defining both the schema and instances of MD data as RDF graphs. The importance and applicability of MD data warehouses over RDF is widely studied yet none of the works support a spatially enhanced MD model on the SW. Spatial support in DWs is a desirable feature for enhanced analysis, since adding encoded spatial information of the data allows to query with spatial functions. In this paper we propose to empower the spatial dimension of data warehouses by adding spatial data types and topological relationships to the existing QB4OLAP vocabulary, which already supports the representation of the constructs of the MD models in RDF. With QB4SOLAP, spatial constructs of the MD models can be also published in RDF, which allows to implement spatial and metric analysis on spatial members along with OLAP operations. In our contribution, we describe a set of spatial OLAP (SOLAP) operations, demonstrate a spatially extended metamodel as, QB4SOLAP, and apply it on a use case scenario. Finally, we show how these SOLAP queries can be expressed in SPARQL.

Information systems allow companies to collect a large number of operational and transactional data. Data warehouses are increasingly used by organisations to extract concise information supporting decision processes. However, data warehouse design strategies as well as the structure and content of the original database largely influence the effectiveness of such tools. Our research is focused on data-driven approaches, and in this paper, we present a solution that, based on a set of metrics measuring different characteristics of the original data sources, effectively supports the creation of data warehouses. Moreover, combining the indexes computed by our metrics on selected dimensions and measures, it is also possible to derive quantitative information on the design quality of the final data warehouse. To demonstrate the effectiveness of our solution, we briefly present the results obtained from the analysis of a selling data warehouse derived from a real-word ERP system.

The proliferation of projects that are part of the semantic Web is truly impressive. In fact, ontologies become increasingly present in information systems, they constitute great data sources that arouse the interest of being analyzed. Ontologies are used for standardizing, structuring and formalizing the Web, Web Service, E-learning systems, and other fields. Regarding multidimensional approaches, researches in this field have focused on direct Data Warehouse conception from an ontology, which do not integrate the intervention of the expert in this process. In this case, the transformations are global and not very customizable; it can reproduce the inherent defects from the data sources into the resulting data warehouse. In this paper, we propose a new multi-approach model based on the coupling of relational database design approaches from an ontology with Data Warehouse design approaches from a relational database. Our model is semi-automatic allowing Data Warehouse design from an ontology by giving the designer more ability to intervene in this process and closely control the transformations. To assess the usefulness of our approach, we evaluated it by applying it on an example case study. The results of the example show that our approach is more accurate in terms of useful data filtering and adaptation of the multidimensional model to the end-users business-needs.

Data Warehouses (DWs) are currently considered to be the cornerstone of Business Intelligence (BI) systems. Security is a key issue in DWs since the business information that they manage is crucial and highly sensitive, and should be carefully protected. However, the increasing amount of data available on the Web signifies that more and more DW systems are considering the Web as the primary data source through which to populate their DWs. XML is therefore widely accepted as being the principal means through which to provide easier data and metadata interchange among heterogeneous data sources from the Web and the DW systems.Although security issues have been considered during the whole development process of traditional DWs, current research lacks approaches with which to consider security when the target platform is based on the Web and XML technologies. The idiosyncrasy of the unstructured and semi-structured data available on the Web definitely requires particular security rules that are specifically tailored to these systems in order to permit their particularities to be captured correctly.In order to tackle this situation, in this paper, we propose a methodological approach based on the Model Driven Architecture (MDA) for the development of Secure XML DWs. We therefore specify a set of transformation rules that are able to automatically generate not only the corresponding XML structure of the DW from secure conceptual DW models, but also the security rules specified within the DW XML structure, thus allowing us to implement both aspects simultaneously. A case study is provided at the end of the paper to show the benefits of our approach.

Today, the term Big Data is already widely distributed in many areas, and one of the efficient tools to handle this amount of data is a young, but rapidly growing data warehousing technology. This article reviews general data warehousing understanding and its areas of usage and application in order to understand and implement data integration solutions. Traditional design and concept model of data warehouse is explained. In addition we will overview of data warehouse organization and data extraction to Hadoop architecture which has long been associated with the concept of Big Data and allows distributed processing of it on clusters of commodity hardware.