A Framework for Supporting Interoperability of Data Warehouse Islands Using XML

In this paper author presents an experiment, which shows that it is possible to form a federation of data warehouses that may simulate effectively one, “super” data warehouse. There is no need to create complete ETL tool to load data from source data warehouses into one, dedicated data warehouse. Good relations between global schema and local schemas extracted during schema integration are indispensable to create an effective federation.

With the increasing personalization of clinical therapies, translational research is evermore dependent on multisite research cooperations to obtain sufficient data and biomaterial. Distributed research networks rely on the availability of high-quality data stored in local databases operated by their member institutions. However, reusing data documented by independent health providers for the purpose of care, rather than research ("secondary use"), reveal a high variability in terms of data formats, as well as poor data quality, across network sites. The aim of this work is the provision of a process for the assessment of data quality with regard to completeness and syntactic accuracy across independently operated data warehouses using common definitions stored in a central (network-wide) metadata repository (MDR). For assessment of data quality across multiple sites, we employ a framework of so-called bridgeheads. These are federated data warehouses, which allow the sites to participate in a research network. A central MDR is used to store the definitions of the commonly agreed data elements and their permissible values. We present the design for a generator of quality reports within a bridgehead, allowing the validation of data in the local data warehouse against a research network's central MDR. A standardized quality report can be produced at each network site, providing a means to compare data quality across sites, as well as to channel feedback to the local data source systems, and local documentation personnel. A reference implementation for this concept has been successfully utilized at 10 sites across the German Cancer Consortium. We have shown that comparable data quality assessment across different partners of a distributed research network is feasible when a central metadata repository is combined with locally installed assessment processes. To achieve this, we designed a quality report and the process for generating such a report. The final step was the implementation in a German research network.

Interoperability and the exchange of metadata between photographic archives has become an important issue for archives digitizing their collections. However, due to incompatible metadata formats and incompatible storage and dissemination software, interoperability is hard to achieve. In order to facilitate these archives and tackle aforementioned problems, we propose a framework using an extensive component based data element set for photographic metadata (SEPIADES) in combination with the XML based Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). Using this framework, archives can assemble their own application profile consisting of SEPIADES elements and use this as a metadata format for description, storage and dissemination of photographic metadata records. Due to the general nature of the problems being solved by this framework, it is proposed that a standardization effort will follow based on the framework. As a consequence, this effort will provide an improved way to access multiple collections based on established standards.

Queries using outerjoins appear very frequently in traditional applications such as data warehousing. Lately, they have been widely used in newly emerged systems such as Object-Relational Mapping (ORM) tools, schema integration and information exchange systems, and probabilistic databases. Materialized views using outerjoins are allowed in many database management systems but without support for their incremental maintenance. In this paper we present the algorithms used in SQL Anywhere RDBMS for the incremental maintenance of materialized views with outerjoins. The algorithms achieve the following improvements over the previous work with respect to the class of materialized outerjoin views which can be incrementally maintained, and with respect to the performance of the view updates: (1) Relax the requirement for the existence of the primary key attribute in the select list of the view to only some of the relations (namely only the relations. referenced as a preserved side in an outerjoin predicate). 2) Relax the null-intolerant property requirement for only some predicates used in the view definition (namely, those outerjoin predicates referencing relations which can be null-supplied by another nested outerjoin). 3) The maintenance of outerjoin views is implemented by using exactly one update statement per view for each relation referenced in the view.Another main characteristic of the algorithms is that they allow the design and implementation of the incremental maintenance of materialized views with outerjoins to be easily integrated into the SQL Anywhere Optimizer by relying on the normalized join tree representation used for optimizing queries with outerjoins.

In the face of exponential data growth, libraries grapple with effectively managing and extracting knowledge from massive datasets. Data warehousing technologies have emerged as indispensable tools to address this challenge. This research investigates the significance of data warehousing within the library domain, focusing on its capacity to streamline data management and analysis processes. A comparative assessment of ten prominent data warehousing platforms is conducted, including Teradata Vantage, OCLC WorldShare Analytics, Snowflake, Amazon Redshift, IBM Db2 Warehouse, Alma, Oracle Autonomous Data Warehouse, SAP Data Warehouse Cloud, Microsoft Azure Synapse Analytics, and Google BigQuery. This evaluation scrutinizes crucial aspects such as architectural design, user-friendliness, applicability within library contexts, security mechanisms, distinctive features, and, crucially, cost-effectiveness. Particular emphasis is placed on identifying solutions that harmonize functionality with the budgetary realities of library operations. The study elucidates the strengths and shortcomings of each platform, providing actionable guidance for libraries to select the most suitable data warehousing solution based on their unique operational requirements and financial limitations. This research underscores the imperative of adopting a suitable data warehousing tool to augment data management and analytical capabilities within libraries, thereby facilitating informed decision-making and enhancing service delivery.

Today’s concept of Open Government Data (OGD) for openness, transparency and ease of access of data owned by government agencies becomes increasingly important. This initiative emerges from the demand of data users for the data belongs to the government agencies. The data services providing an easy access, cheap, fast, and interoperability are needed by the users and becomes important indicator performance for respective government agencies. Statistical Data and Metadata Exchange (SDMX) is a new standard format in the data dissemination activities particularly in the exchange of statistical data and metadata via Internet. In this respect SDMX support the implementation of OGD project. This paper is on the technical design, development and implementation of data and metadata exchange service of statistical data using SDMX format to support interoperability data through web services. Three results are proposed: (i) framework for standardization of structure of statistical publications data model with SDMX; (ii) design architecture of data sharing model; and (iii) web service implementation of data and metadata exchange service using Service Oriented Analysis and Design (SOAD) method. Implementation at Statistics Indonesia (BPS) is chosen as a case study to prove the design concept. It is shown through quantitative assessment and black box testing that the design achieves its objective. DOI: http://dx.doi.org/10.11591/telkomnika.v14i2.7505

Today, with the increase in the number of business web application users, the data size is obviously getting quiet huge. This data is composed together from integrated heterogeneous data sources to form Data Warehouse (DW). DW consists of critical and sensitive data used in many intelligent business models for decision support processes. Therefore, retaining data security for DW systems is a very important issue for researchers; also, it is the first priority for organizations. The current paper presents a new framework for implementing security issues in DWs named Data Warehouse Signature (DWS). The DWS framework ensures the security issues, which are Confidentiality, Integrity and Availability (CIA). In fact, this approach solves the security problems such as network overhead and data changing that occurs during transmission on Client-Server basis. The DWS approach achieves high performance by using parallel computing through a middleware named View Manager Layer (VML). Finally, this paper also presents the elastic architecture of the DWS framework with a vast applicability in multiple DW organizations depending on its network infrastructure.

One of the important research and technological issues in data warehouse performance is the optimization of analytical queries. Most of the research have been focusing on optimizing such queries by means of materialized views, data and index partitioning, as well as various index structures including: join indexes, bitmap join indexes, multidimensional indexes or index-based multidimensional clusters. These structures neither well support navigation along dimension hierarchies nor optimize joins with the Time dimension, which in practice is used in the majority of analytical queries. In this chapter we overview the basic index structures, namely: a bitmap index, a join index, and a bitmap join index. Based on these indexes, we show how to build another index, called Time-HOBI, for optimizing queries that address the Time dimension and compute aggregates along dimension hierarchies. We further discuss the extension of the index with additional data structure for storing aggregate values along the hierarchical structure of the index. The aggregates are used for speeding up aggregate queries along dimension hierarchies. Furthermore, we show how the index is used for answering queries in an example data warehouse. Finally, we discuss its performance-related characteristics, based on experiments.

Data warehouses are information repositories specialized in supporting decision making. Since the decisional process typically requires an analysis of historical trends, time and its management acquire a huge importance. In this paper we consider the variety of issues, often grouped under term temporal data warehousing, implied by the need for accurately describing how information changes over time in data warehousing systems. We recognize that, with reference to a three-levels architecture, these issues can be classified into some topics, namely: handling data/schema changes in the data warehouse, handling data/schema changes in the data mart, querying temporal data, and designing temporal data warehouses. After introducing the main concepts and terminology of temporal databases, we separately survey these topics. Finally, we discuss the open research issues also in connection with their implementation on commercial tools.

DaWaII (Data Warehouse IntegratIon) is a tool supporting the various activities related to the integration of multidimensional data. This problem arises in common scenarios where there is the need to combine independently developed data warehouses. Actually, a today common practice for building a data warehouse is to develop a collection of integrated data marts, each of which provide a dimensional view of a single business process. These data marts should be based on shared dimensions but very often, even within the same company, designers develop their data marts independently and it turns out that their integration is a difficult task. Indeed, the problem arises in other common cases. For instance, when different companies get involved in a federated project or when there is the need to combine a proprietary data warehouse with external information, for instance, with multidimensional data wrapped from the Web.

4 - Physical Design of the Data Warehouse

Open-access thematic data portals play a critical role in advancing an open data culture by addressing knowledge gaps and reducing uncertainties in research. The Arctic Permafrost Geospatial Center (APGC) serves as a pivotal platform for providing high-quality geospatial data to the permafrost research community. By facilitating easy access to diverse data products, APGC supports multi-scale, interdisciplinary analyses that integrate field observations, remote sensing, and modeling efforts.APGC’s primary objectives are to ensure (i) the delivery of data that is impactful, user-friendly, and scientifically valuable, and (ii) to streamline data discovery, visualization, and metadata exchange through its comprehensive data catalog, accessible at https://apgc.awi.de. This portal is well-suited to support research initiatives and fulfills requirements for publishing and visualizing data and metadata, a growing necessity in securing project funding.The APGC catalog accommodates datasets of varying formats, spatial scales, temporal extents, and themes. Users can search datasets by geographic location—either through spatial keywords or interactive map selection—as well as by category, product type, project, tags, keywords, licensing, or data format. Download links provide direct access to repositories such as PANGAEA, ensuring seamless data acquisition.Built on the open-source CKAN framework and utilizing the DCAT metadata standard, the catalog adheres to FAIR (Findable, Accessible, Interoperable, Reusable) data principles. Each dataset is accompanied by detailed metadata, a concise abstract, and a preview option, with metadata available in multiple formats such as RDF/XML, JSON, or Turtle.Initially established through the ERC PETA-CARB and ESA GlobPermafrost projects, APGC now hosts over 360 curated datasets from various sources. These datasets encompass a wide range of permafrost-related themes, including surface and subsurface characteristics such as soil temperature, carbon content, ground ice, land cover, vegetation, periglacial landforms, and subsidence. Data formats range from vector and raster to time series and are derived from in-situ measurements, Earth observation, and modeling studies. WebGIS tools further enhances user engagement by enabling interactive exploration of most datasets (https://maps.awi.de & https://apgc-map.awi.de/).APGC encourages data contributions from individual researchers and project consortia. Submitted datasets undergo evaluation based on criteria such as relevance to permafrost research, scientific significance, accessibility, quality, and metadata completeness. To ensure long-term preservation and accessibility, datasets must be archived in repositories like PANGAEA.By integrating tools for data publication, visualization, and long-term preservation, APGC provides an essential service for research projects, enabling them to meet funding requirements while advancing the understanding of permafrost systems across the Arctic and beyond.

Barely more than 10 years after the birth of the World Wide Web, the Global Information Infrastructure is daily reality. In spite of the many applications in all domains of our societies: e-business, e-commerce, e-learning, e-science, and e-government, for instance, and in spite of the tremendous advances by engineers and scientists, the seamless integration of information and services remains a major challenge. The current shared vision for the future is one of semantically rich information and service oriented architecture for global information systems. This vision is at the convergence of progress in technologies such as XML, Web services, RDF, OWL, of multimedia, multimodal, and multilingual information retrieval, and of distributed, mobile and ubiquitous computing. Semantic Information Management (SIM) can be identified as the discipline that studies the integration and management of multi-modal information from distributed, heterogeneous, and autonomous sources based on their meaning. SIM is deemed necessary since there is an increasing tendency of inter-business and inter-applications sharing of data. The data can be written and stored using different standard and format. Hence, we need a platform to manage the data so that it can be universally understood. SIM as a research has variety research challenges. The first challenge is the identification of suitable meta-model that can ensure the quality of the integrated data. The next challenge is the adoption of ever-increasing number of standards used by enterprises. To comply with standards, enterprises have to represent their data uniquely and thus, the challenge to integrate them with data from other enterprises increases. XML has become a standard data format widely used in many organizations, thus leading to a growing need for exchanging and integrating multiple XML data sources across different application systems. XML has been regarded as a de-facto standard for data exchanges in enterprises. Integrating these sources requires an integration of their schemas. The main challenge is schema integration where the semantic is maintained. Aligning with the previous challenge, many enterprises these days have to comply with various government and professional regulations in managing their data. It makes the integration of data from multi-discipline enterprises and cross-regulated regions even more difficult. Another challenge is the ability to trace back the integrated information to its original source and format. The issue of forward and backward compatibility is far more complex than in traditional data integration research domain. The issue becomes more complex in a social network setting. Social networks consist mainly of groups of inter-connected people. This has played an important role in changing the way people interact with each other. Social networks analysis focuses on studying patterns of communication and exchange of information between people, which may influence not only on the individuals who adopt them, but also on the societies and organizations that enclose them. Since social network is not only heterogeneous, SIM in social networks imposes even a greater challenge, since each social group may have their own information semantic, and therefore, semantic in a global information exchange is a complex problem. The availability of cloud computing makes SIM even more challenging. Cloud computing has been increasingly regarded as a revolution in the computing practices through the concept of utility computing, whereby computing services are provided as utilities. There are various number of services provided by the cloud, from computing power, storage, to various software applications. SIM in the cloud will provide Information as a Service, a totally novel concept. The main challenge will be to manage such information so that the G. Kotsis (*) : I. Khalil Department of Telecooperation, Johannes Kepler University Linz, Altenberger Strasse 69, 4040, Linz, Austria e-mail: gabriele.kotsis@jku.at

Business Intelligence (BI) technology with Extract, Transform, and Loading process, Data Warehouse, and OLAP have demonstrated the ability of information and knowledge generation for supporting decision making. In the last decade, the advancement of the Web 2.0 technology is improving the accessibility of web of data across the cloud. Linked Open Data, Linked Open Statistical Data, and Open Government Data is increasing massively, creating a more significant computer-recognizable data available for sharing. In agricultural production analytics, data resources with high availability and accessibility is a primary requirement. However, today’s data accessibility for production analytics is limited in the 2 or 3-stars open data format and rarely has attributes for spatiotemporal analytics. The new data warehouse concept has a new approach to combine the openness of data resources with mobility or spatiotemporal data in nature. This new approach could help the decision-makers to use external data to make a crucial decision more intuitive and flexible. This paper proposed the development of a spatiotemporal data warehouse with an integration process using service-oriented architecture and open data sources. The data sources are originating from the Village and Rural Area Information System (SIDeKa) that capture the agricultural production transaction in a daily manner. This paper also describes the way to spatiotemporal analytics for agricultural production using a new spatiotemporal data warehouse approach. The experiment results, by executing six relevant spatiotemporal query samples on DW with fact table contains 324096 tuples with temporal integer/float for each tuple, 4495 tuples of field dimension with geographic data as polygons, 80 tuples of village dimension, dozens of tuples of the district, subdistrict, province dimensions. The DW time dimension contains 3653 tuples representing a date for ten years, proved that this new approach has a convenient, simple model, and expressive performance for supporting executive to make decisions on agriculture production analytics based on spatiotemporal data. This research also underlines the prospects for scaling and nurturing the spatiotemporal data warehouse initiative.

BackgroundData harmonisation (DH) has emerged amongst health managers, information technology specialists and researchers as an important intervention for routine health information systems (RHISs). It is important to understand what DH is, how it is defined and conceptualised, and how it can lead to better health management decision-making. This scoping review identifies a range of definitions for DH, its characteristics (in terms of key components and processes), and common explanations of the relationship between DH and health management decision-making.MethodsThis scoping review identified relevant studies from 2000 onwards (date filter), written in English and published in PubMed, Web of Science and CINAHL. Two reviewers independently screened records for potential inclusion for the abstract and full-text screening stages. One reviewer did the data extraction, analysis and synthesis, with built-in reliability checks from the rest of the team. We developed a narrative synthesis of definitions and explanations of the relationship between DH and health management decision-making.ResultsWe sampled 61 of 181 included to synthesis definitions and concepts of DH in detail. We identified six common terms for data harmonisation: record linkage, data linkage, data warehousing, data sharing, data interoperability and health information exchange. We also identified nine key components of data harmonisation: DH involves (a) a process of multiple steps; (b) integrating, harmonising and bringing together different databases (c) two or more databases; (d) electronic data; (e) pooling data using unique patient identifiers; and (f) different types of data; (g) data found within and across different departments and institutions at facility, district, regional and national levels; (h) different types of technical activities; (i) has a specific scope. The relationship between DH and health management decision-making is not well-described in the literature. Several studies mentioned health providers’ concerns about data completeness, data quality, terminology and coding of data elements as barriers to data utilisation for clinical decision-making.ConclusionTo our knowledge, this scoping review was the first to synthesise definitions and concepts of DH and address the causal relationship between DH and health management decision-making. Future research is required to assess the effectiveness of data harmonisation on health management decision-making.