Rule-Based Conditioning of Probabilistic Data

The enormous volumes of data created and maintained by industries, research institutions are on the verge of outgrowing its infrastructure. The advancements in the organization's work flow include data storage, data management, data maintenance, data integration, and data interoperability. Among these levels, data integration and data interoperability can be the two major focus areas for the organizations which tend to implement advancements in their workflow. Overall, data integration and data interoperability influence the organization's performance. The data integration and data interoperability are complex challenges for the organizations deploying big data architectures due to the heterogeneous nature of data used by them. Therefore, it requires a comprehensive approach to negotiate the challenges in integration and interoperability. This paper focuses on the challenges of data integration and data interoperability in big data.

Many data driven organisations need to integrate data from multiple, distributed and heterogeneous resources for advanced data analysis. A data integration system is an essential component to collect data into a data warehouse or other data analytics systems. There are various alternatives of data integration systems which are created in-house or provided by vendors. Hence, it is necessary for an organisation to compare and benchmark them when choosing a suitable one to meet its requirements. Recently, the TPC-DI is proposed as the first industrial benchmark for evaluating data integration systems. When using this benchmark, we find some typical data quality problems in the TPC-DI data source such as multi-meaning attributes and inconsistent data schemas, which could delay or even fail the data integration process. This paper explains processes of this benchmark and summarises typical data quality problems identified in the TPC-DI data source. Furthermore, in order to prevent data quality problems and proactively manage data quality, we propose a set of practical guidelines for researchers and practitioners to conduct data quality management when using the TPC-DI benchmark.

Data integration is a critical problem in our increasingly interconnected but inevitably heterogeneous world. There are numerous data sources available in organizational databases and on public information systems like the World Wide Web. Not surprisingly, the sources often use different vocabularies and different data structures, being created, as they are, by different people, at different times, for different purposes. The goal of data integration is to provide programmatic and human users with integrated access to multiple, heterogeneous data sources, giving each user the illusion of a single, homogeneous database designed for his or her specific need. The good news is that, in many cases, the data integration process can be automated. This book is an introduction to the problem of data integration and a rigorous account of one of the leading approaches to solving this problem, viz., the relational logic approach. Relational logic provides a theoretical framework for discussing data integration. Moreover, in many important cases, it provides algorithms for solving the problem in a computationally practical way. In many respects, relational logic does for data integration what relational algebra did for database theory several decades ago. A companion web site provides interactive demonstrations of the algorithms. Table of Contents: Preface / Interactive Edition / Introduction / Basic Concepts / Query Folding / Query Planning / Master Schema Management / Appendix / References / Index / Author Biography

After years accumulation of informationization construction, power enterprises have established huge and complex application systems, which include multi-class data related to business like production, operation and management. Along with the demands of developing new business application, the data integration is urgently needed to grasp overall data view; however the complex data integration environment causes the data quality problem particularly outstanding. Under the background of the power industry, this paper first studies the data quality definition and dimension recognition; then proposes a data integration platform architecture and studies the data quality control methods in data integration; elaborates data quality assessment methods and procedures; in the view of data product, designs a total data quality management (TDQM) process finally. In a typical case of application, the results show the practicability of the proposed method.

Uncertain data are inherent in some important applications, such as environmental surveillance, market analysis, and quantitative economics research. Due to the importance of those applications and the rapidly increasing amount of uncertain data collected and accumulated, analyzing large collections of uncertain data has become an important task and has attracted more and more interest from the database community. Recently, uncertain data management has become an emerging hot area in database research and development. In this tutorial, we systematically review some representative studies on answering various queries on uncertain and probabilistic data.

In data integration efforts, portal development in particular, much development time is devoted to entity resolution. Often advanced similarity measurement techniques are used to remove semantic duplicates or solve other semantic conflicts. It proves impossible, however, to automatically get rid of all semantic problems. An often-used rule of thumb states that about 90% of the development effort is devoted to semi-automatically resolving the remaining 10% hard cases. In an attempt to significantly decrease human effort at data integration time, we have proposed an approach that strives for a `good enough' initial integration which stores any remaining semantic uncertainty and conflicts in a probabilistic database. The remaining cases are to be resolved with user feedback during query time. The main contribution of this paper is an experimental investigation of the effects and sensitivity of rule definition, threshold tuning, and user feedback on the integration quality. We claim that our approach indeed reduces development effort--and not merely shifts the effort--by showing that setting rough safe thresholds and defining only a few rules suffices to produce a `good enough' initial integration that can be meaningfully used, and that user feedback is effective in gradually improving the integration quality.

Nowadays, high volumes of valuable uncertain data can be easily collected or generated at high velocity in many real-life applications. Mining these uncertain Big data is computationally intensive due to the presence of existential probability values associated with items in every transaction in the uncertain data. Each existential probability value expresses the likelihood of that item to be present in a particular transaction in the Big data. In some situations, users may be interested in mining all frequent patterns from these uncertain Big data; in other situations, users may be interested in only a tiny portion of these mined patterns. To reduce the computation and to focus the mining for the latter situations, we propose a tree-based algorithm that (i) allows users to express the patterns to be mined according to their intention via the use of constraints and (ii) uses MapReduce to mine uncertain Big data for only those frequent patterns that satisfy user-specified constraints. Experimental results show the effectiveness of our algorithm in mining probabilistic databases of uncertain Big data.

Most geostatistical estimation and simulation methodologies assume the experimental data as hard measurements, meaning that the measures of a given property of interest are not associated with uncertainty. The challenge of integrating uncertain experimental data at the geostatistical estimation or simulation models is not new. Several attempts have been made, either considering the uncertain data as soft data or interpreting it as inequality constraints, based on the indicator formalism or decreasing the weight of soft data in kriging procedures. This paper presents a stochastic simulation methodology where the uncertain experimental data are modelled by a probability distribution at each sample location. Data values are firstly drawn, by stochastic simulation, at these locations prior to the simulation of the rest of the grid nodes. This method is also extended to the simulation of categorical uncertain data, as well as to the simulation with uncertain block support data. To illustrate the proposed methodology, an application to a real case study of pore pressure prediction of oil reservoirs is presented, as well as an upscaling problem.

Academia, industry and government as well, are involved in big data projects. Many researches on big data applications and technologies are actively being conducted. This paper presents a literature review of recent researches on key technologies and open issues for big data management via cloud computing. Its goal is to identify and evaluate the main technology components and their impacts on cloud-based big data implementations. This is achieved by reviewing 40 publications published in the latest four years, 2014–2017. We classified the results based on the main technical aspects: frameworks, databases and data processing techniques, and programming languages. This paper also provides a reference source for researchers and developers, to determine the best emerging technologies for big data project implementation.

<p>Since initiated by the Chinese Academy of Sciences (CAS) and the National Earth Observation Data Center (NODA) of China, Cooperation on Reanalysis of Carbon Satellite Data (CASA) had already advanced to the second stage. China aims to hit peak emissions before 2030 and for carbon neutrality by 2060.Carbon neutrality research involves Terrestrial-Marine-Atmospheric multiple fields, which inevitably require the support of scientific big data and Scientific Data e-Infrastructure (SDI). Open space-borne carbon data interconnectivity and interoperability across the massive carbon data (GOSAT, GOSAT-2, OCO-2/3, TanSat, Sentinel-5P, FY-3D, GF-5 and the second generation carbon satellites) and related auxiliary data resources integrated into the CASA platform is a key enabler to become more data-driven, to broader data value, and to meet the major demand of global and regional monitoring of anthropogenic carbon emissions. This study explores the technological barriers for carbon satellite data interconnectivity, discusses the concepts of carbon data interoperability and integration, management and governance in more detail, highlight some useful tools, and demonstrate examples in urban air pollution and CO<sub>2</sub> emissions that can help researchers in their application studies upon estimation of anthropogenic carbon emissions based on “top-down” methods. We linked carbon data connection and interoperability both to carbon data collection and use within programmatic cycles and reflected interoperability both in organizational practices and data management plans that cover the full breadth of the data value chain. This will extend carbon data information service and provide better ways to utilizing carbon data across domains where innovation and integration are now necessarily needed.</p>

Data integration has become one of the main challenges in the era of Big Data analytics. Often to enable decision-making, data from different sources have to be integrated and linked together. For example, multi-source data integration is vital to police, counter terrorism and national security to allow efficient and accurate verification of people. One of the key challenges in the data integration process is matching records that represent the same real-world entity (e.g. person). This process is referred to as record linkage. In many cases, data sets do not share a unique identifier (e.g. National Insurance Number), hence records need to be matched by comparing their corresponding attributes. Most of the existing record linkage methods require assistance from a domain expert for handcrafting domain-specific linking rules. More automatic approaches, based on using machine learning, were also proposed. However, those approaches relay on having a substantial set of manually labelled records, which makes them inapplicable in real-world scenarios. Given the importance of the problem, record linkage has witnessed a strong interest in the past decade. As a result, significant progress has been made in this area. In particular, the problem of reducing the manual effort and the amount of labelled data required for constructing record linkage models has been addressed in many studies. In this chapter, we review the most recently proposed approaches to semi-supervised and unsupervised record linkage.

Complex subsurface operations are characterized by a life‐threatening environment, a skilled and initiative opponent, and the absence of predictability of the events due to a high level of interdependencies. The reduction of complexity by provision of essential information is crucial for decision‐making and rapid integration and visualization of heterogeneous data is essential for successful mission accomplishment. Currently, only standalone applications are available for the underground operational environment, and collaborative planning and working spaces in command and control are missing. The RApid Data Integration and Visualization (RADIV) process addresses exactly this challenge and ensures the lateral continuity of visualization systems across the entire reality‐virtuality continuum (2D ⇔ 3D ⇔ mixed reality). It provides a comprehensive command and control system for subterranean operations by processing and visualizing data in different views for different purposes. Integration of these data within the Subsurface Operations Mission Tool (SOMT) will increase the decision quality by improved perception and collaboration. Close cooperation and information exchange between operators and action forces is a prerequisite for success by displaying the relevant information within the truly comprehensive common operational picture, thereby enabling more accurate and precise action reducing own losses and collateral damage.

To meet customer demands, companies must manage numerous variants and versions of their products. Since product-related data (e.g., requirements’ specifications, geometric models, and source code, or test cases) are usually scattered over a large number of heterogeneous, autonomous information systems, their integration becomes crucial when developing complex products on one hand and aiming at reduced development costs on the other. In general, product data are created in different stages of the product development process. Furthermore, they should be integrated in a complete and consistent way at certain milestones during process development (e.g., prototype construction). Usually, this data integration process is accomplished manually, which is both costly and error prone. Instead semi-automated product data integration is required meeting the data quality requirements of the various stages during product development. In turn, this necessitates a close monitoring of the progress of the data integration process based on proper metrics. Contemporary approaches solely focus on metrics assessing schema integration, while not measuring the quality and progress of data integration. This paper elicits fundamental requirements relevant in this context. Based on them, we develop appropriate metrics for measuring product data quality and apply them in a case study we conducted at an automotive original equipment manufacturer.KeywordsProduct data integrationIntegration qualityIntegration Process

Mesoeconomics represents a critical intermediate layer of economic analysis, bridging the gap between microeconomics and macroeconomics. With the growing complexity of global supply chains, traditional economic frameworks and data analysis techniques have proven insufficient. The increasing availability and use of big data technologies have highlighted the relevance of mesoeconomic approaches in analyzing networked economic interactions across international supply chains. The research aims to explore how mesoeconomic frameworks can be effectively utilized to analyze global supply chain networks through big data analytics. It seeks to identify the advantages of applying mesoeconomic analysis in understanding the structure, flow, and dynamics of supply chains, as well as the challenges associated with data integration and analytics. The research employs a qualitative and conceptual analysis of mesoeconomic theory in the context of supply chains, supplemented by a review of current big data analytics methods used by companies. The analysis includes an examination of various data sources (primary and secondary), network structures, and classification techniques that support the processing and interpretation of large-scale data related to goods and services flows. The findings suggest that mesoeconomics, when combined with big data analytics, offers a robust framework for capturing the complexities of global supply chains. This approach enhances the understanding of relationships among products, producers, and processes, enabling more efficient and informed decision-making. However, challenges such as data quality, interoperability, privacy, and security remain significant barriers to the full realization of big data’s potential in supply chain analysis.

The process and functioning of data integration is termed as combining information from several sources to provide users with a coherent perspective. The fundamental idea behind data integration is to open up data and make it simpler for individuals and systems to access, utilize, and process. The process of converting data from one format to another, typically from that of a source system into that required by a destination system, is known as data transformation. Data transformation is a component of the majority of data integration and management processes, including data manipulation and data warehousing. Many organizations carry out data transformation and integration because they have requirements with respect to data usage that is important in every situation. This paper proposes an architecture that reduces manual work and abstracts the decisions to be made in the integration and transformation process. This approach can lower the risk of human error and result in significant financial savings for various organizations. A modular approach is followed to ease these complex tasks and get desired results.