RApid Data Integration and Visualization (RADIV) in Subsurface Operations

Ontologies enable explicit expression of collective concepts and support Machine-to-Machine (M2M) interactions at the semantic level. Ontologies expressed in a standard language, such as the Web Ontology Language (OWL) and exposed on a network offer the potential for unprecedented interoperability solutions since they are semantically rich, computer interpretable and inherently extensible. In this chapter, we describe how we applied ontologies in OWL for rapid enterprise integration of heterogeneous data sources to track objects in a battlespace. We found that once a robust foundational domain ontology is established, it is easy and quick to integrate new data sources and therefore rapidly provide new system capabilities. In particular, we demonstrate how moving tracks can be quickly integrated with intelligence and space events to provide enhanced situational awareness using ontologies. This chapter also describes the overall SEER and SWORIER systems we developed, the latter of which translated OWL ontologies (and RDF instances) and Semantic Web Rule Language (SWRL) rules into Prolog, applied knowledge compilation techniques, and then at runtime, utilized a combined OWL/logic programming reasoned for efficient automated reasoning. We also briefly describe a more recent extension to the prototype and to the ontologies that we made to address more rigorous geospatial rules for unmanned autonomous vehicle (UAV) avoidance. Finally, we consider some issues raised by our work and future lines of research to address these.

Data integration is a critical problem in our increasingly interconnected but inevitably heterogeneous world. There are numerous data sources available in organizational databases and on public information systems like the World Wide Web. Not surprisingly, the sources often use different vocabularies and different data structures, being created, as they are, by different people, at different times, for different purposes. The goal of data integration is to provide programmatic and human users with integrated access to multiple, heterogeneous data sources, giving each user the illusion of a single, homogeneous database designed for his or her specific need. The good news is that, in many cases, the data integration process can be automated. This book is an introduction to the problem of data integration and a rigorous account of one of the leading approaches to solving this problem, viz., the relational logic approach. Relational logic provides a theoretical framework for discussing data integration. Moreover, in many important cases, it provides algorithms for solving the problem in a computationally practical way. In many respects, relational logic does for data integration what relational algebra did for database theory several decades ago. A companion web site provides interactive demonstrations of the algorithms. Table of Contents: Preface / Interactive Edition / Introduction / Basic Concepts / Query Folding / Query Planning / Master Schema Management / Appendix / References / Index / Author Biography

The process and functioning of data integration is termed as combining information from several sources to provide users with a coherent perspective. The fundamental idea behind data integration is to open up data and make it simpler for individuals and systems to access, utilize, and process. The process of converting data from one format to another, typically from that of a source system into that required by a destination system, is known as data transformation. Data transformation is a component of the majority of data integration and management processes, including data manipulation and data warehousing. Many organizations carry out data transformation and integration because they have requirements with respect to data usage that is important in every situation. This paper proposes an architecture that reduces manual work and abstracts the decisions to be made in the integration and transformation process. This approach can lower the risk of human error and result in significant financial savings for various organizations. A modular approach is followed to ease these complex tasks and get desired results.

Big data has emerged as the next technological revolution in IT industry after cloud computing and the Internet of Things. With the development of climate observing systems, particularly satellite meteorological observation and high-resolution climate models, and the rapid growth in the volume of climate data, climate prediction is now entering the era of big data. The application of big data will provide new ideas and methods for the continuous development of climate prediction. The rapid integration, cloud storage, cloud computing, and full-sample analysis of massive climate data makes it possible to understand climate states and their evolution more objectively, thus predicting the future climate more accurately. This paper describes the application status of big data in operational climate prediction in China; it analyzes the key big data technologies, discusses the future development of climate prediction operations from the perspective of big data, speculates on the prospects for applying climatic big data in cloud computing and data assimilation, and puts forward the notion of big data-based super-ensemble climate prediction methods and computerbased deep learning climate prediction methods.

To meet customer demands, companies must manage numerous variants and versions of their products. Since product-related data (e.g., requirements’ specifications, geometric models, and source code, or test cases) are usually scattered over a large number of heterogeneous, autonomous information systems, their integration becomes crucial when developing complex products on one hand and aiming at reduced development costs on the other. In general, product data are created in different stages of the product development process. Furthermore, they should be integrated in a complete and consistent way at certain milestones during process development (e.g., prototype construction). Usually, this data integration process is accomplished manually, which is both costly and error prone. Instead semi-automated product data integration is required meeting the data quality requirements of the various stages during product development. In turn, this necessitates a close monitoring of the progress of the data integration process based on proper metrics. Contemporary approaches solely focus on metrics assessing schema integration, while not measuring the quality and progress of data integration. This paper elicits fundamental requirements relevant in this context. Based on them, we develop appropriate metrics for measuring product data quality and apply them in a case study we conducted at an automotive original equipment manufacturer.KeywordsProduct data integrationIntegration qualityIntegration Process

BackgroundNowadays, many public repositories containing large microarray gene expression datasets are available. However, the problem lies in the fact that microarray technology are less powerful and accurate than more recent Next Generation Sequencing technologies, such as RNA-Seq. In any case, information from microarrays is truthful and robust, thus it can be exploited through the integration of microarray data with RNA-Seq data. Additionally, information extraction and acquisition of large number of samples in RNA-Seq still entails very high costs in terms of time and computational resources.This paper proposes a new model to find the gene signature of breast cancer cell lines through the integration of heterogeneous data from different breast cancer datasets, obtained from microarray and RNA-Seq technologies. Consequently, data integration is expected to provide a more robust statistical significance to the results obtained. Finally, a classification method is proposed in order to test the robustness of the Differentially Expressed Genes when unseen data is presented for diagnosis.ResultsThe proposed data integration allows analyzing gene expression samples coming from different technologies. The most significant genes of the whole integrated data were obtained through the intersection of the three gene sets, corresponding to the identified expressed genes within the microarray data itself, within the RNA-Seq data itself, and within the integrated data from both technologies. This intersection reveals 98 possible technology-independent biomarkers. Two different heterogeneous datasets were distinguished for the classification tasks: a training dataset for gene expression identification and classifier validation, and a test dataset with unseen data for testing the classifier. Both of them achieved great classification accuracies, therefore confirming the validity of the obtained set of genes as possible biomarkers for breast cancer. Through a feature selection process, a final small subset made up by six genes was considered for breast cancer diagnosis.ConclusionsThis work proposes a novel data integration stage in the traditional gene expression analysis pipeline through the combination of heterogeneous data from microarrays and RNA-Seq technologies. Available samples have been successfully classified using a subset of six genes obtained by a feature selection method. Consequently, a new classification and diagnosis tool was built and its performance was validated using previously unseen samples.

A business process monitor for a mobile phone recharging system

Many data driven organisations need to integrate data from multiple, distributed and heterogeneous resources for advanced data analysis. A data integration system is an essential component to collect data into a data warehouse or other data analytics systems. There are various alternatives of data integration systems which are created in-house or provided by vendors. Hence, it is necessary for an organisation to compare and benchmark them when choosing a suitable one to meet its requirements. Recently, the TPC-DI is proposed as the first industrial benchmark for evaluating data integration systems. When using this benchmark, we find some typical data quality problems in the TPC-DI data source such as multi-meaning attributes and inconsistent data schemas, which could delay or even fail the data integration process. This paper explains processes of this benchmark and summarises typical data quality problems identified in the TPC-DI data source. Furthermore, in order to prevent data quality problems and proactively manage data quality, we propose a set of practical guidelines for researchers and practitioners to conduct data quality management when using the TPC-DI benchmark.

Data cubes for OLAP (On‐Line Analytical Processing) often need to be constructed from data located in several distributed and autonomous information sources. Such a data integration process is challenging due to semantic, syntactic, and structural heterogeneity among the data. While XML (extensible markup language) is the de facto standard for data exchange, the three types of heterogeneity remain. Moreover, popular path‐oriented XML query languages, such as XQuery, require the user to know in much detail the structure of the documents to be processed and are, thus, effectively impractical in many real‐world data integration tasks. Several Lowest Common Ancestor (LCA)‐based XML query evaluation strategies have recently been introduced to provide a more structure‐independent way to access XML documents. We shall, however, show that this approach leads in the context of certain—not uncommon—types of XML documents to undesirable results. This article introduces a novel high‐level data extraction primitive that utilizes the purpose‐built Smallest Possible Context (SPC) query evaluation strategy. We demonstrate, through a system prototype for OLAP data cube construction and a sample application in informetrics, that our approach has real advantages in data integration.

Data integration has become one of the main challenges in the era of Big Data analytics. Often to enable decision-making, data from different sources have to be integrated and linked together. For example, multi-source data integration is vital to police, counter terrorism and national security to allow efficient and accurate verification of people. One of the key challenges in the data integration process is matching records that represent the same real-world entity (e.g. person). This process is referred to as record linkage. In many cases, data sets do not share a unique identifier (e.g. National Insurance Number), hence records need to be matched by comparing their corresponding attributes. Most of the existing record linkage methods require assistance from a domain expert for handcrafting domain-specific linking rules. More automatic approaches, based on using machine learning, were also proposed. However, those approaches relay on having a substantial set of manually labelled records, which makes them inapplicable in real-world scenarios. Given the importance of the problem, record linkage has witnessed a strong interest in the past decade. As a result, significant progress has been made in this area. In particular, the problem of reducing the manual effort and the amount of labelled data required for constructing record linkage models has been addressed in many studies. In this chapter, we review the most recently proposed approaches to semi-supervised and unsupervised record linkage.

Data interoperability is a major issue in data management for data science and big data analytics. Probabilistic data integration (PDI) is a specific kind of data integration where extraction and integration problems such as inconsistency and uncertainty are handled by means of a probabilistic data representation. This allows a data integration process with two phases: (1) a quick partial integration where data quality problems are represented as uncertainty in the resulting integrated data, and (2) using the uncertain data and continuously improving its quality as more evidence is gathered. The main contribution of this paper is an iterative approach for incorporating evidence of users in the probabilistically integrated data. Evidence can be specified as hard or soft rules (i.e., rules that are uncertain themselves).

This paper presents a scalable digital twin solution designed to improve drilling equipment reliability amid increasingly demanding operational conditions. Built on three core pillars: data integration, advanced analytics, and human-centered design; the system combines real-time equipment monitoring with predictive models to quantify asset health and proactively address failure risks. A robust data pipeline processes over 200 GB of data daily, supporting Artificial Intelligence (AI)- and physics-based models that forecast the health of more than 7,200 components under varying geologic and operating conditions. Deployed across more than 100 rigs in 18 months, the solution reduced equipment-related downtime by more than 75%, failure-related credits by 95%, and normalized operating costs by 38%. Rigs using the system demonstrated a 13% year-over-year improvement in drilling performance and a 28% gain versus peers. Developed on a low-code, functional block programming platform, the system achieved commercial deployment within three months. This approach accelerated development, enabled rapid user feedback integration, and demonstrated a practical pathway for large-scale GenAI-enabled reliability solutions in drilling operations.

Weighted kernel Fisher discriminant analysis for integrating heterogeneous data

The basic precondition for effective management and protection of a forest is a concept built on modern methods of collection, processing, analysis and publication of spatial data about forest coverage, as well as its health status. This paper provides a structural concept for integration of heterogeneous data in the support of the forest protection with implementation of the latest methods of remote sensing data collection. The main principles of the structural concept of Forest protection management system (FPMS) result from the assessment of the current and new available data sources (represented by the Diagram of data sources), data analysis and development of innovative mathematical techniques of image processing (represented by the Diagram of methods and tools). Mind maps of the proposed diagrams were created in the free mind mapping application FreeMind. The structural concept is represented by an analytical model of the co-operation of data sources, tools and applications. The dynamic structure was proposed using Unified Modelling Language (UML). The Diagram of the use cases is represented by a Use Case Diagram in UML. The Diagram of the processes, which describes the main processes realized within forest protection management, is represented by an Activity Diagram in UML. The UML diagrams were created in open-source software StarUML. The resulted structural concept of FPMS is the basis of a predictive model improvement and a web application development for the forest protection from the bark beetle.

The mixed reality conference system proposed in this paper is a robust, real-time video conference application software that makes up for the simple interaction and lack of immersion and realism of traditional video conference, which realizes the entire process of holographic video conference from client to cloud to the client. This paper mainly focuses on designing and implementing a video conference system based on AI segmentation technology and mixed reality. Several mixed reality conference system components are discussed, including data collection, data transmission, processing, and mixed reality presentation. The data layer is mainly used for data collection, integration, and video and audio codecs. The network layer uses Web-RTC to realize peer-to-peer data communication. The data processing layer is the core part of the system, mainly for human video matting and human-computer interaction, which is the key to realizing mixed reality conferences and improving the interactive experience. The presentation layer explicitly includes the login interface of the mixed reality conference system, the presentation of real-time matting of human subjects, and the presentation objects. With the mixed reality conference system, conference participants in different places can see each other in real-time in their mixed reality scene and share presentation content and 3D models based on mixed reality technology to have a more interactive and immersive experience.