EnAli: entity alignment across multiple heterogeneous data sources

ObjectiveBiomedical research increasingly relies on the integration of information from multiple heterogeneous data sources. Despite the fact that structural and terminological aspects of interoperability are interdependent and rely on a common set of requirements, current efforts typically address them in isolation. We propose a unified ontology-based knowledge framework to facilitate interoperability between heterogeneous sources, and investigate if using the LexEVS terminology server is a viable implementation method.Materials and methodsWe developed a framework based on an ontology, the general information model (GIM), to unify structural models and terminologies, together with relevant mapping sets. This allowed a uniform access to these resources within LexEVS to facilitate interoperability by various components and data sources from implementing architectures.ResultsOur unified framework has been tested in the context of the EU Framework Program 7 TRANSFoRm project, where it was used to achieve data integration in a retrospective diabetes cohort study. The GIM was successfully instantiated in TRANSFoRm as the clinical data integration model, and necessary mappings were created to support effective information retrieval for software tools in the project.ConclusionsWe present a novel, unifying approach to address interoperability challenges in heterogeneous data sources, by representing structural and semantic models in one framework. Systems using this architecture can rely solely on the GIM that abstracts over both the structure and coding. Information models, terminologies and mappings are all stored in LexEVS and can be accessed in a uniform manner (implementing the HL7 CTS2 service functional model). The system is flexible and should reduce the effort needed from data sources personnel for implementing and managing the integration.

Selection of genes that are differentially expressed and critical to a particular biological process has been a major challenge in post-array analysis. Recent development in bioinformatics has made various data sources available such as mRNA and miRNA expression profiles, biological pathway and gene annotation, etc. Efficient and effective integration of multiple data sources helps enrich our knowledge about the involved samples and genes for selecting genes bearing significant biological relevance. In this work, we studied a novel problem of multi-source gene selection: given multiple heterogeneous data sources (or data sets), select genes from expression profiles by integrating information from various data sources. We investigated how to effectively employ information contained in multiple data sources to extract an intrinsic global geometric pattern and use it in covariance analysis for gene selection. We designed and conducted experiments to systematically compare the proposed approach with representative methods in terms of statistical and biological significance, and showed the efficacy and potential of the proposed approach with promising findings.

Graph model are increasingly popular among modern applications for its ability to model complex relationships between entities. Users tend to query the data as a graph with graph operations (e.g., graph navigation and exploration). However, a large fraction of the data resides in relational databases or other storage systems. Challenges arise in uniformly querying multiple heterogeneous data sources as a graph. Traditional solutions are limited by time-consuming data integration, expensive development effort, and incomplete query requirements. Thus, we developed Lynx, a general graph query framework, to simplify querying graph data by converting complex statements into basic graph operations. Instead of connecting directly to the data sources, Lynx retrieves data through user-implemented interfaces for those graph operations. We demonstrate Lynx's capabilities through real-world scenarios, showcasing Lynx's ability to process graph queries on multiple heterogeneous data sources and also to be used as a generic graph query engine development framework.

Relatively mature integration technologies are available that make application integration tractable from an engineering perspective. However, integrating multiple heterogeneous data sources into applications is still a time-consuming, costly and error-prone engineering task, because existing integration technologies are based on static integration architectures. Such architectures tightly couple a data source to the data integration infrastructure. This paper describes the architecture and evolving prototype implementation of the Data Concierge, which provides a dynamic solution to integrate heterogeneous data sources. The core architecture and mechanisms in the Data Concierge can be used for dynamically attaching to a previously unidentified source of information, without the need for a pre-existing adapter component. In this paper, an ontology based API description mechanism supported by the Data Concierge is described, along with the architecture and prototype tools for describing the metadata necessary to facilitate dynamic integration. The use of an ontology and associated state machine to generate API calls to access data sources is illustrated using an FTP data source.

Stakeholders in Open Source Software (OSS) projects need to determine whether a project is likely to sustain for a sufficient period of time in order to justify their investments into this project. In an OSS project context, there are typically several data sources and OSS processes relevant for determining project health indicators. However, even within one project these data sources often are technically and/or semantically heterogeneous, which makes data collection and analysis tedious and error prone. In this paper, the authors propose and evaluate a framework for OSS data analysis (FOSSDA), which enables the efficient collection, integration, and analysis of data from heterogeneous sources. Major results of the empirical studies are: (a) the framework is useful for integrating data from heterogeneous data sources effectively and (b) project health indicators based on integrated data analyses were found to be more accurate than analyses based on individual non-integrated data sources.

BackgroundProtein-protein interaction (PPI) prediction remains a central task in systems biology to achieve a better and holistic understanding of cellular and intracellular processes. Recently, an increasing number of computational methods have shifted from pair-wise prediction to network level prediction. Many of the existing network level methods predict PPIs under the assumption that the training network should be connected. However, this assumption greatly affects the prediction power and limits the application area because the current golden standard PPI networks are usually very sparse and disconnected. Therefore, how to effectively predict PPIs based on a training network that is sparse and disconnected remains a challenge.ResultsIn this work, we developed a novel PPI prediction method based on deep learning neural network and regularized Laplacian kernel. We use a neural network with an autoencoder-like architecture to implicitly simulate the evolutionary processes of a PPI network. Neurons of the output layer correspond to proteins and are labeled with values (1 for interaction and 0 for otherwise) from the adjacency matrix of a sparse disconnected training PPI network. Unlike autoencoder, neurons at the input layer are given all zero input, reflecting an assumption of no a priori knowledge about PPIs, and hidden layers of smaller sizes mimic ancient interactome at different times during evolution. After the training step, an evolved PPI network whose rows are outputs of the neural network can be obtained. We then predict PPIs by applying the regularized Laplacian kernel to the transition matrix that is built upon the evolved PPI network. The results from cross-validation experiments show that the PPI prediction accuracies for yeast data and human data measured as AUC are increased by up to 8.4 and 14.9% respectively, as compared to the baseline. Moreover, the evolved PPI network can also help us leverage complementary information from the disconnected training network and multiple heterogeneous data sources. Tested by the yeast data with six heterogeneous feature kernels, the results show our method can further improve the prediction performance by up to 2%, which is very close to an upper bound that is obtained by an Approximate Bayesian Computation based sampling method.ConclusionsThe proposed evolution deep neural network, coupled with regularized Laplacian kernel, is an effective tool in completing sparse and disconnected PPI networks and in facilitating integration of heterogeneous data sources.

• Offers a new vision on automated data harmonization in Dataspace (DS) systems. • Introduces LLM-based methods for scalable DS ingestion of heterogeneous datasources. • Presents a system with Harmonizer, Transformer, Evaluator components for ingestion. • Demonstrated an automated data ingestion prototype using LLM agents. • Validates the system with healthcare use case harmonizing heterogeneous data sources. Dataspaces (DS) enable stakeholders to collaborate on innovative, data-driven services by integrating data across domains. However, the realization and adoption of DS remain challenging due to domain-specific heterogeneity at the system, service, and data levels. While system and service-level heterogeneity can often be addressed through standards, data-level heterogeneity, namely data structures and semantics variations, remains challenging. To effectively ingest data into the DS, two communication endpoints must correctly interpret each other’s data models, therefore, DS ecosystems rely on “harmonization”, the process of generating a unified target data model from heterogeneous sources and transforming incoming data accordingly. Currently, harmonization and transformation are performed manually whenever new data sources are integrated. This is time-consuming, costly, and difficult to scale, posing a critical barrier to the realization and adoption of DS in practice. This study proposes a novel methodology for automated data harmonization during ingestion into DS ecosystems. The approach integrates harmonization, transformation, and human-in-the-loop evaluation within an automated system powered by modern LLM-based AI agents. These agents address data-level heterogeneity and generate harmonized target data models, representing a substantial departure from current manually-handled data harmonization. The system is validated through a healthcare use case, demonstrating its practical feasibility for harmonization during data ingestion into the DS. Overall, this work provides a foundational step toward seamless, efficient, and scalable data integration in DS. By automating data harmonization, it delivers substantial value to industry digital solutions as well as domains where data heterogeneity persists, including IoT or Big Data platforms.

The increasing availability of numerous heterogeneous but related sources of data arises in many fields, such as bioinformatics, robotics, computer vision, information retrieval, and many others. Multiple sources of data can be viewed as different, related views of the same learning problem, where dependencies between the views could potentially take on complex structures. This gives rise to interesting and challenging machine learning problems where data sources are combined for learning. This framework encompasses several data fusion tasks and related topics, such as transfer learning, multitask learning, multiview learning, and learning under covariate shift. Often the general concept is to augment training data for each problem of interest with data from other partially related learning problems. Several approaches for inferring and exploiting complex relationships between sources have been presented. The advantages of the multiple source learning paradigm can be seen in situations where individual data sources are noisy, incomplete, and learning from more than one source can filter out problem-independent noise. Additionally, another aspect of multiple source learning is that data sources need only contain partially relevant information to the desired learning problem, making it possible to exploit a large number of data sources. This special issue on “Learning from Multiple Sources” was inspired by a successful (eponymous) NIPS workshop which took place in Whistler (Canada) on December 13, 2008. The workshop received twenty-three submissions, from which seven contributed talks and

RDF knowledge graphs (KG) are powerful data structures to represent factual statements created from heterogeneous data sources. KG creation is laborious and demands data management techniques to be executed efficiently. This paper tackles the problem of the automatic generation of KG creation processes declaratively specified; it proposes techniques for planning and transforming heterogeneous data into RDF triples following mapping assertions specified in the RDF Mapping Language (RML). Given a set of mapping assertions, the planner provides an optimized execution plan by partitioning and scheduling the execution of the assertions. First, the planner assesses an optimized number of partitions considering the number of data sources, type of mapping assertions, and the associations between different assertions. After providing a list of partitions and assertions that belong to each partition, the planner determines their execution order. A greedy algorithm is implemented to generate the partitions’ bushy tree execution plan. Bushy tree plans are translated into operating system commands that guide the execution of the partitions of the mapping assertions in the order indicated by the bushy tree. The proposed optimization approach is evaluated over state-of-the-art RML-compliant engines, and existing benchmarks of data sources and RML triples maps. Our experimental results suggest that the performance of the studied engines can be considerably improved, particularly in a complex setting with numerous triples maps and large data sources. As a result, engines that time out in complex cases are enabled to produce at least a portion of the KG applying the planner.

Stimulated by the arrival of the big data era, various and heterogeneous data sources such as data in social networks, mobile devices and sensor data for users have emerged, mirroring characteristics and preferences of data owners. These data sources are often used to construct user profiles so as to facilitate personalized services like recommendations or personalized data access. In the context of second language learning, learner data involve learning logs, standard test results, and individual learning preferences and styles. Given its attribute of reflecting the characteristics of learners, such data can be exploited to build the learner profiles. However, these data sources possibly include noises or bias, and hence influence the reliability of the correspondingly constructed learner profiles. Consequently, the inaccurate profiles may result in ineffective learning tasks that are generated by e-Learning systems. To tackle this issue, it is significant and critical to evaluate the accuracy of learner profiles. In a response to this call, we propose a novel metric named “profile mean square error” to examine the accuracy of learner profiles founded upon diverse sources. We also demonstrate how to construct various learner profiles though applying different data sources such as learning logs, standard test results, and personal learning preferences in e-Learning systems and pedagogical activities. Moreover, we conduct an experimental study among some second language learners, the results of which illustrate that the most accurate profiles are generated from multiple data sources if they are integrated in a rational way.

Purpose of ReviewThe complexity of the data collected for sleep research is increasing, and the focal point of sleep research is dependent on a higher number of data sources. Data collected for sleep studies often includes both subjective and objective measurements of sleep quality and is gathered over a more extended period, e.g., for weeks, months, or even years. However, this variety and volume of data make it challenging and time-consuming for researchers to utilize. Therefore, sophisticated data structures are necessary to utilize data in sleep research.Recent FindingsThis paper explores how heterogeneous data sources can be represented in a homogeneous database design. The following research questions drove our work: (i) How can we represent sleep data from heterogeneous sources in a homogenous digital platform database? and (ii) How can a data source pipeline transform various data sources into a homogeneous data format?SummaryThis paper’s main contributions are conceptualizing the design and development of a homogeneous database and digital platform architecture and a data source pipeline that fits well for sleep research in particular and healthcare research in general.

The analysis of modern production environments requires a clear and understandable uniform visualization of the information from all levels of the production. This visualization has to be on the one hand light weight to avoid cognitive overload of the users and on the other hand rich enough to provide information about all production information sources. To solve this issue we propose to use a unified component-based visualization where single items can be exchanged to fulfill the informational needs of the users from heterogeneous data sources.

Data forms the key success enabler for any business organization. Structured collection and organized data along with this analytical enablement will provide more insights to the organization's existing operations and ways for better improvement. In order to do efficient data collection, integration of data from various sources forms the vital role. Many of the vital business initiatives for an organization requires the data to integrate. Such initiatives performed by creating a customized solution across every data source. In such given architecture, the complexity to integrate data source is proportionate to the increase in data sources. Use of eXtensible Markup Language (XML) for standardizing the information exchange across several sources is an industry standard implementation. It is the technology used to implement and deliver an enterprise-level data integration solution using heterogeneous data sources. The transformation capability of XML is capable of handling very complex and big documents and can do processing of ~thousands of real-time transactions using message bus or Web Services methods. The XML provided solutions are flexible and easily adaptable, thus enabling them to incorporate into a broader range of organizational solutions. This paper discusses in detail, the data integration methods and their challenges and processing of XML data using Extraction, Transformation and Load method. The paper also provides the several ways to parse the XML data as part of the data processing with the case study for an On Line Transaction Process (OLTP) system of a manufacturing company.

Combining in-car multiple sensors measuring parameters that can be used to improve both safety and efficiency with a plethora of external data sources (e.g., traffic conditions, weather) which, if properly used, can significantly improve the overall trip experience. One source that can help the navigation and provide "context awareness", especially for autonomous driving, are the High Definition (HD) maps, which have recently witnessed a tremendous growth of popularity in vehicular technology and use. As they are limited to a particular geographic area with respect to a given point along a trip, different portions need to be downloaded (and processed) on multiple occasions throughout a given trip, along with the other data from internal and external sources. We take a first step towards formalizing the problem of Predicting Map Data Consumption (PMDC) in the future time instants for a given trip, based on a (time) window from its history, and investigate the use of Long Short-Term Memory (LSTM) networks - a special type of Recurrent Neural Networks (RNN). Significant efforts were focused on generating an appropriate dataset for this study, towards which we fused the information available in multiple heterogeneous data sources. We conducted experimental observations demonstrating the benefits of the proposed approach.

Data integration is a critical problem in our increasingly interconnected but inevitably heterogeneous world. There are numerous data sources available in organizational databases and on public information systems like the World Wide Web. Not surprisingly, the sources often use different vocabularies and different data structures, being created, as they are, by different people, at different times, for different purposes. The goal of data integration is to provide programmatic and human users with integrated access to multiple, heterogeneous data sources, giving each user the illusion of a single, homogeneous database designed for his or her specific need. The good news is that, in many cases, the data integration process can be automated. This book is an introduction to the problem of data integration and a rigorous account of one of the leading approaches to solving this problem, viz., the relational logic approach. Relational logic provides a theoretical framework for discussing data integration. Moreover, in many important cases, it provides algorithms for solving the problem in a computationally practical way. In many respects, relational logic does for data integration what relational algebra did for database theory several decades ago. A companion web site provides interactive demonstrations of the algorithms. Table of Contents: Preface / Interactive Edition / Introduction / Basic Concepts / Query Folding / Query Planning / Master Schema Management / Appendix / References / Index / Author Biography