Scaling up knowledge graph creation to large and heterogeneous data sources

RDF Knowledge graphs are semantic networks to infer information from data based on RDF triples using SPARQL queries. Querying RDF Knowledge graphs using visualization allows different views of the graph and provides more insights. A new visualization tool, KG-Visual is developed in this paper using Python dashboard, JavaScript and RDF libraries. This tool tackles the limitations of existing tools by providing a customizable and reliable visualization for RDF knowledge graphs. In addition to the essential functionalities such as search, drag and drop nodes, zoom-in and zoom-out over a knowledge graph, the tool holds the capability to run SPARQL queries and visualize the results on the knowledge graph, which makes this open-source tool a potential candidate amongst most RDF visualization tools.

In the last years, enterprises increase their effort to collect large amounts of data from many heterogeneous data sources and store it in modern architectures like data lakes. However, this approach faces different drawbacks for finding and understanding data sources. Ontology-Based Data Access (OBDA) originating from the Semantic Web enables a homogeneous access to the data sources by using a mapping, called semantic model, between a data source and a target ontology. However, OBDA requires a detailed ontology, which is usually created by ontology engineers and domain experts resulting along with high effort for designing and maintaining. To overcome these limitations, we develop an approach consisting of a knowledge graph, which features an internal growing ontology and linked data-source specific semantic models. The ontology continuously evolves on-demand based on newly added data sources along with their corresponding semantic models, which are created by domain experts. To ensure the knowledge graph's stability, we develop an intuitive user-oriented assistant and combine it with a semi-supervised evolving strategy that assists the user with the help of external knowledge bases. We evaluate accuracy and usability of our approach by conducting a user study with a heterogeneous group of participants that define semantic models upon pre-defined data sets. The results show that semantic models become more objective and consistent with our provided user assistant and thus lead to a knowledge graph with higher interconnectivity and stability.

Entity alignment is the problem of identifying which entities in a data source refer to the same real-world entity in the others. Identifying entities across heterogeneous data sources is paramount to many research fields, such as data cleaning, data integration, information retrieval and machine learning. The aligning process is not only overwhelmingly expensive for large data sources since it involves all tuples from two or more data sources, but also need to handle heterogeneous entity attributes. In this paper, we propose an unsupervised approach, called EnAli, to match entities across two or more heterogeneous data sources. EnAli employs a generative probabilistic model to incorporate the heterogeneous entity attributes via employing exponential family, handle missing values, and also utilize the locality sensitive hashing schema to reduce the candidate tuples and speed up the aligning process. EnAli is highly accurate and efficient even without any ground-truth tuples. We illustrate the performance of EnAli on re-identifying entities from the same data source, as well as aligning entities across three real data sources. Our experimental results manifest that our proposed approach outperforms the comparable baseline.

SummaryRDF knowledge graphs (KG) usually contain billions of labeled entities, and how to obtain the desired results efficiently on RDF KG for given SPARQL queries have attracted increasing attentions recently. However, it is difficult for users to write a complex SPARQL query without full knowledge of the underlying KG schema due to the “schema‐free” feature of RDF KG. In this paper, we study the problem of acquiring semantic approximate results for a simple SPARQL query instead of the complex expression. The basic idea behind is to use the real knowledge semantics to extend the query intention of given simplified SPARQL query, and then a semantic based greedy search over is designed to return top‐k similar results. To obtain the knowledge semantics efficiently, we define type similarity to reorganize the original KG as a corpus with semantic locality and then a context aware text embedding model is adopted to achieve the semantic vectors of existing knowledge. Afterwards, an approximate query method over RDF KG is designed to obtain top‐k similar results based on the knowledge semantics above. Extensive experiments over DBpedia dataset and QALD‐4 benchmark confirm our solution's superiority on both effectiveness and efficiency.

Knowledge graphs are often constructed from heterogeneous data sources, using declarative rules that map them to a target ontology and materializing them into RDF. When these data sources are large, the materialization of the entire knowledge graph may be computationally expensive and not suitable for those cases where a rapid materialization is required. In this work, we propose an approach to overcome this limitation, based on the novel concept of mapping partitions. Mapping partitions are defined as groups of mapping rules that generate disjoint subsets of the knowledge graph. Each of these groups can be processed separately, reducing the total amount of memory and execution time required by the materialization process. We have included this optimization in our materialization engine Morph-KGC, and we have evaluated it over three different benchmarks. Our experimental results show that, compared with state-of-the-art techniques, the use of mapping partitions in Morph-KGC presents the following advantages: (i) it decreases significantly the time required for materialization, (ii) it reduces the maximum peak of memory used, and (iii) it scales to data sizes that other engines are not capable of processing currently.

Recently there has been a rapid increase in the number of data sources and data services, such as cloud-based data markets and data portals, that facilitate the collection, publishing and trading of data. Data sources typically exhibit large heterogeneity in the type and quality of data they provide. Unfortunately, when the number of data sources is large, it is difficult for users to reason about the actual usefulness of sources for their applications and the trade-offs between the benefits and costs of acquiring and integrating sources. In this demonstration we present \textsc{SourceSight}, a system that allows users to interactively explore a large number of heterogeneous data sources, and discover valuable sets of sources for diverse integration tasks. \textsc{SourceSight}~uses a novel multi-level source quality index that enables effective source selection at different granularity levels, and introduces a collection of new techniques to discover and evaluate relevant sources for integration.

Big data analytics (BDA) applications use machine learning algorithms to extract valuable insights from large, fast, and heterogeneous data sources. New software engineering challenges for BDA applications include ensuring performance levels of data-driven algorithms even in the presence of large data volume, velocity, and variety (3Vs). BDA software complexity frequently leads to delayed deployments, longer development cycles, and challenging performance assessment. This paper proposes a Domain-Specific Model (DSM), and DevOps practices to design, deploy, and monitor performance metrics in BDA applications. Our proposal includes a design process, and a framework to define architectural inputs, software components, and deployment strategies through integrated high-level abstractions to enable QS monitoring. We evaluate our approach with four use cases from different domains to demonstrate a high level of generalization. Our results show a shorter deployment and monitoring times, and a higher gain factor per iteration compared to similar approaches.

There are many problems facing scientists in their efforts to use computers to manage, access, and analyze data. One serious problem is the heterogeneity of data sources. Previous research into accessing heterogeneous data has not addressed problems with accessing large data sources that are not managed by database management systems. We believe that an object-oriented database management system can be used to implement an object-oriented heterogeneous database (OOHDB) that efficiently addresses the problem of heterogeneity among large scientific property databases. The object-oriented data model provides a powerful modeling paradigm for representing complex scientific data. Objects within the OOHDB hide heterogeneity among various data sources. Efficient access to external data is achieved by representing external data with a small amount of data stored within the OOHDB.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

A large number of datasets are being made available on the Web using a variety of formats and according to diverse data models. Ontology Based Data Integration (OBDI) has been traditionally proposed as a mechanism to facilitate access to such heterogeneous datasets, providing a unified view over their data by means of ontologies. Recently, the term “Virtual Knowledge Graph Access” has begun to be used to refer to the mechanisms that provide query-based access to knowledge graphs virtually generated from heterogeneous data sources. Several OBDI engines exist in the state of the art, with overlapping capabilities but also clear differences among them (in terms of the data formats that they can deal with, mapping languages that they support, query expressivity that they allow, etc.). These engines have been evaluated with different testbeds and benchmarks. However, their heterogeneity has made it difficult to come up with a common comprehensive benchmark that allows for comparisons among them to facilitate their selection by practitioners, and more importantly, for their continuous improvement by the teams that maintain them. In this paper we present GTFS-Madrid-Bench, a benchmark to evaluate OBDI engines that can be used for the provision of access mechanisms to virtual knowledge graphs. Our proposal introduces several scenarios that aim at measuring the query capabilities, performance and scalability of all these engines, considering their heterogeneity. The data sources used in our benchmark are derived from the GTFS data files of the subway network of Madrid. They have been transformed into several formats (CSV, JSON, SQL and XML) and scaled up. The query set aims at addressing a representative number of SPARQL 1.1 features while covering usual queries that data consumers may be interested in.

PurposeAttention Deficit Hyperactivity Disorder (ADHD) is a widespread condition that affects human behaviour and can interfere with daily activities and relationships. Medication or medical information about ADHD can be found in several data sources on the Web. Such distribution of knowledge raises notable obstacles since researchers and clinicians must manually combine various sources to deeply explore aspects of ADHD. Knowledge graphs have been widely used in medical applications due to their data integration capabilities, offering rich data stores of information built from heterogeneous sources; however, general purpose knowledge graphs cannot represent knowledge in sufficient detail, thus there is an increasing interest in domain-specific knowledge graphs.MethodsIn this work we propose a Knowledge Graph of ADHD. In particular, we introduce an automated procedure enabling the construction of a knowledge graph that covers knowledge from a wide range of data sources primarily focusing on adult ADHD. These include relevant literature and clinical trials, prescribed medication and their known side-effects. Data integration between these data sources is accomplished by employing a suite of information linking procedures, which aim to connect resources by relating them to common concepts found in medical thesauri.ResultsThe usability and appropriateness of the developed knowledge graph is evaluated through a series of use cases that illustrate its ability to enhance and accelerate information retrieval.ConclusionThe Knowledge Graph of ADHD can provide valuable assistance to researchers and clinicians in the research, training, diagnostic and treatment processes for ADHD.

• Offers a new vision on automated data harmonization in Dataspace (DS) systems. • Introduces LLM-based methods for scalable DS ingestion of heterogeneous datasources. • Presents a system with Harmonizer, Transformer, Evaluator components for ingestion. • Demonstrated an automated data ingestion prototype using LLM agents. • Validates the system with healthcare use case harmonizing heterogeneous data sources. Dataspaces (DS) enable stakeholders to collaborate on innovative, data-driven services by integrating data across domains. However, the realization and adoption of DS remain challenging due to domain-specific heterogeneity at the system, service, and data levels. While system and service-level heterogeneity can often be addressed through standards, data-level heterogeneity, namely data structures and semantics variations, remains challenging. To effectively ingest data into the DS, two communication endpoints must correctly interpret each other’s data models, therefore, DS ecosystems rely on “harmonization”, the process of generating a unified target data model from heterogeneous sources and transforming incoming data accordingly. Currently, harmonization and transformation are performed manually whenever new data sources are integrated. This is time-consuming, costly, and difficult to scale, posing a critical barrier to the realization and adoption of DS in practice. This study proposes a novel methodology for automated data harmonization during ingestion into DS ecosystems. The approach integrates harmonization, transformation, and human-in-the-loop evaluation within an automated system powered by modern LLM-based AI agents. These agents address data-level heterogeneity and generate harmonized target data models, representing a substantial departure from current manually-handled data harmonization. The system is validated through a healthcare use case, demonstrating its practical feasibility for harmonization during data ingestion into the DS. Overall, this work provides a foundational step toward seamless, efficient, and scalable data integration in DS. By automating data harmonization, it delivers substantial value to industry digital solutions as well as domains where data heterogeneity persists, including IoT or Big Data platforms.

Stakeholders in Open Source Software (OSS) projects need to determine whether a project is likely to sustain for a sufficient period of time in order to justify their investments into this project. In an OSS project context, there are typically several data sources and OSS processes relevant for determining project health indicators. However, even within one project these data sources often are technically and/or semantically heterogeneous, which makes data collection and analysis tedious and error prone. In this paper, the authors propose and evaluate a framework for OSS data analysis (FOSSDA), which enables the efficient collection, integration, and analysis of data from heterogeneous sources. Major results of the empirical studies are: (a) the framework is useful for integrating data from heterogeneous data sources effectively and (b) project health indicators based on integrated data analyses were found to be more accurate than analyses based on individual non-integrated data sources.

The analysis of modern production environments requires a clear and understandable uniform visualization of the information from all levels of the production. This visualization has to be on the one hand light weight to avoid cognitive overload of the users and on the other hand rich enough to provide information about all production information sources. To solve this issue we propose to use a unified component-based visualization where single items can be exchanged to fulfill the informational needs of the users from heterogeneous data sources.

ObjectiveBiomedical research increasingly relies on the integration of information from multiple heterogeneous data sources. Despite the fact that structural and terminological aspects of interoperability are interdependent and rely on a common set of requirements, current efforts typically address them in isolation. We propose a unified ontology-based knowledge framework to facilitate interoperability between heterogeneous sources, and investigate if using the LexEVS terminology server is a viable implementation method.Materials and methodsWe developed a framework based on an ontology, the general information model (GIM), to unify structural models and terminologies, together with relevant mapping sets. This allowed a uniform access to these resources within LexEVS to facilitate interoperability by various components and data sources from implementing architectures.ResultsOur unified framework has been tested in the context of the EU Framework Program 7 TRANSFoRm project, where it was used to achieve data integration in a retrospective diabetes cohort study. The GIM was successfully instantiated in TRANSFoRm as the clinical data integration model, and necessary mappings were created to support effective information retrieval for software tools in the project.ConclusionsWe present a novel, unifying approach to address interoperability challenges in heterogeneous data sources, by representing structural and semantic models in one framework. Systems using this architecture can rely solely on the GIM that abstracts over both the structure and coding. Information models, terminologies and mappings are all stored in LexEVS and can be accessed in a uniform manner (implementing the HL7 CTS2 service functional model). The system is flexible and should reduce the effort needed from data sources personnel for implementing and managing the integration.

As technological development gains more momentum and competition in the global economy intensifies, organizations have resorted to data-driven decision-making (DDDM) and artificial intelligence (AI) as a way of strengthening the strategy management and ensuring the sustainability of a business growth. The strategic management based on the data improves the capacity of the organizations to mince actionable information out of the large heterogeneous and live time data sources so that these organizations can make the better forecast and effective operation and competitive advantage. At the same time, the artificial intelligence technologies, including machine learning, predictive analytics, natural language processing and intelligent automation, are changing the customary frameworks of decision-making and enhancing the abilities of all businesses to think strategically, distributing resources more efficiently, and modifying their business models. This paper will present an argument about the combination of DDDM and AI in strategic management and the possibility of the two issues to cooperate to ensure higher sustainability, value creation in the long-term, minimized risks, and organizational resilience. It deals with a more contemporary literature, and covers the state-of-the-art technique of analysis and relies as well as offers a conceptual side of the ways AI-based decision-systems may assist in strategic planning, performance measurements, stakeholder management and environmental, social and governance (ESG) activities. Besides it, the paper discusses real-life use in the industries as well as outlines the barriers to the adoption and also offers recommendations on the further development of the data-centric and AI-enabled strategic ecosystems. The findings spur the notion that these types of organizations that are open to data intelligence and AI-driven decision architecture are more likely to expand sustainably, at scale, and be future-craft in an extremely dynamic digital landscape.