Big Data Integration

The volume of data in the world is growing very fast and generated from verity of sources like social media, sensors airline industry or scientific data in different formats. Biggest challenge is how to infer meaningful insights from such a varietyful and big data along with concern of data storage and management of fast growing data. The size of the databases used in today’s enterprises has been growing at exponential rates day by day. Hence, industries requirement to quickly process and analyze the big data volumes for business decision making and customer insights has also grown exponentially. Data pouring from various sources may be can be structured or unstructured in nature. Structured data refers to a relatively well-organized information, which can be further inserted into traditional RDBMS. As Traditional RDBMS are efficient and easy queries by simple, straightforward search algorithms or SQL queries. In contrast to structured data, unstructured data can be considered as information, which does not, comes in a pre-defined data format, well organized data storage model, or cannot be stored well into relational tables. It is assumed to be fastest growing type of data, e.g. image, sensors data, web chats, social networking messaging data, video, documents, log files, and email data. There are many techniques and software available, which can process and provide efficient storage of unstructured data and help organization to perform analytics on unstructured data. Unstructured data does not well-organized and not stored in predefined manner e.g. logs, web chats. The variety and on ordered nature of data makes storage methods and structure makes execution a time and resource-consuming affair. Advancement into technology has open floodgates to push huge volume of unstructured type of data. Multimedia data is one of the example of unstructured big data, which spans all over the Internet. This needs high execution capability to extract useful information. Rapid processing of multimedia data such as video is important for e.g. criminal investigations, surveillance monitoring, news analysis, sports analytics domain, emotion extraction, etc. Hence, analysis of multimedia data in minimum timeframe is one of the latest research areas. Therefore, we have researched techniques for analyzing unstructured data to extract meaningful information hidden in the big data. In addition, we will describe about various techniques and software used to Manage, process unstructured big data in efficient manner, and increases the performance of complexity analysis.

This study aims to develop high-performing Machine Learning and Deep Learning models in predicting hospital length of stay (LOS) while enhancing interpretability. We compare performance and interpretability of models trained only on structured tabular data with models trained only on unstructured clinical text data, and on mixed data. The structured data was used to train fourteen classical Machine Learning models including advanced ensemble trees, neural networks and k-nearest neighbors. The unstructured data was used to fine-tune a pre-trained Bio Clinical BERT Transformer Deep Learning model. The structured and unstructured data were then merged into a tabular dataset after vectorization of the clinical text and a dimensional reduction through Latent Dirichlet Allocation. The study used the free and publicly available Medical Information Mart for Intensive Care (MIMIC) III database, on the open AutoML Library AutoGluon. Performance is evaluated with respect to two types of random classifiers, used as baselines. The best model from structured data demonstrates high performance (ROC AUC = 0.944, PRC AUC = 0.655) with limited interpretability, where the most important predictors of prolonged LOS are the level of blood urea nitrogen and of platelets. The Transformer model displays a good but lower performance (ROC AUC = 0.842, PRC AUC = 0.375) with a richer array of interpretability by providing more specific in-hospital factors including procedures, conditions, and medical history. The best model trained on mixed data satisfies both a high level of performance (ROC AUC = 0.963, PRC AUC = 0.746) and a much larger scope in interpretability including pathologies of the intestine, the colon, and the blood; infectious diseases, respiratory problems, procedures involving sedation and intubation, and vascular surgery. Our results outperform most of the state-of-the-art models in LOS prediction both in terms of performance and of interpretability. Data fusion between structured and unstructured text data may significantly improve performance and interpretability.

In the digital era, the rapid growth of social media and online platforms has led to an explosion of unstructured textual data that holds significant business value. Traditional marketing strategies, once reliant on structured data such as demographics and purchase history, now benefit from insights derived from text analytics and sentiment analysis. This paper explores the integration of structured and unstructured data to strengthen marketing intelligence and customer segmentation. By utilizing text mining techniques and Natural Language Processing (NLP), unstructured data such as customer reviews and comments can be analyzed to extract sentiments, identify emerging trends, and refine customer relationship strategies. The study proposes an integrated framework that combines data extraction, transformation, and loading (ETL) processes with a data warehouse system for unified analysis. Using clustering algorithms such as K-Means and visualization tools, insights into customer behavior, preferences, and market segmentation are revealed. The paper also discusses the challenges of handling multilingual and context-dependent text, ethical and privacy considerations, and the technical architecture necessary for business intelligence implementation. Findings suggest that effective integration of textual analytics with structured data can lead to more informed decision-making, improved marketing strategies, and stronger customer engagement.

Construct a knowledge graph is time-consuming and the knowledge graph in the scientific domain requires extremely high labor costs due to it requires high prior knowledge to extract knowledge from resources. To build a scientific research knowledge graph, the most of input are papers, patent, the description of their project and some national program (such as National High Technology Research and Development Program of China, Major State Basic Research Development Program of China, General Program, Key Program and Major Program) which all of them are unstructured data, that make human participation are mostly necessary to measure the quality. In this paper, we design and proposed a framework using active learning; this framework can be used to extract entity and relation from unstructured science and technology research data. This framework combines the human and machine learning approach together, which is active learning, to help user extract entity from those unstructured data with less time cost. By using those data to construct a CKG as annotation label, it further implements active learning tools and helps the expert to rapidly annotate the data with high accuracy. Those knowledge graph constructed by this framework can be used to finding similar research area, finding similar researchers, finding popular research areas and so on.

In Intensive Care Unit (ICU), it is important to anticipate interventions for patients at a high-risk of death. This requires identifying those patients ideally at the time of their admission in ICU, and update their initial risk rate every time new data is available. This predictive task can be performed by analyzing structured and unstructured routine data to make sure that we can initiate a prediction for every patient. Traditional statistical tools have been used to assess disease like pneumonia and predict the outcome of a patient. Recently, machine learning models emerged and have shown better performances on such tasks. Although authors have published various results, their works rely on a single datatype, either structured or unstructured data. Using the Medical Information Mart for intensive Care data-set, we are proposing an ensemble model, that aggregates different data-types to predict the outcome of a pneumonia patient admitted in ICU using limited data that can be available at the very early stage of his stay. To demonstrate the importance of this approach, we compared it with 2 different models, one based only on structured data, and another one based on narratives text from caregivers, and we were able to show that our ensemble model can perform way better with an accuracy of 0.98 of F1-score(0.97 MCC), while a model using only structured data had 0.79 of F1-score and where text notes predicted the outcome with an accuracy of 0.89 of Matthews Correlation Coefficient. In addition to showing how ensemble learning models can outperform other models on this task, we demonstrated the importance and usefulness of interpreting the predictions pointing out the leading factors that are determining the global and individual outcome predictions.

Recently, various detection approaches that identify anomalous events (e.g., discoloration, contamination) by analyzing data collected from smart meters (so-called structured data) have been developed for many water distribution systems (WDSs). However, although some of them have showed promising results, meters often fail to collect/transmit the data (i.e., missing data) thus meaning that these methods may frequently not work for anomaly identification. Thus, the clear next step is to combine structured data with another type of data, unstructured data, that has no structural format (e.g., textual content, images, and colors) and can often be expressed through various social media platforms. However, no previous work has been carried out in this regard. This study proposes a framework that combines structured and unstructured data to identify WDS water quality events by collecting turbidity data (structured data) and text data uploaded to social networking services (SNSs) (unstructured data). In the proposed framework, water quality events are identified by applying data-driven detection tools for the structured data and cosine similarity for the unstructured data. The results indicate that structured data-driven tools successfully detect accidents with large magnitudes but fail to detect small failures. When the proposed framework is used, those undetected accidents are successfully identified. Thus, combining structured and unstructured data is necessary to maximize WDS water quality event detection.

To examine the value of unstructured electronic health record (EHR) data (free-text notes) in identifying a set of geriatric syndromes. Retrospective analysis of unstructured EHR notes using a natural language processing (NLP) algorithm. Large multispecialty group. Older adults (N=18,341; average age 75.9, 58.9% female). We compared the number of geriatric syndrome cases identified using structured claims and structured and unstructured EHR data. We also calculated these rates using a population-level claims database as a reference and identified comparable epidemiological rates in peer-reviewed literature as a benchmark. Using insurance claims data resulted in a geriatric syndrome prevalence ranging from 0.03% for lack of social support to 8.3% for walking difficulty. Using structured EHR data resulted in similar prevalence rates, ranging from 0.03% for malnutrition to 7.85% for walking difficulty. Incorporating unstructured EHR notes, enabled by applying the NLP algorithm, identified considerably higher rates of geriatric syndromes: absence of fecal control (2.1%, 2.3 times as much as structured claims and EHR data combined), decubitus ulcer (1.4%, 1.7 times as much), dementia (6.7%, 1.5 times as much), falls (23.6%, 3.2 times as much), malnutrition (2.5%, 18.0 times as much), lack of social support (29.8%, 455.9 times as much), urinary retention (4.2%, 3.9 times as much), vision impairment (6.2%, 7.4 times as much), weight loss (19.2%, 2.9 as much), and walking difficulty (36.34%, 3.4 as much). The geriatric syndrome rates extracted from structured data were substantially lower than published epidemiological rates, although adding the NLP results considerably closed this gap. Claims and structured EHR data give an incomplete picture of burden related to geriatric syndromes. Geriatric syndromes are likely to be missed if unstructured data are not analyzed. Pragmatic NLP algorithms can assist with identifying individuals at high risk of experiencing geriatric syndromes and improving coordination of care for older adults.

Knowledge enhancement-based large-scale natural language training model is an advanced language model that combines deep learning and knowledge enhancement. By learning from massive unlabeled data and combining with external knowledge such as knowledge graph, it breaks through the limitations of traditional models in interpretability and reasoning ability. Introducing knowledge into data-driven artificial intelligence model is an important way to realize human-machine hybrid intelligence. However, since most pre-trained models are trained on large-scale unstructured corpus data, the defects in certainty and explainability can be remedied to some extent by introducing external knowledge. To solve the above problems, we present a knowledge-enhanced large-scale natural language training model that integrates deep learning with external knowledge sources (e.g., knowledge graphs) to improve interpretability and reasoning ability. This approach addresses the limitations of traditional models trained on unstructured data by incorporating external knowledge to enhance certainty and explainability. We propose a new knowledge enhancement method and demonstrate its effectiveness through a long text representation model. This model processes structured, knowledge-rich long texts by extracting and integrating knowledge and semantic information at the sentence and document levels. It then fuses these representations to generate an enhanced long text representation. Experiments on legal case matching tasks show that our model significantly outperforms existing methods, highlighting its innovation and practical value.

1.4 - Demographics of Corporate Data

With the meteoric development of the big data, internet of things (IoT), the very first question is about how to scout the massive volume of data from heterogeneous data sources. The modern tools and techniques commonly used in successful transmuting of well-structured data into business intelligence (BI) clearly don't work when started on unlabeled data. It is needed to have effective and efficient mapping processes to transform unstructured text data to structured text data with assigned categories to make rapid decisions. With the support of modern automatic text mining tools, the decision-making process can now be done efficiently and cost-effectively. The deep learning concept has attained state-of-the-art results. The mapping operation of unstructured text data to classified structured text data will symbolize unstructured data as renewable assets that are well arranged, useful, and meaningful to serve organizational operations. This chapter presented a comparison of conventional and deep learning methods for unstructured text data classification and its challenges.

We present an online multilingual system for event detection and comprehension from media feeds. The system retrieves information from news sites, aggregates them into events (event detection), and summarizes them by extracting semantic labels of its most relevant entities (event representation) in order to answer the journalism Ws: who, what, when and where. The generated events populate VLX-Stories -an event ontology- transforming unstructured text data to a structured knowledge base representation. Our system exploits an external entity Knowledge Graph (VKG) to help populate VLX-Stories. At the same time, this external knowledge graph can also be extended with a Dynamic Entity Linking (DEL) module, which detects emerging entities (EE) on unstructured data. The system is currently deployed in production and used by media producers in the editorial process, providing real-time access to breaking news. Each month, VLX-Stories detects over 9000 events from over 4000 news feeds from seven different countries and in three different languages. At the same time, it detects over 1300 EE per month, which populate VKG.

In big data epoch, one of the major challenges is the large volume of mixed structured and unstructured data, which comes in heterogeneous sources. Because of different form, structured and unstructured data are often considered apart from each other. However, they may speak about the same entities of the world. If a query involve both structured data and its unstructured counterpart, it is inefficient to execute it separately. The paper presents a novel index structure tailored towards the combinations of structured and unstructured data. The combined index is a joint index over structured database and unstructured document, based on entity co-occurrences. It is also a semantic index which describes the semantic relationships between entities and their multiple resources. We store the index as RDF graphs and queries are SPARQL-like. Experiments show that the associated index can not only provide apposite information but also execute queries efficiently.

The primary goal of this invention is to revolutionize production engineering by addressing the challenges of integrating unstructured intervention reports with structured production data. Existing systems are unable to provide a seamless integration that enables comprehensive analysis and decision-making. This solution leverages Generative AI and an end-to-end automated pipeline to transform data handling, optimize intervention planning, and provide real-time insights, ultimately enhancing the efficiency and intelligence of production engineers. The invention employs several interconnected modules. First, the **Data Ingestion Module** collects unstructured intervention reports and structured production data from various sources, including databases and cloud platforms. The **Unstructured Data Handling Module** uses domain-driven Generative AI to extract key performance indicators (KPIs) from intervention reports, converting them into structured data formats. Next, the **Integration Module** unifies structured and unstructured data, synchronizing it in real-time. The **Knowledge Enrichment Module** creates a high dimensional knowledge base using Retrieval Augmented Generation (RAG), allowing for interaction with thousands of reports. The **Visualization Module** provides interactive dashboards for users to explore results, while the **Intelligent Assistant Module** enables engineers to query data in real-time, enhancing decision-making capabilities. The integration of structured and unstructured data allows production engineers to assess intervention impacts comprehensively. By automating the extraction of insights from reports and merging them with production time-series data, the solution provides a holistic view of interventions. The ability to interact with large datasets in real-time through an intelligent assistant enables faster and more informed decisions. Case studies have demonstrated that this approach can identify trends in intervention outcomes, optimize future interventions, and reduce missed opportunities due to incomplete analysis. In one instance, it prevented premature shut-off of oil-bearing zones by analyzing water shut-off (WSO) interventions and recommending an alternative, more effective intervention. This work has been implemented on 1000-page scanned documents combined with daily production rate for more than 20 wells for a field. This solution represents a significant leap in production engineering data. analysis by combining the power of Generative AI with real-time data integration. Its ability to seamlessly process both unstructured and structured data is unmatched in the industry. Unlike existing solutions that rely on separate analysis or manual data handling, this system automates the entire process. The use of Retrieval-Augmented Generation (RAG) technology for enriched knowledge interaction and a real-time visualization platform provides a deeper and more scalable understanding of intervention impacts, offering engineers a powerful tool for optimizing production performance.

Objective: Through the practice discussion and exchange of the integration of big data and pharmaceutical standards, to provide experience reference for the practice of information technology in the management of rational drug use. Methods: The application of data information in the practice of hospital pharmacy management is demonstrated through the introduction of practical cases and data achievements at different levels. Results: The integration of medical big data and pharmaceutical standards effectively improved the hospital's quantitative quality management indicators and the informatization of many pharmaceutical management methods in the hospital is realized. Conclusions: The integration of medical big data and pharmaceutical standards can effectively help improve the management level of rational drug use in hospitals, and there are still many possibilities for future exploration.

In the last two decades, unstructured information has become a major challenge in information management. Such challenge is caused by the massive and increasing amount of information resulting from the conversion of almost all daily tasks into digital format. Tools and applications are necessary in organizing unstructured information, which can be found in structured data, such as in relational database management systems (RDBMS). RDBMS has robust and powerful structures for managing, organizing, and retrieving data. However, structured data still contains unstructured information. In this paper, the methods used for managing unstructured data in RDBMS are investigated. In addition, an incremental and dynamic repository for managing unstructured data in relational databases are introduced. The proposed technique organizes unstructured information through linkages among textual data based on semantics. Furthermore, it provides users with a good picture of the unstructured information. The proposed technique can rapidly and easily obtain useful data, and thus, it can be applied in numerous domains, particularly those who deal with textual data, such as news articles.KeywordsRelational databasesUnstructured dataDocument clusteringQuery efficiencyTextual data