Chapter 18 - Introduction to Big Data Integration

Data integration has been extensively studied by the data management community and is a core task in the data pre-processing step of ML pipelines. When the integrated data is used for analysis and model training, responsible data science requires addressing concerns about data quality and bias. We present a tutorial on data integration and responsibility, highlighting the existing efforts in responsible data integration along with research opportunities and challenges. In this tutorial, we encourage the community to audit data integration tasks with responsibility measures and develop integration techniques that optimize the requirements of responsible data science. We focus on three critical aspects: (1) the requirements to be considered for evaluating and auditing data integration tasks for quality and bias; (2) the data integration tasks that elicit attention to data responsibility measures and methods to satisfy these requirements; and, (3) techniques, tasks, and open problems in data integration that help achieve data responsibility.

Adverse drug reactions (ADRs) remain a major challenge in clinical pharmacology, contributing significantly to patient morbidity and mortality. Traditional methods of ADR detection, such as spontaneous reporting systems (SRS) and clinical trials, often fail to identify ADRs in real-time, limiting their effectiveness. However, the advent of Artificial Intelligence (AI) and Big Data analytics has opened new avenues for improving ADR prediction and prevention. By leveraging vast amounts of clinical, genomic, and pharmacological data, AI models, particularly machine learning (ML) and deep learning, offer powerful tools for predicting ADRs with high accuracy. This review explores the application of AI and Big Data technologies in ADR prediction, detailing the methodologies, data sources, and models used, as well as challenges and opportunities for their future implementation. Furthermore, the integration of real-time data and AI-based algorithms into clinical practice could enhance pharmacovigilance systems, leading to safer drug prescriptions and better patient outcomes.

The Internet of Things (IoT) has recently received considerable interest due to the development of smart technologies in today’s interconnected world. With the rapid advancement in Internet technologies and the proliferation of IoT sensors, myriad systems and applications generate data of a massive volume, variety and velocity which traditional databases and systems are unable to manage effectively. Many organizations need to deal with these massive datasets that encounter different types of data (e.g., IoT streaming data, static data) in different formats (e.g., structured, semi-structured) coming from multiple sources. Several data integration mechanisms have been designed to process mostly static data. Unfortunately, these techniques are not able to deal with and integrate IoT streaming datasets from multiple sources. In this paper, we identify the challenges of IoT Streaming Data Integration (ISDI) and present a formal approach for the real-time integration of such IoT streaming datasets. We address one of the important issues of timing conflict/alignment among streaming data coming from multiple sources. A generic window-based ISDI approach is proposed to deal with IoT data in different formats and algorithms are developed to integrate IoT streaming data from multiple sources. In particular, we extend the basic windowing algorithm for real-time data integration and to deal with the timing alignment issue. We also introduce a de-duplication algorithm to deal with data redundancy and to demonstrate the useful fragments of the integrated data. We conduct several sets of experiments and quantify the performance of our proposed window-based approach. In particular, we compare our local experimental results with a real setup for streaming data, using Apache Spark. The results of the experiments, which are performed on several IoT datasets, show the efficiency of our proposed solution in terms of processing time. The results are also used to provide an integrated data view to the users.

The purpose of this article is to investigate modern approaches, methodologies, and principles in the application of geoinformation systems (GIS) in urban planning decisions. The study provides an in-depth analysis of the functional capabilities of GIS and emphasizes their integration with complementary technologies such as remote sensing, global positioning systems (GPS), and building information modeling (BIM). By consolidating diverse spatial data sources, GIS facilitates comprehensive analysis and visualization of urban environments, thereby supporting informed decision-making in urban development. Recent technological advancements have enabled the creation of dynamic digital urban models that simulate various planning scenarios. These models allow urban planners to assess infrastructure requirements, optimize land use, and predict future growth trends. The integration of real-time data via cloud-based platforms further enhances the responsiveness and accuracy of urban planning processes, contributing significantly to sustainable urban development. The integration of data from numerous sources - satellite imagery, aerial photography, cadastral registers, GPS measurements, and remote sensing results - allows us to create a comprehensive digital model of urban space that provides a sound analysis of spatial processes. Integrated geographic information systems open up new horizons for urban planning and management, providing operational monitoring and accurate forecasting of changes in the urban environment. Further research should focus on optimizing data analysis algorithms and developing flexible information platforms with the introduction of artificial intelligence and the Internet of Things technologies, which will significantly expand the capabilities of modern urban planning solutions. Such a synergistic approach allows identifying potential problem areas, optimizing the location of infrastructure facilities, and developing recommendations for modernizing urban planning policy. The research findings confirm that geoinformation systems are an essential tool in modern urban planning. They not only improve the precision of spatial analysis but also contribute to the efficient allocation of resources and the formulation of proactive strategies to tackle urban challenges. Consequently, the use of GIS is instrumental in promoting sustainable growth and enhancing the quality of life in urban areas.

BackgroundThe field of “Big Health,” which encompasses the integration of big data in healthcare, has seen rapid development in recent years. As big data technologies continue to transform healthcare, understanding emerging trends and key advancements within the field is essential.MethodsWe retrieved and filtered articles and reviews related to big data analytics in health research from the Web of Science Core Collection, including SCI Expanded and SSCI, covering the period from 2009 to 2024. Bibliometric and co-citation analyses were conducted using VOSviewer and CiteSpace.ResultsA total of 13,609 papers were analyzed, including 10,702 original research and 2,907 reviews. Co-occurrence word analysis identified six key research areas: (1) the application of big data analytics in health decision-making; (2) challenges in the technological management of health and medical big data; (3) integration of machine learning with health monitoring; (4) privacy and ethical issues in health and medical big data; (5) data integration in precision medicine; and (6) the use of big data in disease management and risk assessment. The co-word burst analysis results indicate that topics such as personalized medicine, decision support, and data protection experienced significant growth between 2015 and 2020. With the advancement of big data technologies, research hotspots have gradually expanded from basic data analysis to more complex application areas, such as the digital transformation of healthcare, digital health strategies, and smart health cities.ConclusionThis study highlights the growing impact of big data analytics in healthcare, emphasizing its role in decision-making, disease management, and precision medicine. As digital transformation in healthcare advances, addressing challenges in data integration, privacy, and machine learning integration will be crucial for maximizing the potential of big data technologies in improving health outcomes.

Real-time data integration from sources becomes critical in today's world to make efficient decisions for industries across finance, health, and retail. Informatica PowerCenter is considered one of the fastest and most fully featured Transform, Load (ETL) tools that face many challenges in real-time cloud integrations through Snowflake and Microsoft Azure, among many more. This assignment looks at a study of how real-time cloud integration is enabled by Informatica PowerCenter, as well as its architecture, features, and applicability [8]. With the key features of Change Data Capture (CDC), parallel processing, and workflow automation, PowerCenter manages high-frequency data ingestion with minimum latency and data consistency. We discuss some of the critical challenges around latency, security, and cost management and review the functionality of PowerCenter in the cloud, which is a scalable platform that allows compliance with industry standards. These findings show that with the integration of cloud platforms, PowerCenter can take the real-time processing of data further, thereby providing a scalable solution to organizations for pursuing analytics on the cloud. Many case studies in the financial and healthcare industries focus on how the PowerCenter toolset can drive operational efficiencies and timely decision-making. Therefore, this suggests that future advanced automation and machine learning integration would be recommended in this sphere of research at PowerCenter, optimizing real-time data integration.

In the evolving landscape of modern healthcare, real-time data integration has emerged as a critical component for delivering optimal patient care while maintaining robust security and regulatory compliance. This comprehensive article explores the implementation strategies, challenges, and outcomes of healthcare data integration systems across diverse clinical settings. The article examines the impact of integrated systems on clinical workflows, patient outcomes, and operational efficiency while addressing the complex requirements of data security and regulatory compliance. Through detailed case studies of large hospital networks and rural healthcare providers, this article demonstrates how strategic implementation of integrated healthcare systems can transform care delivery, enhance patient outcomes, and optimize resource utilization. The findings provide valuable insights for healthcare organizations at various stages of their digital transformation journey, offering evidence-based strategies for successful system implementation and optimization.

Predictive analytics has become a crucial tool in modern business decision-making, particularly in demand forecasting. By leveraging historical data and statistical modeling techniques, businesses can enhance resource allocation, optimize inventory management, and improve overall operational efficiency. Among various predictive analytics methods, time series models are widely employed due to their ability to capture trends, seasonality, and patterns in demand fluctuations. This explores the application of time series models, including Moving Average (MA), Autoregressive (AR), Autoregressive Integrated Moving Average (ARIMA), Seasonal ARIMA (SARIMA), Exponential Smoothing (Holt-Winters method), and advanced deep learning models such as Long Short-Term Memory (LSTM) networks. These models enable businesses to make data-driven decisions by accurately predicting future demand, thereby reducing costs and improving supply chain responsiveness. Furthermore, the integration of predictive analytics in business resource allocation extends beyond inventory management. It influences workforce planning, financial forecasting, and supply chain optimization, ensuring that organizations align their operations with anticipated market trends. However, despite the benefits, challenges such as data quality issues, model interpretability, external disruptions, and scalability concerns remain key obstacles in implementing time series forecasting. Emerging advancements in artificial intelligence (AI), big data analytics, and cloud-based forecasting solutions are shaping the future of predictive analytics. The integration of real-time data and machine learning models offers new opportunities for businesses to enhance forecasting accuracy and agility. As demand forecasting continues to evolve, organizations must adopt a strategic approach to model selection and implementation to fully capitalize on the advantages of predictive analytics. This study provides a comprehensive overview of predictive analytics for demand forecasting, highlighting the role of time series models in optimizing business resource allocation and discussing future trends that will drive further innovation in this field.

The enormous volumes of data created and maintained by industries, research institutions are on the verge of outgrowing its infrastructure. The advancements in the organization's work flow include data storage, data management, data maintenance, data integration, and data interoperability. Among these levels, data integration and data interoperability can be the two major focus areas for the organizations which tend to implement advancements in their workflow. Overall, data integration and data interoperability influence the organization's performance. The data integration and data interoperability are complex challenges for the organizations deploying big data architectures due to the heterogeneous nature of data used by them. Therefore, it requires a comprehensive approach to negotiate the challenges in integration and interoperability. This paper focuses on the challenges of data integration and data interoperability in big data.

Objective: Through the practice discussion and exchange of the integration of big data and pharmaceutical standards, to provide experience reference for the practice of information technology in the management of rational drug use. Methods: The application of data information in the practice of hospital pharmacy management is demonstrated through the introduction of practical cases and data achievements at different levels. Results: The integration of medical big data and pharmaceutical standards effectively improved the hospital's quantitative quality management indicators and the informatization of many pharmaceutical management methods in the hospital is realized. Conclusions: The integration of medical big data and pharmaceutical standards can effectively help improve the management level of rational drug use in hospitals, and there are still many possibilities for future exploration.

Biological data are generated at unprecedentedly exponential rates, posing considerable challenges in big data deposition, integration and translation. The BIG Data Center, established at Beijing Institute of Genomics (BIG), Chinese Academy of Sciences, provides a suite of database resources, including (i) Genome Sequence Archive, a data repository specialized for archiving raw sequence reads, (ii) Gene Expression Nebulas, a data portal of gene expression profiles based entirely on RNA-Seq data, (iii) Genome Variation Map, a comprehensive collection of genome variations for featured species, (iv) Genome Warehouse, a centralized resource housing genome-scale data with particular focus on economically important animals and plants, (v) Methylation Bank, an integrated database of whole-genome single-base resolution methylomes and (vi) Science Wikis, a central access point for biological wikis developed for community annotations. The BIG Data Center is dedicated to constructing and maintaining biological databases through big data integration and value-added curation, conducting basic research to translate big data into big knowledge and providing freely open access to a variety of data resources in support of worldwide research activities in both academia and industry. All of these resources are publicly available and can be found at http://bigd.big.ac.cn.

Urban flood prevention faces challenges in real-time monitoring, data integration, and decision-making support, particularly under constraints of sensitive data security and efficient emergency response. This study proposes an artificial intelligence-assisted decision-making framework for urban flood prevention, integrating multi-dimensional data fusion via an intranet. A city-level digital twin platform is developed, combining real-time sensor data, meteorological information, and social media data using a geographic information system, the Internet of Things, and artificial intelligence algorithms. The framework ensures secure data processing through intranet-based operations, enhancing flood prediction accuracy and emergency response efficiency. The system was successfully implemented in Hebi City, Henan Province, demonstrating significant improvements in flood risk management. This research advances urban resilience and provides a scientific basis for flood management in smart cities.

The main problem has been managing databases because they are continuously expanding quickly and getting more complicated in terms of volume, variety, and velocity. Currently, Relational Database Management Systems (RDMS), also known as SQL, or traditional search engines are primarily used to manage and utilize data collection. Due to their highly semantic properties and applications, relational databases have emerged as powerful and superior during the previous two decades. The handling of a high volume of data and the variability in data type and structure has become a laborious task since Big Data entered the IT industry. Relational databases are inadequate to handle such big data due to their rigid data limitations, structure, relations, and other factors. As a result, data aggregation is rendered impossible. NoSQL databases offer a practical and understandable foundation for combining massive amounts of data, structures, and interactions. Data modeling and migration are now needed in order to define the issue. There isn't yet a tool for switching from relational to no-SQL databases. For this migration, relational (SQL) database queries must be converted to NoSQL database queries. Since relational databases (RDBMS) have found it difficult to keep up with modernization, NoSQL has emerged as the most practical database. This essay aims to highlight and assess the applications of NoSQL, the notion of NoSQL data modeling, and the NoSQL database migration procedure. NoSQL, NoSQL Roadmap, Data Modeling, ETL Approach for Data Migration, Data Loading, and Data Extraction.

Enterprise Resource Planning (ERP) systems have emerged as essential tools in streamlining financial reporting and improving management efficiency across various industries, including large-scale energy operations. This theoretical review explores how ERP systems enhance financial reporting by integrating real-time data, automating financial processes, and fostering transparency. In energy operations, where financial activities are complex and data-intensive, ERP systems centralize financial data, reducing errors, improving compliance, and supporting decision-making through enhanced visibility and accuracy. By automating tasks such as budgeting, forecasting, and financial statement generation, ERP systems enable energy companies to efficiently manage their financial resources, thus optimizing cost control and profitability. Additionally, ERP systems provide significant benefits for management efficiency, particularly in large-scale operations. The integration of financial, operational, and managerial processes within a single platform allows for better resource allocation, improved communication across departments, and faster response times to market fluctuations. These capabilities are crucial in the energy sector, where companies face volatile market conditions, regulatory pressures, and high operational risks. ERP systems enhance managerial decision-making by providing real-time insights into key performance indicators (KPIs), enabling executives to make data-driven decisions that support long-term strategic goals. However, implementing ERP systems in large-scale energy operations presents challenges, including high costs, lengthy implementation periods, and the need for employee training. This review also examines these barriers and suggests strategies for successful ERP adoption, including phased implementation, stakeholder engagement, and customization to industry-specific needs. In conclusion, ERP systems hold the potential to revolutionize financial reporting and management efficiency in large-scale energy operations. By leveraging ERP's real-time data integration and process automation, energy companies can significantly improve their financial management, enhance decision-making, and adapt more quickly to market changes. Keywords: ERP Systems, Financial Reporting, Management Efficiency, Large-Scale Energy Operations, Process Automation, Real-Time Data Integration, Decision-Making, Cost Control, Market Fluctuations, Energy Sector.

Rice (Oryza sativa L.) is a staple crop feeding over half of the global population, yet its productivity is significantly threatened by diseases such as bacterial blight, blast, sheath blight, and tungro. Traditional diagnostic methods are often time-consuming, reliant on expert intervention, and inaccessible to smallholder farmers. This study aimed to develop and validate a mobile-based expert system for rice disease diagnosis using forward chaining inference algorithms and real-time data integration from IoT-based environmental sensors and GPS-enabled devices. The mobile application was designed with a structured knowledge base derived from scientific literature and expert consultations. Field trials were conducted across ten farms, where farmers and extension workers provided real-time data inputs, including visual symptoms and environmental parameters. Diagnostic accuracy, sensitivity, and specificity of the system were evaluated statistically using confusion matrix analysis and ROC curve analysis. Results revealed an average diagnostic accuracy of 97.01%, with sensitivity at 90.69% and specificity at 85.37%. Farms with better environmental data calibration and trained users demonstrated higher diagnostic precision. Statistical analysis confirmed significant improvement over traditional diagnostic methods, with variability attributed to environmental data noise and user inconsistencies. The study highlights the importance of real-time data acquisition, localized knowledge bases, and user-friendly interfaces in enhancing diagnostic reliability. Future recommendations include integrating AI-based predictive analytics, expanding multilingual support, and implementing farmer training programs for improved data accuracy. This mobile-based expert system represents a scalable, practical solution for disease management, with significant potential to improve crop health, farmer decision-making, and sustainable agriculture practices.