Intelligent Visualization System for Big Multi-source Medical Data Based on Data Lake

As the complexity of intelligent public transportation systems continues to increase and the degree of interconnection of equipment at all levels gradually deepens, building a digital twin system will generate a large amount of multi-source heterogeneous data. Effective processing and in-depth mining of data can provide real-time and intelligent decision-making optimization for the construction of digital twin systems, thereby efficiently handling complex emergencies. This paper conducts a systematic review on the processing methods and technologies of multi-source heterogeneous twin data in traffic dynamic scenarios. First, the content and classification of multi-source heterogeneous twin data in the operation process of intelligent bus systems are clarified; secondly, the multi-source The data processing methods and technologies applied in each stage of data collection, data storage, data fusion and data analysis in heterogeneous twin data processing are analyzed, and the advantages, disadvantages and applications of various methods and technologies are analyzed; finally, corresponding to the multiple This paper summarizes the methods and technologies for processing multi-source heterogeneous twin data, and points out the challenges and development trends faced by the current multi-source heterogeneous twin data processing methods and technologies.

The study of the prediction of stock market volatility is of great significance to rationally control financial market risks and increase excessive investment returns and has received extensive attention from academic and commercial circles. However, as a dynamic and complex system, the stock market is affected by multiple factors and has a comprehensive capability to include complex financial data. Given that the explanatory variables of influencing factors are diverse, heterogeneous and complex, the existing intelligent algorithms have great limitations for the analysis and processing of multi-source heterogeneous data in the stock market. Therefore, this study adopts the edge weight and information transmission mechanism suitable for subgraph data to complete node screening, the gate recurrent unit (GRU) and long short-term memory (LSTM) to aggregate subgraph nodes. The compiled data contain the metapaths of three types of index data, and the introduction of the association relationship attention dimension effectively mines the implicit meanings of multi-source heterogeneous data. The metapath attention mechanism is combined with a graph neural network to complete the classification of multi-source heterogeneous graph data, by which the prediction of stock market volatility is realized. The results show that the above method is feasible for the fusion of heterogeneous stock market data and the mining of implicit semantic information of association relations. The accuracy of the proposed method for the prediction of stock market volatility in this study is 16.64% higher than that of the dimensional reduction index and 14.48% higher than that of other methods for the fusion and prediction of heterogeneous data using the same model.

With the rapid development of smart grid technology, a large amount of multi-source heterogeneous data has been generated in the power system, and its effective utilization is crucial for the optimization operation, demand prediction, and anomaly detection of the power system. However, the fusion processing of multi-source heterogeneous data faces many challenges, such as inconsistent data format, granularity, and quality, and direct fusion can easily lead to information redundancy and contradictions. A multi-source heterogeneous data fusion technology based on big data mining has been proposed to address the above issues. This method combines the advantages of convolutional neural networks and gated recurrent units to automatically extract features from image and sequence data and handle long-term dependency issues in time series data. Meanwhile, the K-means clustering algorithm is used to preprocess the data and train a specialized ConvGRU model. The results showed that in short-term load forecasting and abnormal electricity consumption behaviour detection tasks, the accuracy of this method reached 96.3% and 98.7%, respectively, with AUC values of 0.994 and 0.996. Compared to models that use only CNN or GRU, the performance is significantly improved. This method effectively solves the problem of integrating and processing multi-source heterogeneous power data, improves the accuracy and efficiency of power system data analysis, and provides strong support for the optimized operation of smart grids.

In medical domain, huge amounts of data are generated at all times. These data are usually difficult to access, with poor data quality and many data islands. Besides, with a wide range of sources and complex structure, these data contain essential information and are difficult to manage. However, few existing data management frameworks based on Data Lake excel in solving the persistence and the analysis efficiency for medical multi-source heterogeneous data. In this paper, we propose an efficient Multi-source Heterogeneous Data Lake Platform (MHDP) to realize the efficient medical data management. Firstly, we propose an efficient and unified method based on Data Lake to store data of different types and different sources persistently. Secondly, based on the unified data store, an efficient multi-source heterogeneous data fusion is implemented to effectively manage data. Finally, an efficient data query strategy is carried out to assist doctors in medical decision-making. In-depth analysis on applications shows that MHDP delivers better performance for data management in medical domain.

Aiming at the low cleaning rate of the traditional multi-source heterogeneous power grid big data cleaning model, a multi-source heterogeneous power grid big data cleaning model based on machine learning classification algorithm is designed. By capturing high-quality multi-source heterogeneous power grid big data, weight labeling of data source importance measurement, data attributes and tuples, and constructing Tan network based on the idea of machine learning classification algorithm, the data probability value is finally used to complete the classification and cleaning of inaccurate data. Experiments show that the model based on machine learning classification algorithm can effectively improve the imprecise data cleaning rate compared with the traditional model to solve multi-source heterogeneous imprecise data cleaning.

The development of an Automated Subgrade Monitoring System (ASMS) represents an important means of advancing the digitalization and intelligent operation and maintenance of high-speed railway (HSR) subgrades. However, current systems exhibit considerable constraints in managing multi-source heterogeneous data and effectively integrating engineering information. This study addresses these limitations through the design and implementation of a Building Information Modeling (BIM)-integrated Data Management and Analysis Subsystem (DMAS) founded on digital twin theory. The system establishes an engineering data management architecture optimized for the efficient integration and governance of heterogeneous multi-source data. Through the implementation of the Entity-Relationship (E-R) model, complex engineering data are transformed into an atomic-level storage structure, enhancing data consistency and retrieval efficiency. The system incorporates a hybrid sharding strategy and hierarchical access control mechanism to enhance scalability and security. Furthermore, a BIM attribute extension and logical mapping engine facilitates seamless integration between the physical model and sensor data, effectively addressing the frequently overlooked challenge of data coupling. The integration of a lightweight WebGL rendering engine enables cross-platform model visualization, dynamic data binding, real-time interaction, and precise alarm localization. Focusing on the response layer’s core functions, DMAS establishes a structured and functionally coordinated data support system, demonstrating robust engineering adaptability in multi-source data governance, data modeling, and visualized interaction. Through its modular and extensible architecture, DMAS enhances system performance while providing capacity for future intelligent evolution toward advanced digital twin capabilities, thereby establishing a solid technical foundation for intelligent infrastructure monitoring and management.

Cloud computing and big data: Technologies and applications

The exponential growth of heterogeneous enterprise data has exposed the limitations of traditional Big Data architectures, which prioritize storage scalability over semantic understanding. This paper presents a Semantic Data Lake (SDL) framework that integrates Big Data technologies with knowledge graphs and ontology-driven reasoning to enhance enterprise decision support. The SDL architecture introduces a multi-layered system encompassing data ingestion, semantic annotation, ontology alignment, and graph-based reasoning for contextual query processing. Implemented using Hadoop, Spark, and GraphDB, the framework demonstrates superior performance in query efficiency, scalability, and semantic accuracy compared to conventional data lakes. Experimental evaluations show up to a 40% reduction in query latency and a 19% improvement in semantic precision, achieved through ontology mapping and reasoning-based query optimization. The results validate that semantic enrichment transforms data lakes into intelligent ecosystems capable of delivering explainable, context-aware analytics. The paper concludes by outlining future research directions, including AI-driven ontology learning, federated semantic integration, and hybrid reasoning for real-time knowledge discovery.

Advances in Big Data architectures and AI techniques based on Federated Learning improve the opportunities for collaboration in healthcare for use cases such as pandemic surveillance and precision medicine. Big Data infrastructures have recently evolved from classical data warehouses to include data lakes and data hubs. Fleets of data lakes and data hubs form a cloud of trust, enabling federated computing across organization boundaries. These concepts provide the foundations for a Big Data architecture for global healthcare based on semantically compliant integration of data hubs and lakes using open data exchange and description standards created by the international community. Recent developments in open-source software libraries for Federated Learning support data protection and data sovereignty with minimal data exposure to other organizations. Proposed solutions combine the benefits of Federated Learning with the elastic, self-service, and pay-as-you-go aspects of the public cloud for AI in healthcare.Future developments target global health, with specific applications to pandemic surveillance and response, and to precision medicine and genomics. The evolution of architectures for AI in health, based on Federated Learning and Federated Reinforcement Learning, is guided by the cross-functional collaboration model FR-FA-CFCM. The model formalizes the rules, roles, and mechanisms for co-creating software solutions among different stakeholder roles across organization boundaries, on-premise and in the cloud.

In this paper, we propose a novel framework of integration of multi-source heterogeneous data and intelligent maintenance decision support for High-speed Railway Signal System. The framework uses the multi-agent system to deal with the multi-source data. Each agent has its own task varying from transferring RDB (Rational Database) data to RDF (Resource Description Framework) file, and then fusing local RDF to global RDF by Apache Jena API and D2RQ. In addition, a CBR (Case-Based Reasoning) based intelligent maintenance decision support for the fault diagnosis has been constructed by combing the rough set with the Euclidean distance. The correctness and efficiency of the proposed framework has been verified by maintenance data derived from VOBE (Vehicle Equipment On Board) of CTCS-300T operating on Wuhan-Guangzhou Line.

This paper proposes an energy efficiency optimisation method combined with big data analysis to address the energy efficiency management challenges in intelligent manufacturing workshops. The research constructs a multi-source heterogeneous energy consumption data collection and processing architecture, develops energy consumption pattern recognition technology based on wavelet transform and deep learning, establishes equipment energy efficiency assessment and parameter optimisation mechanism, realises production and energy co-scheduling, and constructs a digital twin-driven energy efficiency closed-loop optimisation system. Through the implementation and application in automotive parts, electronic products and petrochemical enterprises, the method is verified to be of significant value in improving energy efficiency, reducing production costs and promoting green manufacturing, providing technical support for the green transformation of the manufacturing industry.

Big Sensor Data Applications in Urban Environments

Managing, processing and understanding big healthcare data is challenging, costly and demanding. Without a robust fundamental theory for representation, analysis and inference, a roadmap for uniform handling and analyzing of such complex data remains elusive. In this article, we outline various big data challenges, opportunities, modeling methods and software techniques for blending complex healthcare data, advanced analytic tools, and distributed scientific computing. Using imaging, genetic and healthcare data we provide examples of processing heterogeneous datasets using distributed cloud services, automated and semi-automated classification techniques, and open-science protocols. Despite substantial advances, new innovative technologies need to be developed that enhance, scale and optimize the management and processing of large, complex and heterogeneous data. Stakeholder investments in data acquisition, research and development, computational infrastructure and education will be critical to realize the huge potential of big data, to reap the expected information benefits and to build lasting knowledge assets. Multi-faceted proprietary, open-source, and community developments will be essential to enable broad, reliable, sustainable and efficient data-driven discovery and analytics. Big data will affect every sector of the economy and their hallmark will be ‘team science’.

In our digital environment, there are both opportunities and challenges associated with the unprecedented growth of data. We will discuss the two areas of investigation (Big Data Analytics and Data Visualization), which are seldom related fields, and both are essential to achieving insights from Big Data. We will discuss the fundamental traits of Big Data, including volume, velocity, and variety, and how data is transitioned from its initial state to data from analytics. We will then discuss the key component that Data Visualization represents as a cognitive tool to make the complex nature of analytics intuitive, readable, and actionable. This presentation will argue for the complementary nature of the two fields, further supporting the notion that this is not optional, but mandatory, given the nature of Big Data Analytics in relation to science, commerce, and government. We will focus on the potential of the lifecycle of data, from collection to visualization, data use, embedded analytics, and the ethical conceptualization of challenges, as well as a future forecast of system technology that will impact the reestablishment of our knowledge and knowledge practices

The personalized health care service utilizes the relational patient data and big data analytics to tailor the medication recommendations. However, most of the health care data are in unstructured form and it consumes a lot of time and effort to pull them into relational form. This study proposes a novel data lake architecture to reduce the data ingestion time and improve the precision of healthcare analytics. It also removes the data silos and enhances the analytics by allowing the connectivity to the third-party data providers (such as clinical lab results, chemist, insurance company, etc.). The data lake architecture uses the Hadoop Distributed File System (HDFS) to provide the storage for both structured and unstructured data. This study uses K-means clustering algorithm to find the patient clusters with similar health conditions. Subsequently, it employs a support vector machine to find the most successful healthcare recommendations for the each cluster. Our experiment results demonstrate the ability of data lake to reduce the time for ingesting data from various data vendors regardless of its format. Moreover, it is evident that the data lake poses the potential to generate clusters of patients more precisely than the existing approaches. It is obvious that the data lake provides an unified storage location for the data in its native format. It can also improve the personalized healthcare medication recommendations by removing the data silos.