MHDP: An Efficient Data Lake Platform for Medical Multi-source Heterogeneous Data

With the rapid development of smart grid technology, a large amount of multi-source heterogeneous data has been generated in the power system, and its effective utilization is crucial for the optimization operation, demand prediction, and anomaly detection of the power system. However, the fusion processing of multi-source heterogeneous data faces many challenges, such as inconsistent data format, granularity, and quality, and direct fusion can easily lead to information redundancy and contradictions. A multi-source heterogeneous data fusion technology based on big data mining has been proposed to address the above issues. This method combines the advantages of convolutional neural networks and gated recurrent units to automatically extract features from image and sequence data and handle long-term dependency issues in time series data. Meanwhile, the K-means clustering algorithm is used to preprocess the data and train a specialized ConvGRU model. The results showed that in short-term load forecasting and abnormal electricity consumption behaviour detection tasks, the accuracy of this method reached 96.3% and 98.7%, respectively, with AUC values of 0.994 and 0.996. Compared to models that use only CNN or GRU, the performance is significantly improved. This method effectively solves the problem of integrating and processing multi-source heterogeneous power data, improves the accuracy and efficiency of power system data analysis, and provides strong support for the optimized operation of smart grids.

The study of the prediction of stock market volatility is of great significance to rationally control financial market risks and increase excessive investment returns and has received extensive attention from academic and commercial circles. However, as a dynamic and complex system, the stock market is affected by multiple factors and has a comprehensive capability to include complex financial data. Given that the explanatory variables of influencing factors are diverse, heterogeneous and complex, the existing intelligent algorithms have great limitations for the analysis and processing of multi-source heterogeneous data in the stock market. Therefore, this study adopts the edge weight and information transmission mechanism suitable for subgraph data to complete node screening, the gate recurrent unit (GRU) and long short-term memory (LSTM) to aggregate subgraph nodes. The compiled data contain the metapaths of three types of index data, and the introduction of the association relationship attention dimension effectively mines the implicit meanings of multi-source heterogeneous data. The metapath attention mechanism is combined with a graph neural network to complete the classification of multi-source heterogeneous graph data, by which the prediction of stock market volatility is realized. The results show that the above method is feasible for the fusion of heterogeneous stock market data and the mining of implicit semantic information of association relations. The accuracy of the proposed method for the prediction of stock market volatility in this study is 16.64% higher than that of the dimensional reduction index and 14.48% higher than that of other methods for the fusion and prediction of heterogeneous data using the same model.

As the complexity of intelligent public transportation systems continues to increase and the degree of interconnection of equipment at all levels gradually deepens, building a digital twin system will generate a large amount of multi-source heterogeneous data. Effective processing and in-depth mining of data can provide real-time and intelligent decision-making optimization for the construction of digital twin systems, thereby efficiently handling complex emergencies. This paper conducts a systematic review on the processing methods and technologies of multi-source heterogeneous twin data in traffic dynamic scenarios. First, the content and classification of multi-source heterogeneous twin data in the operation process of intelligent bus systems are clarified; secondly, the multi-source The data processing methods and technologies applied in each stage of data collection, data storage, data fusion and data analysis in heterogeneous twin data processing are analyzed, and the advantages, disadvantages and applications of various methods and technologies are analyzed; finally, corresponding to the multiple This paper summarizes the methods and technologies for processing multi-source heterogeneous twin data, and points out the challenges and development trends faced by the current multi-source heterogeneous twin data processing methods and technologies.

With continuous innovation of the “Internet+” and development of related industries, huge amounts of data need to be integrated on the basis of spatial data analysis and processing. Therefore, efficient integration of multi-source heterogeneous spatial data and data interoperability requirements has become more and more urgent. The design of a multisource heterogeneous vector spatial data integration scheme on WebGIS has been proposed by GeoJSON as the universal transformation format, based on the analysis of the existing multi-source heterogeneous spatial data conversion method. Finally, a prototype of the online prototype system was developed with. Net platform to verify the feasibility and practicability of the scheme. The scheme has the characteristics of high efficiency, easy to expand and low cost, and provides a new way to realize the data integration of “Internet+”.

최근 증가하는 협업 연구는 원격 의료 데이터 공유 및 외부 기관의 데이터 접근을 요구하고 있다. 본 논문에서는 융통성 있고 효율적인 의료 데이터 관리를 위해 웹서비스를 이용하여 그리드와 PACS 간의 상호운용성 프레임워크를 제안하고 구현하였다. DICOM 표준은 PACS와 이미지 데이터베이스 사이에 의료 이미지 데이터의 교환과 전송을 정의한다. 그러나 병원들 간 의료 데이터 교환은 신뢰할 수 있는 기관들 사이에서만 제공된다. 더군다나 DICOM은 의료 데이터 관리 기능을 제공하지 않고 그리드 미들웨어는 DICOM 데이터에 접근 가능한 표준 툴킷을 제공하고 있지 않고 있다. 본 논문에서는 이러한 문제를 해결하기 위하여 PACS 간의 연동과 의료 데이터 관리를 가능하게 하는 웹 서비스기반의 그리드 서비스 중계자(Web Services-based Grid Service Mediator, WGSM)를 개발하였다. WGSM은 압축 중계자, GridFTP 중계자, RFT 중계자, MyProxy 중계자, MDS 중계자, RLS 중계자 등으로 구성된다. 제안한 웹서비스 기반의 프레임워크는 협업 환경에서 PACS 간 사용자 인증 및 안전한 데이터 접근을 제공한다. 특히, WGSM은 일반 사용자들이 그리드 미들웨어에 대한 어떠한 지식이 없어도 간단하고 효율적인 방법으로 원격 PACS를 접근할 수 있게 한다. Recently the increasing collaborative research requires the remote medical and clinical data sharing and access of external institutions. In this paper, an interoperability framework between Grid and PACS using Web services is proposed and implemented in order to provide flexible and efficient medical data management. The Digital Imaging and Communications in Medicine(DICOM) standard defines medical image data exchange and transfer between PACSs and image databases. However, medical data exchange between hospitals is limited within the trusted and static environments. Moreover, DICOM does not provide medical data management and the Grid middleware does not include standard toolkit to access DICOM data. To address this issue, a Web services-based Grid Service Mediator (WGSM) which provides PACS integration and medical image data management is developed. The WGSM consists of several service mediators such as compress mediator, GridFTP mediator, RFT mediator, MyProxy mediator, MDS mediator, and RLS mediator and others. The proposed Web services-based framework provides user authentication and secure data access between PACSs in collaborative environments. In particular, the WGSM allows ordinary users to access remote PACS data in a simple and efficient manner without any the knowledge about underlying Grid middleware.

The complexity and diversity of equipments presented in the development of modern industrial technology brings the wide application of deep learning in fault diagnosis. However, the multi-source heterogeneous data collected from different sensors applied in industrial production cannot be used by traditional neural networks directly, which registers a difficulty to fault diagnosis based on deep learning. In addition, with only one of the data used, traditional deep learning fault diagnosis methods ignores the correlation between heterogeneous data, which may result in the loss of useful information and the accuracy of fault diagnosis. To solve the above problems, a fault diagnosis framework based on deep learning for multi-source heterogeneous data fusion is proposed. In the feature learning stage, stack self-encoder (SAE) and convolution neural network (CNN) are used to extract deep features from one-dimensional vibration data and two-dimensional image data respectively, and then feature fusion fault diagnosis is carried out. It solves the problem that traditional neural network can't make full use of multi-source heterogeneous data, and improves the accuracy of fault diagnosis. What's more, the bearing data of Case Western Reserve University are used to verify the validity of the diagnosis method proposed in this paper.

This paper proposes an optimization algorithm for constructing education knowledge graphs based on multi-source heterogeneous data fusion (MEGO). The algorithm first cleans and extracts features from multi-source education data, and uses deep learning models to mine deep features of different types of data, then dynamically adjusts the weights of each data source through the attention mechanism to achieve efficient fusion of multi-source data, finally, the graph structure is optimized with the help of graph convolutional networks to remove noise and redundant information. The experiment selects <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{1 0 0, 0 0 0}$</tex> multi-source heterogeneous education data from 10 universities in a certain region within 5 years, and compares them with the traditional TransE algorithm and the KG-Basic algorithm based on a single data source. The results show that the accuracy of the MEGO algorithm is 92.3%, which is 8.7% higher than the TransE algorithm and 12.5% higher than the KG -Basic algorithm, the recall rate is 89.6%, which is 7.2% and 10.8% higher than the two comparison algorithms respectively, the F1 value is 90.9%, which is 7.9% and 11.6% higher respectively, the graph construction time is only 12.5 minutes, which is 35.2% shorter than TransE and 42.8% shorter than KG-Basic. In summary, the MEGO algorithm can effectively improve the quality and construction efficiency of the education knowledge graph, and provide solid technical support for multi-dimensional and precise education quality evaluation.

With the growing importance of the fuzzy spatiotemporal data in information application, there is an increasing need for researching on the integration method of multi-source heterogeneous fuzzy spatiotemporal data. In this paper, we first propose a fuzzy spatiotemporal RDF graph model based on RDF (Resource Description Framework) that proposed by the World Wide Web Consortium (W3C) to represent data in triples (subject, predicate, object). Secondly, we analyze and classify the related heterogeneous problems of multi-source heterogeneous fuzzy spatiotemporal data, and use the fuzzy spatiotemporal RDF graph model to define the corresponding rules to solve these heterogeneous problems. In addition, based on the characteristics of RDF triples, we analyze the heterogeneous problem of multi-source heterogeneous fuzzy spatiotemporal data integration in RDF triples, and provide the integration methods FRDFG in this paper. Finally, we report our experiments results to validate our approach and show its significant superiority.

Aiming at the low cleaning rate of the traditional multi-source heterogeneous power grid big data cleaning model, a multi-source heterogeneous power grid big data cleaning model based on machine learning classification algorithm is designed. By capturing high-quality multi-source heterogeneous power grid big data, weight labeling of data source importance measurement, data attributes and tuples, and constructing Tan network based on the idea of machine learning classification algorithm, the data probability value is finally used to complete the classification and cleaning of inaccurate data. Experiments show that the model based on machine learning classification algorithm can effectively improve the imprecise data cleaning rate compared with the traditional model to solve multi-source heterogeneous imprecise data cleaning.

In order to fully utilize the features of multi-source heterogeneous data and effectively improve the accuracy and efficiency of fault diagnosis of rotating machinery, a multi-source heterogeneous data diagnosis method based on parameter collaborative optimization multi-scale convolutional autoencoder (MSCAE) is proposed. Firstly, multi-scale information learning is integrated into the convolutional autoencoder (CAE) to consider the temporal and spatial feature information of the diagnostic object simultaneously. To improve the training and diagnostic efficiency of MSCAE, a quantum particle swarm optimization (QPSO) module is used to perform hyperparameter optimization on it using chaos initialization and dynamic weight strategy (DWS). Besides, the sparse attention mechanism is introduced into the MSCAE model to improve the recognition rate of key fault features hidden in the original heterogeneous signals. Finally, the confusion matrix and visualization techniques are used to achieve fault classification. The experimental results demonstrate that after 100 experiments, the proposed method has an average diagnostic accuracy of 98.5 % and strong robustness to noise, providing a new method for rotating machinery fault diagnosis based on multi-source heterogeneous data.

With the rapid development of information technology, large amounts of multi-source data are constantly being generated in medical field. The automatic visualization system based on them has gained a lot of attention, since the intuitive data presentation can help even non-professional users effectively get the information hidden behind the separate data obtained from different scenarios and make better decisions. In this paper, based on the Data Lake architecture, we improve the performance of an existing novel data visualization recommendation system and resolve three challenges about the processing of multi-source and heterogeneous data. First, we build the framework based on Data Lake to store multi-source and heterogeneous data. Second, we optimize the data manipulation module in the visualization system based on the distributed processing power of Data Lake to get potentially interesting visualization candidates in a short time. Third, we efficiently run exploratory queries on large datasets based on the calculation capability of Data Lake to meet the actual needs of users. According to the experiment results, our system demonstrates a remarkable acceleration effect on the task of automatic visualization of big multi-source medical data.

Modern physics experiments are often led by large collaborations including scientists and institutions from different parts of the world. To cope with the ever increasing computing and storage demands, computing resources are nowadays offered as part of a distributed infrastructure. A critical challenge for present and future experiments is an efficient and reliable data distribution and access system. Rucio is a framework for data management, access and distribution, originally developed by the ATLAS experiment and later adopted by several scientific collaborations. It is currently used by the second generation gravitational wave (GW) detectors LIGO and Virgo, and is being evaluated for the Einstein Telescope (ET), the future third generation interferometer in preparation in Europe. In this contribution, the on-going R&amp;D for integration of Rucio within the ET computing infrastructure will be outlined. The activities include the setup of a Data Lake based on Rucio for future ET Mock Data Challenges and the customization of Rucio features for the GW community. The evaluation of RucioFS, a FUSE mount filesystem to provide the user with the well known POSIX-like view of the Rucio catalogue, and the possible implementation of additional features in it will be described. Moreover, the need for the integration of the ET Data Lake with mock Data Lakes belonging to other experiments within the astrophysics and GW communities will be addressed. This is a critical feature as data analysts in this field often require access to open data from other experiments for sky localisation and multi-messenger analysis.

The invasive pest Hyphantria cunea poses a serious threat to forestry and agricultural ecosystems. To mitigate this threat, accurate and efficient prediction of its distribution is critical for enabling proactive management and targeted control strategies. However, existing traditional statistical models and machine learning approaches exhibit limited predictive performance due to their inability to effectively capture complex spatiotemporal dependencies and process multi-source heterogeneous data. Furthermore, although deep learning has demonstrated progress in pest prediction, current models still suffer from critical limitations including inadequate feature fusion and insufficient critical temporal feature capture. To address this issue, this study proposes a novel Synergistic Fusion Bidirectional Long Short-Term Memory network (SF-BiLSTM). Centered on the synergistic fusion of multi-source heterogeneous data, its architecture consists of three core stages. Firstly, a Bidirectional LSTM (BiLSTM) is utilized to capture complex spatiotemporal dependencies within environmental sequences. Subsequently, dual-path temporal feature refinement is performed: one path employs mean pooling and layer normalization for feature compression, while the other integrates an attention sub-network to focus on critical time steps and conduct weighted fusion. Finally, an adaptive weighted fusion of the dual-path features is achieved through a weight learning network, yielding the final risk prediction. Experiments on the Shandong Province dataset from 2019 to 2024 demonstrate that the SF-BiLSTM model achieves outstanding comprehensive performance, with a Precision of 0.9667 and an F1-score of 0.9496, significantly outperforming traditional machine learning methods such as Gradient Boosting, KNN, and all baseline models. This model successfully reveals the spatiotemporal patterns and risk levels of Hyphantria cunea outbreaks in Shandong Province, providing reliable decision support for precise regional pest prevention and control.

Background A production line is the basic unit of smart factories and smart manufacturing. However, owing to the development of the industrial Internet of Things, sensors, and other technologies, more data are being collected, leading to a data explosion, and the heterogeneous nature of multiple sources makes it difficult to manage data in a unified manner. Methods A production line data collection, storage, and management system based on cloud-fog-edge computing collaboration and a digital twin was designed. Multi-source heterogeneous data were collected and transmitted based on the OPC UA, and an information model of the production line was established. Modules for data mapping, publishing, and receiving were developed to achieve unified data collection and transmission. The data storage and management platform was constructed by front-end and back-end separation technologies. Results The developed data collection and management system was verified for functionality and performance on a digital twin production line. Functional tests show that the system has the functions of data acquisition and transmission, device addition and viewing, device data querying and downloading, data and model visualization, and user rights setting. The average time for edge data collection and transmission is 183.6ms. The average response time of the cloud for fog requests is less than 1s. This shows that the system can satisfy the real-time requirements of a digital twin production line. Conclusions The proposed system is real-time and stable, providing support for big data and virtual-reality interaction in digital twins.

With the acceleration of intelligent transformation of energy system, the monitoring of equipment operation status and optimization of production process in thermal power plants face the challenge of multi-source heterogeneous data integration. In view of the heterogeneous characteristics of physical sensor data, including temperature, vibration and pressure that generated by boilers, steam turbines and other key equipment and real-time working condition data of SCADA system, this paper proposes a multi-source heterogeneous data fusion and analysis platform for thermal power plants based on edge computing and deep learning. By constructing a multi-level fusion architecture, the platform adopts dynamic weight allocation strategy and 5D digital twin model to realize the collaborative analysis of physical sensor data, simulation calculation results and expert knowledge. The data fusion module combines Kalman filter, wavelet transform and Bayesian estimation method to solve the problem of data time series alignment and dimension difference. Simulation results show that the data fusion accuracy can be improved to more than 98%, and the calculation delay can be controlled within 500 ms. The data analysis module integrates Dymola simulation model and AERMOD pollutant diffusion model, supports the cascade analysis of boiler combustion efficiency prediction and flue gas emission monitoring, system response time is less than 2 seconds, and data consistency verification accuracy reaches 99.5%.