Application of Python web crawler technology in infodemiology

With the growing importance of the fuzzy spatiotemporal data in information application, there is an increasing need for researching on the integration method of multi-source heterogeneous fuzzy spatiotemporal data. In this paper, we first propose a fuzzy spatiotemporal RDF graph model based on RDF (Resource Description Framework) that proposed by the World Wide Web Consortium (W3C) to represent data in triples (subject, predicate, object). Secondly, we analyze and classify the related heterogeneous problems of multi-source heterogeneous fuzzy spatiotemporal data, and use the fuzzy spatiotemporal RDF graph model to define the corresponding rules to solve these heterogeneous problems. In addition, based on the characteristics of RDF triples, we analyze the heterogeneous problem of multi-source heterogeneous fuzzy spatiotemporal data integration in RDF triples, and provide the integration methods FRDFG in this paper. Finally, we report our experiments results to validate our approach and show its significant superiority.

With continuous innovation of the “Internet+” and development of related industries, huge amounts of data need to be integrated on the basis of spatial data analysis and processing. Therefore, efficient integration of multi-source heterogeneous spatial data and data interoperability requirements has become more and more urgent. The design of a multisource heterogeneous vector spatial data integration scheme on WebGIS has been proposed by GeoJSON as the universal transformation format, based on the analysis of the existing multi-source heterogeneous spatial data conversion method. Finally, a prototype of the online prototype system was developed with. Net platform to verify the feasibility and practicability of the scheme. The scheme has the characteristics of high efficiency, easy to expand and low cost, and provides a new way to realize the data integration of “Internet+”.

With the rapid development of smart grid technology, a large amount of multi-source heterogeneous data has been generated in the power system, and its effective utilization is crucial for the optimization operation, demand prediction, and anomaly detection of the power system. However, the fusion processing of multi-source heterogeneous data faces many challenges, such as inconsistent data format, granularity, and quality, and direct fusion can easily lead to information redundancy and contradictions. A multi-source heterogeneous data fusion technology based on big data mining has been proposed to address the above issues. This method combines the advantages of convolutional neural networks and gated recurrent units to automatically extract features from image and sequence data and handle long-term dependency issues in time series data. Meanwhile, the K-means clustering algorithm is used to preprocess the data and train a specialized ConvGRU model. The results showed that in short-term load forecasting and abnormal electricity consumption behaviour detection tasks, the accuracy of this method reached 96.3% and 98.7%, respectively, with AUC values of 0.994 and 0.996. Compared to models that use only CNN or GRU, the performance is significantly improved. This method effectively solves the problem of integrating and processing multi-source heterogeneous power data, improves the accuracy and efficiency of power system data analysis, and provides strong support for the optimized operation of smart grids.

As the complexity of intelligent public transportation systems continues to increase and the degree of interconnection of equipment at all levels gradually deepens, building a digital twin system will generate a large amount of multi-source heterogeneous data. Effective processing and in-depth mining of data can provide real-time and intelligent decision-making optimization for the construction of digital twin systems, thereby efficiently handling complex emergencies. This paper conducts a systematic review on the processing methods and technologies of multi-source heterogeneous twin data in traffic dynamic scenarios. First, the content and classification of multi-source heterogeneous twin data in the operation process of intelligent bus systems are clarified; secondly, the multi-source The data processing methods and technologies applied in each stage of data collection, data storage, data fusion and data analysis in heterogeneous twin data processing are analyzed, and the advantages, disadvantages and applications of various methods and technologies are analyzed; finally, corresponding to the multiple This paper summarizes the methods and technologies for processing multi-source heterogeneous twin data, and points out the challenges and development trends faced by the current multi-source heterogeneous twin data processing methods and technologies.

The study of the prediction of stock market volatility is of great significance to rationally control financial market risks and increase excessive investment returns and has received extensive attention from academic and commercial circles. However, as a dynamic and complex system, the stock market is affected by multiple factors and has a comprehensive capability to include complex financial data. Given that the explanatory variables of influencing factors are diverse, heterogeneous and complex, the existing intelligent algorithms have great limitations for the analysis and processing of multi-source heterogeneous data in the stock market. Therefore, this study adopts the edge weight and information transmission mechanism suitable for subgraph data to complete node screening, the gate recurrent unit (GRU) and long short-term memory (LSTM) to aggregate subgraph nodes. The compiled data contain the metapaths of three types of index data, and the introduction of the association relationship attention dimension effectively mines the implicit meanings of multi-source heterogeneous data. The metapath attention mechanism is combined with a graph neural network to complete the classification of multi-source heterogeneous graph data, by which the prediction of stock market volatility is realized. The results show that the above method is feasible for the fusion of heterogeneous stock market data and the mining of implicit semantic information of association relations. The accuracy of the proposed method for the prediction of stock market volatility in this study is 16.64% higher than that of the dimensional reduction index and 14.48% higher than that of other methods for the fusion and prediction of heterogeneous data using the same model.

The development of an Automated Subgrade Monitoring System (ASMS) represents an important means of advancing the digitalization and intelligent operation and maintenance of high-speed railway (HSR) subgrades. However, current systems exhibit considerable constraints in managing multi-source heterogeneous data and effectively integrating engineering information. This study addresses these limitations through the design and implementation of a Building Information Modeling (BIM)-integrated Data Management and Analysis Subsystem (DMAS) founded on digital twin theory. The system establishes an engineering data management architecture optimized for the efficient integration and governance of heterogeneous multi-source data. Through the implementation of the Entity-Relationship (E-R) model, complex engineering data are transformed into an atomic-level storage structure, enhancing data consistency and retrieval efficiency. The system incorporates a hybrid sharding strategy and hierarchical access control mechanism to enhance scalability and security. Furthermore, a BIM attribute extension and logical mapping engine facilitates seamless integration between the physical model and sensor data, effectively addressing the frequently overlooked challenge of data coupling. The integration of a lightweight WebGL rendering engine enables cross-platform model visualization, dynamic data binding, real-time interaction, and precise alarm localization. Focusing on the response layer’s core functions, DMAS establishes a structured and functionally coordinated data support system, demonstrating robust engineering adaptability in multi-source data governance, data modeling, and visualized interaction. Through its modular and extensible architecture, DMAS enhances system performance while providing capacity for future intelligent evolution toward advanced digital twin capabilities, thereby establishing a solid technical foundation for intelligent infrastructure monitoring and management.

In this paper, we propose a novel framework of integration of multi-source heterogeneous data and intelligent maintenance decision support for High-speed Railway Signal System. The framework uses the multi-agent system to deal with the multi-source data. Each agent has its own task varying from transferring RDB (Rational Database) data to RDF (Resource Description Framework) file, and then fusing local RDF to global RDF by Apache Jena API and D2RQ. In addition, a CBR (Case-Based Reasoning) based intelligent maintenance decision support for the fault diagnosis has been constructed by combing the rough set with the Euclidean distance. The correctness and efficiency of the proposed framework has been verified by maintenance data derived from VOBE (Vehicle Equipment On Board) of CTCS-300T operating on Wuhan-Guangzhou Line.

With the acceleration of intelligent transformation of energy system, the monitoring of equipment operation status and optimization of production process in thermal power plants face the challenge of multi-source heterogeneous data integration. In view of the heterogeneous characteristics of physical sensor data, including temperature, vibration and pressure that generated by boilers, steam turbines and other key equipment and real-time working condition data of SCADA system, this paper proposes a multi-source heterogeneous data fusion and analysis platform for thermal power plants based on edge computing and deep learning. By constructing a multi-level fusion architecture, the platform adopts dynamic weight allocation strategy and 5D digital twin model to realize the collaborative analysis of physical sensor data, simulation calculation results and expert knowledge. The data fusion module combines Kalman filter, wavelet transform and Bayesian estimation method to solve the problem of data time series alignment and dimension difference. Simulation results show that the data fusion accuracy can be improved to more than 98%, and the calculation delay can be controlled within 500 ms. The data analysis module integrates Dymola simulation model and AERMOD pollutant diffusion model, supports the cascade analysis of boiler combustion efficiency prediction and flue gas emission monitoring, system response time is less than 2 seconds, and data consistency verification accuracy reaches 99.5%.

Aiming at the low cleaning rate of the traditional multi-source heterogeneous power grid big data cleaning model, a multi-source heterogeneous power grid big data cleaning model based on machine learning classification algorithm is designed. By capturing high-quality multi-source heterogeneous power grid big data, weight labeling of data source importance measurement, data attributes and tuples, and constructing Tan network based on the idea of machine learning classification algorithm, the data probability value is finally used to complete the classification and cleaning of inaccurate data. Experiments show that the model based on machine learning classification algorithm can effectively improve the imprecise data cleaning rate compared with the traditional model to solve multi-source heterogeneous imprecise data cleaning.

In order to fully utilize the features of multi-source heterogeneous data and effectively improve the accuracy and efficiency of fault diagnosis of rotating machinery, a multi-source heterogeneous data diagnosis method based on parameter collaborative optimization multi-scale convolutional autoencoder (MSCAE) is proposed. Firstly, multi-scale information learning is integrated into the convolutional autoencoder (CAE) to consider the temporal and spatial feature information of the diagnostic object simultaneously. To improve the training and diagnostic efficiency of MSCAE, a quantum particle swarm optimization (QPSO) module is used to perform hyperparameter optimization on it using chaos initialization and dynamic weight strategy (DWS). Besides, the sparse attention mechanism is introduced into the MSCAE model to improve the recognition rate of key fault features hidden in the original heterogeneous signals. Finally, the confusion matrix and visualization techniques are used to achieve fault classification. The experimental results demonstrate that after 100 experiments, the proposed method has an average diagnostic accuracy of 98.5 % and strong robustness to noise, providing a new method for rotating machinery fault diagnosis based on multi-source heterogeneous data.

The control and eradication of invasive alien plants in South Africa where about 10.1 million hectares of land has been invaded requires information about their extent and distribution. In this study it has been shown that in order to effectively and economically control invasive alien plants in the St. Lucia Heritage Site, the Working for Water organisation requires data on the extent and distribution of invasion not only of its operational areas but also of the surrounding areas. Information on extent and distribution requires the compiling data from multiple sources. Multi source data integration has been shown to be a troublesome exercise due to differences in methods in data acquisition, representation, coding, classification, scales, map projections, and mapping formats. Such data integration can only be made possible when organizations with a common goal adopt a common approach that would facilitate sharing data and information. In this study it has also been shown that multi-source alien species data integration and comparability of data amongst organizations was made possible through the adoption and successful implementation of data formats, data exchange, and user design standards. It has also been shown that mapping the extent and distribution of invasive alien plants requires the development of a geographic information system known as the Alien Plants Management Information System (APMIS). APMIS integrates data from multiple sources through the development and adoption of standards known as Geocode. Geocode is made up of the Land Mapping Unit (known as NBAL) and a date field so that both the spatial and temporal state of alien distribution and eradication can be recorded. This coding standard does not only indicate which location and time a species was found but also ensures comparability over time on how the distribution and eradication was progressing. The Geocode (NBAL + Date) can be very useful in modelling both the spatial and temporal variability of land mapping units and can be applicable in the areas of agricultural weed eradication programmes and in complex humanitarian environments where multiple organizations need to share data.

In medical domain, huge amounts of data are generated at all times. These data are usually difficult to access, with poor data quality and many data islands. Besides, with a wide range of sources and complex structure, these data contain essential information and are difficult to manage. However, few existing data management frameworks based on Data Lake excel in solving the persistence and the analysis efficiency for medical multi-source heterogeneous data. In this paper, we propose an efficient Multi-source Heterogeneous Data Lake Platform (MHDP) to realize the efficient medical data management. Firstly, we propose an efficient and unified method based on Data Lake to store data of different types and different sources persistently. Secondly, based on the unified data store, an efficient multi-source heterogeneous data fusion is implemented to effectively manage data. Finally, an efficient data query strategy is carried out to assist doctors in medical decision-making. In-depth analysis on applications shows that MHDP delivers better performance for data management in medical domain.

The complexity and diversity of equipments presented in the development of modern industrial technology brings the wide application of deep learning in fault diagnosis. However, the multi-source heterogeneous data collected from different sensors applied in industrial production cannot be used by traditional neural networks directly, which registers a difficulty to fault diagnosis based on deep learning. In addition, with only one of the data used, traditional deep learning fault diagnosis methods ignores the correlation between heterogeneous data, which may result in the loss of useful information and the accuracy of fault diagnosis. To solve the above problems, a fault diagnosis framework based on deep learning for multi-source heterogeneous data fusion is proposed. In the feature learning stage, stack self-encoder (SAE) and convolution neural network (CNN) are used to extract deep features from one-dimensional vibration data and two-dimensional image data respectively, and then feature fusion fault diagnosis is carried out. It solves the problem that traditional neural network can't make full use of multi-source heterogeneous data, and improves the accuracy of fault diagnosis. What's more, the bearing data of Case Western Reserve University are used to verify the validity of the diagnosis method proposed in this paper.

Can multi-source heterogeneous data improve the forecasting performance of tourist arrivals amid COVID-19? Mixed-data sampling approach

Exercise and training data is a core asset for enhancing military training and operational decision-making capabilities. Current military exercise and training data faces management challenges such as multi-source heterogeneity, inconsistent standards, and low data quality, which severely restrict its value realization. To address these challenges, this paper designs and implements a data governance system based on big data and data warehouse technologies. This system first constructs a data resource catalog and standard system, utilizing technologies like Flink and HDFS inotify to achieve the aggregation and integration of multi-source heterogeneous data. Through standardized ETL processes and a layered data warehouse, the data undergoes deep processing and organization. Finally, a unified data service and visualization platform supports upper-layer decision-making analysis. Application practices demonstrate that this system significantly improves data standardization, processing efficiency, and analytical capability, providing key technical support for the digital and intelligent transformation of military training.