Containerized Scheduling Method Based on Kubernetes and YARN in Big Data Scenarios

For such difficulties as low mining efficiency and large amount of data that the data mining of residential electricity data has to be faced with, the research on improved K-means algorithm and cloud computing-based analysis on massive data of power utilization is performed. Firstly, as the initial cluster center and the value K is difficult to determine in traditional K-means algorithm, an improved K-means algorithm based on density is proposed. Defining the product of sample density, the reciprocal of the average distance between the samples in the cluster, and the distance between the clusters as weight product, the initial center is determined successively according to the maximum weight product method and the accuracy of the clustering is improved. Then, the parallelization of improved K-means algorithm based on MapReduce model is realized and the efficiency of clustering is improved. Finally, the mining experiment of massive power utilization data is carried out on basis of 400 households' electricity data. Taking family as unit, calculate such features as electricity consumption rate during peak hour, load rate, valley load coefficient and the percentage of power using during normal hour, and establish the data vector dimension, completing the clustering of similar user types, while analyzing the behavioral characteristics of each types of users. Experiment results on Hadoop cluster show that the improved algorithm can be stable and efficient operated and it can achieve better clustering effect.

With the continuous construction of intelligent power grids, a large amount of data on power grid operation, production management and market operation have been accumulated in power systems, which are affected by factors such as collection, communication, and environment. There are quality problems in terms of integrity, consistency, and effectiveness of data. In this paper, by studying the characteristics of massive data and data anomaly identification method, a method of error correction for massive data in power grid based on arbitrary partition area is proposed. Taking abnormal data as the center and dividing the area based on the network topology structure, the abnormal data of the network is corrected layer by layer to achieve the purpose of improving the data quality.

Overlay analysis is a common task in geographic computing that is widely used in geographic information systems, computer graphics, and computer science. With the breakthroughs in Earth observation technologies, particularly the emergence of high-resolution satellite remote-sensing technology, geographic data have demonstrated explosive growth. The overlay analysis of massive and complex geographic data has become a computationally intensive task. Distributed parallel processing in a cloud environment provides an efficient solution to this problem. The cloud computing paradigm represented by Spark has become the standard for massive data processing in the industry and academia due to its large-scale and low-latency characteristics. The cloud computing paradigm has attracted further attention for the purpose of solving the overlay analysis of massive data. These studies mainly focus on how to implement parallel overlay analysis in a cloud computing paradigm but pay less attention to the impact of spatial data graphics complexity on parallel computing efficiency, especially the data skew caused by the difference in the graphic complexity. Geographic polygons often have complex graphical structures, such as many vertices, composite structures including holes and islands. When the Spark paradigm is used to solve the overlay analysis of massive geographic polygons, its calculation efficiency is closely related to factors such as data organization and algorithm design. Considering the influence of the shape complexity of polygons on the performance of overlay analysis, we design and implement a parallel processing algorithm based on the Spark paradigm in this paper. Based on the analysis of the shape complexity of polygons, the overlay analysis speed is improved via reasonable data partition, distributed spatial index, a minimum boundary rectangular filter and other optimization processes, and the high speed and parallel efficiency are maintained.

A polymorphic heterogeneous security architecture for edge-enabled smart grids

In the process of data integration, medical data is a huge obstacle to the development of medical information because of its complicated data type, large amount of data and semantic heterogeneity among different data sources. Integration of massive medical heterogeneous data is the urgent need to promote the process of medical information to be solved. According to the characteristics of medical data sources, an ontology-based medical heterogeneous data integration scheme is proposed. By introducing the medical similarity algorithm into the hybrid medical ontology, not only the multi-semantic problem is solved, but also the current heterogeneity of mass medical data is eliminated. Finally, experiments show that this method performs well in eliminating the semantic uncertainty of data in the medical field and can improve the integration efficiency of heterogeneous data in the medical data warehouse. It is practically feasible to solve the problem of medical heterogeneous data integration.

Power grid operation and maintenance decision-making reform is an important part of power system reform. With the construction of massive historical quasi real-time data management platform, the reform of power system is also advancing. However, in the face of massive data explosion, the business level and business logic become disorganized and redundant. Based on the actual situation of Shenzhen Power Supply Bureau, the sg-erp data center is composed of structured data center, massive data center, unstructured data center and power grid GIS data center. With the unprecedented growth of business application data, the data center can improve business logic and promote power system reform. The experimental results show that big data technology has a broad application prospect in the reform of power industry.

With the construction and development of industrial informatization, industrial big data has become a trend within the smart industry. To obtain valuable information on massive data, achieving the acquisition, storage, analysis, and mining is becoming an important area of research. Focusing on the application requirements for industrial fields, we propose a data acquisition and analysis system based on the NB‐IoT for industrial applications. The system is an integrated system that includes sensor data acquisition, data transmission, data storage, and analysis mining. In this study, we mainly focused on the use of the NB‐IoT network to collect and transmit real‐time data for sensors. First, for the long time series (e.g., if we collect the data streams for one year for the sensor with a frequency of 1 Hz, the length of the series will reach 107). Then, we propose DSCS‐LTS, a distributed storage and calculation model, and CCCA‐LTS, an algorithm for the correlation coefficient of long time series in a distributed environment. Third, we propose a granularity selection algorithm and query process logic for visualization. We tested the platform in our laboratory and an automated production line for one year, and the experimental results using real data sets show that our approach is effective and scalable, can achieve efficient data management, and provide the basis for intelligent enterprise decision‐making.

Abstract. Natural disasters have a significant impact on the environment and economies of all countries around the world, and a large amount of multi-source heterogeneous geographic information data is generated every day. However, due to a lack of knowledge transformation capabilities, these nations continue to struggle with the issue of "a large amount of data and little knowledge". Therefore, it is of great significance how to extract geographic knowledge related to disasters from the vast data and construct a geographic knowledge graph integrating disaster information. Based on the theory related to knowledge extraction, this paper proposes a method to construct a natural disaster knowledge graph integrating geographic information. The core of this knowledge graph is to construct the association relationship between natural disaster concepts, research areas, and spatial data. The vocabulary and relationships associated with disaster concepts are primarily transformed by an existing word list of geographic narratives, which then provide rich semantic relationships of domain concepts for the entire knowledge graph. The research areas and spatial data types are mainly obtained through knowledge entity extraction and disambiguation methods. This disaster knowledge graph can support applications well such as natural disaster visualization and analysis, data recommendation systems, and intelligent Q&A systems, which can further improve the intelligence of natural disaster knowledge services and is expected to promote the sharing and reuse of domain knowledge graphs to a certain extent.

In order to solve the low prediction accuracy and prediction rate of traditional time series prediction algorithm in the analysis of massive data, according to the characteristics of communication network performance index, time series prediction algorithm based on the big data method is proposed. In addition, based on the traditional three components of time series feature extraction, the concept of emergency component is introduced, and the outlier detection and processing as well as prediction analysis are carried out on the basis of the extraction results. The results show that the algorithm based on big data, through the analysis, fitting, modelling, and prediction of massive data, smaller granularity decomposition of the time series value is conducted, which significantly improved the credibility and accuracy of prediction. Based on the above findings, it is summarized that the time series prediction algorithm has good performance.

With the continuous development of power grids, power cables are widely used in urban power grids. But cables are generally laid underground, and the fault is not easy to be found and the recovery time is long, the reliability of power supply was seriously threatened. The traditional planned maintenance has not been able to solve the existing problems accurately. Therefore, it is of great significance to improve the reliability of power distribution and reduce the failure rate by assessing the operational status of power cables through the relevance analysis of massive monitoring and testing data. This paper proposes a power cable state assessment algorithm based on association rule data mining technology from the perspective of big data and information relevance. Firstly, by analyzing the online monitoring data, detection data, experimental data, and the interrelation of the state parameters of the cable and transformer fault types, the set of cable synthetic status parameters can be set up. Then, use the association rule method to calculate the weight coefficient of each state parameters; In order to effectively avoid the influence of subjective factors on the evaluation results, weighted theory of equilibrium function was used to calculate the weight coefficient of the synthetic status parameters; Finally, combined with the existing assessment process, an accurate and objective evaluation system power cable was established. The results of the example verify the validity and accuracy of the assessment method, which provides a new novel idea for cable status assessment.

A Brief Introduction on Big Data 5Vs Characteristics and Hadoop Technology

Next-generation sequencing (NGS) techniques produce high quantities of DNA and RNA sequencing data, enabling variation analysis at a whole genome level. Due to the massive availability of NGS data, the processing and analysis now constitute a serious challenge for research in life sciences. This necessitates the development of novel, user-friendly and scalable software tools, which are able to jointly handle large NGS data sets in order to break the bottleneck that has already moved from data generation to data analysis in high-throughput biology. Here we introduce a highly efficient NGS analysis and visualization software tool (Rikurator), which is designed for researchers who are struggling with NGS data analysis and may not have access to a dedicated bioinformatics team. Rikurator can be applied to a multitude of genomic analysis tasks, including discovery of pathogenic mutations in familial or sporadic sample sets, analysis of structural variants and somatic characterization of cancer genomes. To these ends the software implements interactive variant filtering and annotation methods based on sample comparison, control data (e.g. ExAC), amino acid changes and quality scores, which provide an intuitive view at the studied samples. Shared candidate genes in a set of dozens of individuals can be discovered in a matter of minutes with an ordinary laptop computer. Importantly, the software is able to analyze hundreds of whole-genome sequenced samples using only a modest amount of RAM. The tool also visualizes and integrates genes, variants (VCF file format), sequence reads (BAM) and user defined features (BED/GFF). In addition, Rikurator is not limited to human studies, supporting reference genomes and genome annotations in standard formats. The software supports targeted (e.g. exome), whole-genome, RNA, Oxford Nanopore and PacBio sequencing data. Rikurator can be run on Windows, Linux and Mac, and it will be freely available after publication. Citation Format: Riku Katainen, Veli Mäkinen, Lauri A. Aaltonen, Esa Pitkänen. Fast and scalable software for comparative variant analysis and visualization of massive next-generation sequencing data. [abstract]. In: Proceedings of the 107th Annual Meeting of the American Association for Cancer Research; 2016 Apr 16-20; New Orleans, LA. Philadelphia (PA): AACR; Cancer Res 2016;76(14 Suppl):Abstract nr 5281.

This paper presents two complementary statistical computing frameworks that address challenges in parallel processing and the analysis of massive data. First, the <b>foreach</b> package allows users of the R programming environment to define parallel loops that may be run sequentially on a single machine, in parallel on a symmetric multiprocessing (SMP) machine, or in cluster environments without platform-specific code. Second, the <b>bigmemory</b> package implements memory- and file-mapped data structures that provide (a) access to arbitrarily large data while retaining a look and feel that is familiar to R users and (b) data structures that are shared across processor cores in order to support efficient parallel computing techniques. Although these packages may be used independently, this paper shows how they can be used in combination to address challenges that have effectively been beyond the reach of researchers who lack specialized software development skills or expensive hardware.

Federated Graph Learning (FGL) enables the aggregation of a large amount of big data while preserving the local privacy of each client, facilitating the collaborative training of efficient graph models. However, existing research often overlooks the issue of fairness in FGL. Most studies assume that clients are always willing to participate in collaborative training, but in practice, clients are often hesitant to engage in collaborations where the rewards do not reflect their contributions. This reluctance arises because such collaborations consume local resources and may expose sensitive information. To address this challenge, this paper proposes a novel fair federated graph learning algorithm. By introducing a new client contribution evaluation method, the algorithm leverages approximate Shapley Value(SV) calculations and prototypes to more efficiently and comprehensively assess each client's contribution during every training round. Given the heterogeneous nature of big data in FGL, accurately evaluating contributions is crucial for maintaining fairness. Clients with varying levels of contributions are rewarded accordingly to ensure fairness in the collaborative learning process. Additionally, the proposed method incorporates the structural information of the graph to effectively enhance the accuracy of the overall model, which is particularly beneficial in big data scenarios where graph relationships play a significant role. The approach is also versatile and applicable in different training environments. Extensive experiments demonstrate that our algorithm achieves promising results in terms of both model accuracy and fairness.

New generation of health care IT systems are collecting and storing more and more data of patients. Useful knowledge can be extracted from the data in EMR or PHR to provide medical advises to patients, while through data analysis the result statistics can be used to support the scientific research. However, RDBMSs-based framework is not able to support the requirements of massive health care data storage, management and analysis. To solve the problem, this paper proposes a massive data management and analysis solution based on Hadoop to archive better performance, scalability and fault tolerance. The data management framework is presented. Besides, 2 different data analysis methods based on MapReduce and Hive are proposed. Experiment results of data upload, data query and data analysis show that the performance of the proposed framework is greatly improved, and a brief summary of the performance and the differences between 2 methods of MapReduce and Hive is also discussed.