Implementation of massive data processing architecture for electric enterprise groups

Cloud security is one of the issues that cloud platform needs to focus on. Cloud platform has powerful computing ability and storage resources, which makes it become the target of hackers. Therefore, in addition to the traditional network security configuration, it is necessary to adopt a more complete security defense measure to protect the data processing platform. In this paper, it proposes a distributed anomaly traffic detection technology based on classifier combination according to the characteristics of massive network data processing in Hadoop platform, which can improve the security of massive network data processing platform.

This article delves into the communication quality monitoring system driven by big data under the C/S (client/server) architecture, aiming to provide a comprehensive, real-time, and efficient security monitoring solution for factory production and manufacturing environments. The system integrates LoRa wireless communication technology, big data processing and analysis technology, and innovative dynamic expansion algorithms to achieve real-time monitoring and rapid response to various safety hazards in the production environment. The core of the system lies in the construction of LoRa wireless communication network, which adopts an innovative umbrella shaped network structure to ensure stable and fast transmission of sensor monitoring data to the server. By using the STM32 development board with LoRa module, dynamic expansion of hardware integration modules has been achieved, enhancing the flexibility and scalability of the system. At the level of data transmission and processing, introducing Kafka distributed message queue as a data cache effectively alleviates the pressure of processing massive real-time data. Combining the Spark Streaming streaming data processing framework, a distributed data processing model was constructed to achieve real-time consumption and efficient parallel processing of messages in Kafka queues. At the same time, the designed dynamic extension algorithm model can automatically persist the data of new monitoring points, ensuring the system's rapid adaptation to data changes. In terms of data storage and visualization, a data persistence layer architecture for a big data platform has been constructed. Real time data, historical data, and log data are stored separately in Redis, MySQL, and HDFS systems through a data streaming model, improving data storage efficiency and system performance. Based on Tomcat network server and SSM architecture, a B/S structured web server visualization platform has been developed, which supports users to remotely query the security status of the production environment through computer or mobile browser, and receive alarm notifications in case of abnormalities. The system integration test results show that the communication quality monitoring system performs excellently in wireless communication quality, data processing speed, data storage efficiency, and data visualization, and can meet the complex monitoring needs of factory production and manufacturing environments, providing solid technical support for enterprise safety production. However, there is still room for improvement in real-time data collection loss rate, alarm function completeness, and mobile access experience. Future research will focus on optimizing data transmission software models, adding multi-channel alarm functions, and developing mobile apps to enhance user experience.

SummaryBrain data processing has been embracing the big data era driven by the rapid advances of neuroscience as well as the experimental techniques for recording neuronal activities. Processing of massive brain data has become a constant in neuroscience research and practice, which is vital in revealing the hidden information to better understand the brain functions and malfunctions. Brain data are routinely non‐linear and non‐stationary in nature, and existing algorithms and approaches to neural data processing are generally complicated in order to characterize the non‐linearity and non‐stationarity. Brain big data processing has pressing needs for appropriate computing technologies to address three grand challenges: (1) efficiency, (2) scalability and (3) reliability. Recent advances of computing technologies are making non‐linear methods viable in sophisticated applications of massive brain data processing. General‐purpose Computing on the Graphics Processing Unit (GPGPU) technology fosters an ideal environment for this purpose, which benefits from the tremendous computing power of modern graphics processing units in massively parallel architecture that is frequently an order of magnitude larger than the modern multi‐core CPUs. This article first recaps significant speed‐ups of existing algorithms aided by GPGPU in neuroimaging and processing electroencephalogram (EEG), functional magnetic resonance imaging (fMRI), magnetoencephalography (MEG) and etc. The article then demonstrates a series of successful approaches to processing EEG data in various dimensions and scales in a massively parallel manner: (1) decomposition: a massively parallel Ensemble Local Mean Decomposition (ELMD) algorithm aided by GPGPU can decompose EEG series, which forms the basis of further time‐frequency transformation, in real‐time without sacrificing the precision of processing; (2) synchronization measurement: a parallelized Nonlinear Interdependence (NLI) method for global synchronization measurement of multivariate EEG with speed‐up of more than 1000 times, and it was successful in localization of epileptic focus; and (3) dimensionality reduction: a large‐scale Parallel Factor Analysis which excels in run‐time performance and scales far better by hundreds of times than conventional approach does, and it supports fast factorization of EEG with more than 1000 channels. Through these practices, the massively parallel computing technology manifests great potentials in addressing the grand challenges of brain big data processing. Copyright © 2016 John Wiley & Sons, Ltd.

E-commerce has been one critical component of information society now; however credit is becoming a bottleneck for its further development. Therefore Shanghai is building a third-party e-commerce credit evaluation platform. The platform will be integrated with many external systems to collect massive credit-related data, do scientific evaluation and provide users with personalized credit services. In order to deal with mass data and provide personalized services to tenants of the platform, a cloud computing model of Shanghai e-commerce credit evaluation platform is designed in this paper. In PaaS layer, we propose a mass data processing technology based on the XML metadata specification; in SaaS layer, we propose a mixed model of data storage and a model of areas-layered tenant customization. Experiments show that these technologies and models can improve the data processing efficiency and resource utilization greatly.

시장의 변화 및 소비자의 요구 변화를 비롯한 기업 내외부의 상황변화에 대응해서 얼마나 빠르게 적응할 수 있는가 하는 것이 실시간 기업의 핵심요건이다. 이러한 실시간 기업이 가진 변화의 속도를 지원하기 위해서 최근 Big Data 처리 기술이 각광받고 있다. 특히 최근 유무선 통신망의 진화 및 고도화가 가속되고 있는 상황에서 대규모 통신 트래픽을 실시간으로 처리하여 안정된 서비스를 제공하는 것과 강력한 보안 관제 기능은 매우 필요하다. 따라서 본 논문은 클라우드 컴퓨팅 기반의 Big Data처리기술을 활용하여 통신 사업자들이 갖고 있는 경영상의 문제점을 해결하고 효과적인 통신망 관리 시스템의 운영에 관한 연구를 진행한다. 【Real-Time Enterprise (RTE)'s key requirement is that it should respond and adapt fast to the change of the firms' internal and external situations including the change of market and customers' needs. Recently, the big data processing technology to support the speedy change of the firms is spotlighted. Under the circumstances that wire and wireless communication networks are evolving with an accelerated rate, it is especially critical to provide a strong security monitoring function and stable services through a real-time processing of massive communication data traffic. By applying the big data processing technology based on a cloud computing architecture, this paper solves the managerial problems of telecommunication service providers and discusses how to operate the network management system effectively.】

Data intensive applications are now ubiquitous in many fields, e.g. high-energy physics, astronomy, climate modeling, and the geosciences. Especially for spatial data processing in the distributed platforms, the parallel tasks are generally required to handle massive spatial datasets and usually treated as data-intensive applications. In order to address the challenge of massive spatial data processing, we propose a hypergraph based tasks scheduling strategy on a master-slave platform. Our task scheduling strategy involves two consecutive steps: mapping and scheduling. In the mapping process, we formulate a hypergraph partition model to decide which tasks will be executed by each slave processor. At the same time, the scheduling process determines the execution order of selected tasks and the order in which the master transfers the files to the slaves. An experiment based on the GridSim toolkit was conducted to evaluate and compare our scheduling algorithm with Min-min heuristic. Our experimental results show that our scheduling strategy outperforms the Min-min heuristic.

With the development of information technology in Colleges and universities, the traditional data integration scheme has been unable to meet the needs of colleges and universities in dealing with massive unstructured and semi-structured data and various types of structured data integration and fusion analysis. Based on this, this paper uses MPP and Hadoop to build university data platform, which can not only meet the needs of deep analysis, complex query and rapid response of structured data, but also meet the performance requirements of massive unstructured data in storage, loading, conversion and other aspects. At last, the paper selects the log data of a university’s online behavior, carries out data storage, preprocessing and analysis query, verifies the performance advantages of the platform in massive data processing, and provides a new idea and method for the construction of university data platform.KeywordsHadoopMPP databaseHybrid architectureData analysis

Healthcare is the diagnosis, treatment, and prevention of disease, illness, injury, and other physical and mental impairments in human beings. Healthcare is delivered by practitioners in allied health, dentistry, midwifery-obstetrics, medicine, nursing, optometry, pharmacy, psychology and other care providers. In healthcare service to provide better performance to identify some diseases like blood pressure, skin cancer, diabetes etc. with the sensor devices those are connected with home server by using wireless sensor or wired network policies. The size of data sets being collected and analyzed in the industry for healthcare business intelligence is growing rapidly. In healthcare service case we have to concern about massive data services. To store and process huge amount of unstructured data by using traditional database system is so difficult. To alleviate this drawback we can use Hadoop architecture that has functionality to store and process huge amount of unstructured data. Hadoop Distributed file system (HDFS) can store and the Hadoop MapReduce framework process huge amount of unstructured data that enables easy development of scalable parallel applications. This paper's main motivation is to ensure better data transmission, data reliability and massive data processing with High Availability (HA) based network storage solution. So, in this paper we proposed MapReduce Agent (MRA) to process Big-Data and also proposed iSCSI Protocol adapted Network Attached Storage (NAS) system for healthcare service.

Spatiotemporal self-supervised predictive learning for atmospheric variable prediction via multi-group multi-attention

The rapid development of Internet technology has brought the problem of information overload, and recommendation algorithm is put forward and considered to be the most effective way to solve the problem. Most of the traditional research about recommendation algorithm is focused on accuracy and diversity. However, in the practical engineering application, massive data process will be the most serious problem. In this paper, we propose a parallel recommendation algorithm based on mapreduce programming model, which runs on Hadoop platform, and in our system, we use mongodb as our auxiliary storage data. Finally, we give some experiments to prove our algorithm is suitable for processing massive data.

Analysis of normal traffic flow usually uses the static or dynamic model based on fluid mechanics. However, in the analysis process, the problem of massive modeling and data handling exists, and the accuracy is desirable. Finite element method has been widely applied in various domains. Based on existing theory of traffic flow, ITS and the development of finite element, the paper presents a simulation method based on finite element for solving problems in traffic flow. Using existing finite element analysis software, the traffic flow is simulated and analyzed with fluid mechanics and dynamics. The problem with massive data processing through manual modeling and Numerical Analysis is solved. Finally, a typical case of special road is analyzed with ADINA finite element analysis software.

The data in the power Internet of Things has the characteristics of multiple types, large amounts of data, and obvious heterogeneity. In order to solve the storage and scheduling problem of massive heterogeneous data in the power grid, this paper designs a hybrid heterogeneous data storage scheduling architecture based on container technology. In order to solve the problem of sensor data analysis with unknown types and dynamic changes, this paper uses the K-Means clustering method to propose a data analysis process based on the idea of unsupervised learning to realize the analysis and adaptation of massive sensor data. In addition, in order to improve scheduling efficiency in big data scenarios, a two-layer scheduling architecture based on YARN and Kubernetes is proposed to achieve unified scheduling at the task level and node level, thereby improving cluster resource utilization. The heterogeneous data storage scheduling architecture proposed in this paper combines the container technology to meet the needs of massive data storage and processing for low latency and improved interrupt response rate and can be applied to power Internet of Things scenarios.KeywordsBig dataTask schedulingContainerKubernetesYARN

This paper studies a novel approach to processing massive uncertain graph data. In this approach, we propose a new framework to simultaneously process a query on a set of randomly sampled possible worlds of an uncertain graph. Based on this framework, we develop a series of algorithms to analyze massive uncertain graphs, including breadth-first search, shortest distance queries, triangle counting, and core decomposition. We implement this approach based on GraphLab, one of the stateof-the-art graph processing frameworks. By sharing fine-grained internal processing steps on common substructures of sampled possible worlds, the new approach achieves tens to hundreds of times speedup in execution time on a cluster of 20 servers.

With the rapid development of digital technology, from the application of traditional databases and scientific computing to the emerging cloud computing services, the analysis and processing of massive data has become the focus of society. Providing low-cost, scalable, and configurable shared cloud services to users on cloud service platforms is a new hotspot for the development of major cloud service providers. Job scheduling plays an important role in improving the overall system performance of cloud service capabilities. Simple job scheduling strategies (such as Fair and FIFO scheduling) do not consider job size and may degrade performance when jobs of different sizes arrive. This paper proposes the MQWAG (Multi-queue Load-Sensitive Greedy Scheduling Algorithm) job scheduling algorithm to reorder multi-queue jobs so that short jobs are executed preferentially in multiple queues. In our experiments, our algorithm shortened the average job completion time by about 26% compared with other algorithms.

Dispersion-based surface wave prediction and suppression methods have obtained more and more attention in recent years. Currently the main problem of this type of method is that the dispersion spectrum accuracy is not high enough leading to inaccurate surface wave prediction, or the dispersion spectrum measurement is precise but the calculation is too time-consuming. We introduce an improved data-derived surface wave prediction and adaptive attenuation method to overcome those problems. With the help of adaptive surface wave data extraction technology, offset-varying dispersive wave reconstruction method, and noise model matching subtraction, we can accurately predict the surface wave noise, and make this procedure effectively apply in the processing of massive data. Through the test of actual data, this method can effectively remove the surface wave, protect the effective signal to the maximum extent, and has a wide application prospect in the area with severe surface wave noise.