Research on Distributed Anomaly Traffic Detection Technology Based on Hadoop Platform

In the past three decades, with the fast development of China's economy, the business data of electric power industry also showed rapid growth and the data size is continual expanding. Thus, the management systems of electric enterprise groups encountered the performance bottlenecks of massive data processing. Hadoop is an open-source software framework developed for reliable, scalable, and efficient distributed computing and storage, and is widely used in massive data processing. Based on the study of the proposed massive business data processing of electric industry, this paper introduces a novel architecture, which is based on Hadoop ecosystem, for management systems of electric enterprise groups to crack the present performance bottlenecks. First, the business logic and processing produce of management system is analyzed. Then new architecture with Hadoop based distributed processing model is introduced in detailed. At last, experimental results with actual business scenarios and data prove the architecture can effectively resolve present performance bottlenecks of massive business data processing, and greatly improve the performance and efficiency of the management system of electric enterprise groups.

The rapid development of Internet technology has brought the problem of information overload, and recommendation algorithm is put forward and considered to be the most effective way to solve the problem. Most of the traditional research about recommendation algorithm is focused on accuracy and diversity. However, in the practical engineering application, massive data process will be the most serious problem. In this paper, we propose a parallel recommendation algorithm based on mapreduce programming model, which runs on Hadoop platform, and in our system, we use mongodb as our auxiliary storage data. Finally, we give some experiments to prove our algorithm is suitable for processing massive data.

In order to fully use Internet of things to solve the agricultural fine production, fertilizer, fine and precise control, full traceability and other bottlenecks, and to solve the quality safety of agricultural products from the source and agriculture environmental pollution, a networking application system for modern agriculture is constructed, and networking intelligent gateway based on open source hardware is designed and developed, which realizes the video monitoring function based on motion detection. In addition, basic cloud platform system for modern agriculture network monitoring system is designed and achieved. Based on the RESTful interface service system provided by cloud platform, ExtJs client technology and WeChat re applied in the development and realization of the Demo system of an application layer. As a result, it shows part of application assumption of agricuture network monitoring system, and designs the big data processing and analysis module. What’s more, the Hadoop platform is used to achieve massive data processing produced by applications of Internet of things, and combined with machine learning technology, the corresponding model is established. It is concluded that the best solution is given such as crop variety selection, production and cultivation management and time to market.

The procedure of matching vehicle location data onto road map is very essential for many ITS (Intelligent Transportation System) applications. However, with the boosting deployment of GPS devices in vehicles, the accumulation of huge amount of GPS data caused great challenge on the efficiency and scalability of traditional serial map matching algorithm. In this paper we address the challenge by presenting a novel parallel map matching algorithm to realize high-performance processing of GPS data. The main idea is to adapt the serial map matching algorithm for cloud computing environment by reforming its' data-intensive or I/O-intensive computing stages using MapReduce paradigm. We implemented the algorithm in Hadoop platform and tested its performance by a large GPS dataset exceeds 120 billion GPS records. Experimental results show that our approach is highly efficient and scalable for massive historical GPS data processing.

High-speed network poses challenges on heavy traffic measurement and analysis, which becomes more serious due to the scale of modern data centers. Cloud platform is an alternative to crack the nut because it gives concurrent considerations to both high-performance computing and high-extensible data storage. Therefore, we use Hadoop to construct a trial cloud platform to tackle the challenges derived from the heavy traffic, and perform network traffic analysis and target packet dissection. Our testbed evaluates the performance against different processing strategies, and shows the statistic results. According to the experimental results, the Hadoop-based platform can fit to process massive traffic data very well, and it's appropriate to manage today's data centers with cloud computing services.

Image segmentation is a critical step in image analysis, and usually involves a high computational cost, especially when dealing with large volumes of data. Given the significant increase in the spatial, spectral and temporal resolutions of remote sensing imagery in the last years, current sequential and parallel solutions fail to deliver the expected performance and scalability. This work proposes a scalable and efficient segmentation method, capable of handling efficiently very large high resolution images. The proposed solution is based on the MapReduce model, which offers a highly scalable and reliable framework for storing and processing massive data in cloud computing environments. The solution was implemented and validated using the Hadoop platform. Experimental results attest the viability of performing region growing segmentation in the MapReduce framework.

This article delves into the communication quality monitoring system driven by big data under the C/S (client/server) architecture, aiming to provide a comprehensive, real-time, and efficient security monitoring solution for factory production and manufacturing environments. The system integrates LoRa wireless communication technology, big data processing and analysis technology, and innovative dynamic expansion algorithms to achieve real-time monitoring and rapid response to various safety hazards in the production environment. The core of the system lies in the construction of LoRa wireless communication network, which adopts an innovative umbrella shaped network structure to ensure stable and fast transmission of sensor monitoring data to the server. By using the STM32 development board with LoRa module, dynamic expansion of hardware integration modules has been achieved, enhancing the flexibility and scalability of the system. At the level of data transmission and processing, introducing Kafka distributed message queue as a data cache effectively alleviates the pressure of processing massive real-time data. Combining the Spark Streaming streaming data processing framework, a distributed data processing model was constructed to achieve real-time consumption and efficient parallel processing of messages in Kafka queues. At the same time, the designed dynamic extension algorithm model can automatically persist the data of new monitoring points, ensuring the system's rapid adaptation to data changes. In terms of data storage and visualization, a data persistence layer architecture for a big data platform has been constructed. Real time data, historical data, and log data are stored separately in Redis, MySQL, and HDFS systems through a data streaming model, improving data storage efficiency and system performance. Based on Tomcat network server and SSM architecture, a B/S structured web server visualization platform has been developed, which supports users to remotely query the security status of the production environment through computer or mobile browser, and receive alarm notifications in case of abnormalities. The system integration test results show that the communication quality monitoring system performs excellently in wireless communication quality, data processing speed, data storage efficiency, and data visualization, and can meet the complex monitoring needs of factory production and manufacturing environments, providing solid technical support for enterprise safety production. However, there is still room for improvement in real-time data collection loss rate, alarm function completeness, and mobile access experience. Future research will focus on optimizing data transmission software models, adding multi-channel alarm functions, and developing mobile apps to enhance user experience.

There is a growing opportunity for self-improvement within the realm of higher education, represented by the combination of big data technologies and higher education. The Education Big Data Processing Framework (EBDPF) is the name of the platform that is the subject of this article, which focuses on the development of an advanced big data processing platform. In order to process huge datasets in an effective manner, the platform makes use of Hadoop's big data storage architecture, in addition to Hive, Flume, and Sqoop for data collection and synchronization, respectively. In this study, the implementation of standard data mining techniques is investigated through the use of MapReduce programming. More specifically, the study investigates the execution efficiency and scalability of these algorithms within the Hadoop platform. The data clustering task in data mining is chosen as a representative problem in this study so that the effectiveness of the EBDPF may be evaluated accordingly. An implementation of the clustering job that is based on MapReduce is created and tested on the Hadoop platform. Extensive experiments are conducted with a range of cluster sizes and data sizes. The findings provide more evidence that Hadoop distributed systems are more efficient and successful than traditional methods when it comes to managing data mining applications. The extended performance analysis of computing power not only provides evidence that the platform is currently effective, but it also provides a signal that it has a significant amount of potential that has not yet been utilized. At the convergence of big data technology and higher education, the EBDPF emerges as a strong and promising framework, demonstrating considerable steps in accelerating data processing within educational contexts. In conclusion, the EBDPF is a framework that demonstrates significant progress.

SummaryBrain data processing has been embracing the big data era driven by the rapid advances of neuroscience as well as the experimental techniques for recording neuronal activities. Processing of massive brain data has become a constant in neuroscience research and practice, which is vital in revealing the hidden information to better understand the brain functions and malfunctions. Brain data are routinely non‐linear and non‐stationary in nature, and existing algorithms and approaches to neural data processing are generally complicated in order to characterize the non‐linearity and non‐stationarity. Brain big data processing has pressing needs for appropriate computing technologies to address three grand challenges: (1) efficiency, (2) scalability and (3) reliability. Recent advances of computing technologies are making non‐linear methods viable in sophisticated applications of massive brain data processing. General‐purpose Computing on the Graphics Processing Unit (GPGPU) technology fosters an ideal environment for this purpose, which benefits from the tremendous computing power of modern graphics processing units in massively parallel architecture that is frequently an order of magnitude larger than the modern multi‐core CPUs. This article first recaps significant speed‐ups of existing algorithms aided by GPGPU in neuroimaging and processing electroencephalogram (EEG), functional magnetic resonance imaging (fMRI), magnetoencephalography (MEG) and etc. The article then demonstrates a series of successful approaches to processing EEG data in various dimensions and scales in a massively parallel manner: (1) decomposition: a massively parallel Ensemble Local Mean Decomposition (ELMD) algorithm aided by GPGPU can decompose EEG series, which forms the basis of further time‐frequency transformation, in real‐time without sacrificing the precision of processing; (2) synchronization measurement: a parallelized Nonlinear Interdependence (NLI) method for global synchronization measurement of multivariate EEG with speed‐up of more than 1000 times, and it was successful in localization of epileptic focus; and (3) dimensionality reduction: a large‐scale Parallel Factor Analysis which excels in run‐time performance and scales far better by hundreds of times than conventional approach does, and it supports fast factorization of EEG with more than 1000 channels. Through these practices, the massively parallel computing technology manifests great potentials in addressing the grand challenges of brain big data processing. Copyright © 2016 John Wiley & Sons, Ltd.

Possibilities of transfer process data from PLC to Cloud platforms based on IoT

In this paper, a communication scheme of cloud platform for the lower limb exoskeleton robot is proposed. Firstly the wireless communication protocol Zigbee is used to build a sensor network to collect and summarize sensors' data of users' physiological information and robot movement. Secondly, Message Queuing Telemetry Transport (MQTT) is used to transmit the data into the cloud platform for data preservation and processing. Finally, the cloud platform is accessed remotely to obtain the physiological information of the user and robot movement based on the HyperText Transfer Protocol (HTTP). Experiments indicate its effectiveness and practicability of the proposed communication scheme to present sensors data to users or doctors in Web pages.


 
 
 
 Apriori algorithm is a classical algorithm of association rule mining and widely used for generating frequent item sets. However, the original Apriori algorithm has some limitation such as it needs to scan the dataset many times to discover all frequent itemset and generate huge number of candidate itemset. To overcome these limitations, researchers have made a lot of improvements to the Apriori such as candidate generation, without candidate generation, transaction reduction, partitioning, and sampling. When it comes to mine massive data, these algorithms failed to prove efficiency because limitation of the processing capacity, storage capacity, and main memory constraints. Therefore, parallel and distributed algorithms are developed to perform large-scale computing in ARM on multiple processors. However, the problems with most of the parallel and distributed framework are overheads of managing distributed system, lack of high level parallel programming language, and node failures. Hadoop-MapReduce is an efficient, scalable, and simplified programming model for massive data processing and it also available on cloud environment. Cloud computing offers huge computing resources, and capacities to solve big data challenges. Recently many parallel algorithms have been proposed on Hadoop-MapReduce to enhance the performance of Apriori algorithm but there are some drawbacks: since multiple scan over the dataset is needed to generate candidate itemset, it consume more execution time. The aim of this study is to propose a parallel Transaction Reduction MapReduce Apriori algorithm (TRMR-Apriori) which is reduce unnecessary transaction values and transactions from the dataset in parallel manner to overcome above problems. The experiments show that TRMR-Apriori is able to achieve better execution time to discover frequent itemset those of previous sequential ARM algorithms such as Apriori, AprioriTid, Eclat, and FP-Growth and the previous parallel algorithms such as PApriori, MRApriori, and Modified Apriori with different condition on homogeneous computing environment using Hadoop-MapReduce platform in cloud. Overall, the TRMR-Apriori shows the strength to extract the frequent itemset from massive dataset in cloud.
 
 
 
 

Today, centralized data-centers or clouds have become the de-facto platform for data-intensive computing in the commercial, and increasingly, scientific domains. This is because clouds such as Amazon AWS and Microsoft Azure offer large amounts of monetized co-located computation and storage well suited to typical processing tasks such as batch analytics. However, many Big Data applications rely on data that is geographically distributed, and is not collocated with the centralized computational resources provided by clouds. Examples of such applications include analysis of user data such as blogs, video feeds taken from geographically separated cameras, monitoring and log analysis of server and content distribution network (CDN) logs, and scientific data collected from distributed instruments and sensors. Such applications lead to a number of challenges for efficient data analytics in today's cloud platforms. First, in many applications, data is both large and widely distributed and data upload may constitute a non-trivial portion of the execution time. Second, centralized cloud resources present a single point of failure and network partitions between the data sources and the cloud can also lead to service disruptions. Third, the cost to transport, store, and process data may be outside of the budget of the small-scale application designer or end-user. The paper present Nebula: a dispersed edge cloud infrastructure that provides both computation and data storage to address the above challenges. The use of edge resources is attractive for several reasons. First, there is an increasing amount of computing and storage resources available on the edge, as evidenced by their use in several volunteer computing, filesharing as well as content delivery (CDN) environments. This capacity is likely to increase further with the provision of powerful multi-core, multi-node desktop and home machines coupled with increasing amount of high bandwidth Internet connectivity. Second, edge resources provide locality to data and users naturally, and hence, can be exploited easily for insitu processing. Finally, if cost is an issue, then volunteer edge resources can be utilized at a relatively low cost.

The importance of advancing intelligence into operational systems—such as intelligent transportation, smart energy, and intelligent supply chain systems—cannot be overstated. Such systems manage demanding physical processes for real-time operations, yet their intelligence has typically been limited to business systems that operate in time frames of seconds to days. Intelligent Operational Systems aim to combine event-driven, real-time data processing with external AI and machine learning for high-frequency, real-time decisions and high-sensitivity supervised learning. To date, the concept has largely been explored using on-premises, private-cloud infrastructure. The analysis here focuses on Cloud Platforms for Intelligent Operational Systems, with an emphasis on definitions, architectural patterns, supporting cloud platform paradigms, enabling data provisioning and processing, cloud support for AI and ML, and key components of reliability, scalability, and availability. The term cloud provides strong intuitive guidance about what is possible from a cloud platform. That is, the service abstraction of Infrastructure as a Service (IaaS) enables the hosting of compute and storage resources in the cloud, which can then be capitalized, instrumented, and scaled to improve developer productivity and reliability. When resources are hosted on IaaS, the Cloud provider is responsible for deploying, managing, and operating the software packages. Platform as a Service (PaaS) provides an even more compelling operational product abstraction with benefits from both Execution as a Service (EaaS) and Software as a Service (SaaS)—lower operational costs and reduced software management overhead.

There are nearly 50 million AIS data generated every day in the Yangtze River Basin. Efficient storage and query of AIS data become the key problem of smart maritime administration. To overcome this problem, a distributed storage and query scheme based on Hadoop platform is proposed. Firstly, the original AIS message is parsed, the abnormal data is removed and the parsed data is pushed to the Kafka message queue. Secondly, the messages in the queue are pushed to different servers to complete the storage of real-time cache, historical trajectory and regional information. Finally, the query interface is provided to realize the real-time query of ship status, trajectory and regional ship in a certain period of time. The real applications show that compared with the traditional relational database, the solution can meet the requirements of real-time query and has high scalability.