NetCube: a comprehensive network traffic analysis model based on multidimensional OLAP data cube

With the complexity and heterogeneity of multi-level networks in energy storage systems (such as station control layer equipment, energy storage compartment battery monitoring networks, etc.), traditional traffic monitoring technologies are unable to effectively cope with challenges such as encrypted traffic covert attacks and dynamic changes in business scenarios. This article proposes a full traffic intelligent analysis model for complex heterogeneous networks, which integrates multi granularity traffic modeling and deep behavior analysis to achieve fine-grained monitoring of network layer and application layer traffic. Firstly, a network traffic baseline model based on spatiotemporal features is constructed, combined with unsupervised learning to dynamically identify traffic patterns in business scenarios; Secondly, design an encrypted traffic metadata association analysis algorithm to detect Advanced Persistent Threat (APT) malicious heartbeat connections without cracking Secure Sockets Layer/Transport Layer Security (SSL/TLS); Finally, the performance of the model is validated based on real energy storage system traffic and simulation attack data. The experiment shows that the model has a detection accuracy of 96.8% for dangerous control instructions, an F1 score of 93.7% for identifying abnormal external communication behavior of encrypted flow, and a false alarm rate of less than 2.5%, which is significantly better than traditional methods. It provides an innovative solution for full flow safety monitoring of energy storage systems.

With the rapid increasing of network scale, the size of traffic data also expands a lot. In traditional traffic data analysis, there are some problems, such as high computation complexity, low analysis efficiency, long learning period, and difficulty of development. To address these problems, we design and implement a GPU-accelerated parallel analysis scheme for network traffic - EasyAnalyze. In EasyAnalyze, we introduce GPU parallel computing, Map/Reduce architecture into network traffic analysis, which greatly improves the efficiency but does not increase the difficulty in programming. In the experiments, EasyAnalyze shows very promising results: 1 the speed is 6-17 times faster than conventional serial analysis in network traffic data analysis; and 2 the size of code is only 2% of the mainstream GPU Map/Reduce.

This paper designs and implements a network traffic collection and analysis system (NTCA). Network traffic is collected from multiple clients. According to the offline traffic analysis, network flows are analyzed and annotated with application protocol. At the same time, the results are stored in the system. NTCA also provides a visual interface to simplify the configuration process. With good flexibility and scalability, NTCA can be extended to a network traffic classification system, which is crucial to the research of network security technology.

Traffic volume data, especially average annual daily traffic (AADT), are important in transportation engineering. They are required in managing and maintaining existing facilities and in planning and designing new facilities. Many state highway agencies use the ramp counting procedure described in FHWA’s Traffic Monitoring Guide to estimate AADTs for freeways. The procedure involves counting all entrance and exit ramps between two established mainline counters (anchor points) and then reconciling the count data to estimate mainline AADT. The reconciling of count data includes three steps. First, AADTs for the ramps and the anchor points are estimated from the count data. Then AADT for each uncounted mainline link is calculated by addition or subtraction of ramp AADT to or from mainline AADT, starting from one anchor point. Finally, adjustments of the AADT are performed to achieve a match at the second anchor point if necessary. The process can be time-consuming and labor-intensive if it is done manually. A computer program to automate the process is required. The traffic count analysis system (TCAS) developed to automate the reconciling of count data in the ramp counting process is described. The TCAS was developed on the basis of data coaptation and data reconciliation techniques frequently used in the processing of network flow rate data. Data coaptation is used to calculate flow rates for uncounted links, and data reconciliation is used to adjust and balance the flow rates. The TCAS has been tested for the two longest freeways in Connecticut. The results are close to those from the ramp counting procedure. However, the TCAS significantly reduces the time and labor required for processing traffic volume data for freeways.

The scale of expressway information networking is constantly expanding. Currently the existing analysis system is still built on the relational database. Traffic data produced by the system has reached a data volume of 3 million items monthly. The performance requirements, including high concurrency, massive throughput, visualisation and scalability, are difficult to be satisfied. The expressway traffic data analysis system (ETDAS) is designed to meet the needs of the collection, analysis and visualisation of increasing expressway traffic data by means of the distributed frameworks. The new system is expected to help regulate the road network traffic flow, reduce traffic congestion, and provide analytical support for the optimisation strategy of road network. ETDAS has been deployed online.

The lack of current network dynamics studies that evaluate the effects of new application and protocol deployment or long-term studies that observe the effect of incremental changes on the Internet, and the change in the overall stability of the Internet under various conditions and threats has made network monitoring challenging. A good understanding of the nature and type of network traffic is the key to solving congestion problems. In this paper we describe the architecture and implementation of a scalable network traffic moni-toring and analysis system. The gigabit interface on the monitoring system was configured to capture network traffic and the Multi Router Traffic Grapher (MRTG) and Webalizer produces graphical and detailed traffic analysis. This system is in use at the Obafemi Awolowo University, IleIfe, Nigeria; we describe how this system can be replicated in another environment.

With the rapid development of cities, massive and complex traffic data is being generated and collected. The traffic data is not intuitive and cannot highlight key information about urban traffic conditions. However, traffic data visualization can directly correlate users with the data, and support users to interact with data in a convenient and visual way. Then realize the feedback of blending user wisdom and machine intelligence. This paper investigates a structured survey of the state of the art in the visualization of traffic data. First, we reviewed five representative traffic data visualization methods including WebVRGIS based traffic analysis and visualization system, TripMiner, IoV distributed architecture, SMASH architecture, and LDA-based topic modelling. Meanwhile, we analyzed the traffic datasets that applied in each method. Then we summarize these methods from seven aspects: scalability, data storage, data update, interactivity, reliability, data anomaly detection, and spatiotemporal visualization. In addition, we make a detailed comparative analysis of the key capabilities of five representative traffic data visualization methods in processing traffic big data. Finally, we conclude that the SMASH architecture performs better in processing high speed and large flow traffic data. Moreover, we propose a novel direction for optimizing traffic data visualization techniques.

Data Warehouse (DW) is topic-oriented, integrated, static datasets which are used to support decision-making. Driven by the constraint of mass spatio-temporal data management and application, Spatio-Temporal Data Warehouse (STDW) was put forward, and many researchers scattered all over the world focused their energy on it.Although the research on STDW is going in depth , there are still many key difficulties to be solved, such as the design principle, system framework, spatio-temporal data model (STDM), spatio-temporal data process (STDP), spatial data mining (SDM) and etc. In this paper, the concept of STDW is discussed, and analyzes the organization model of spatio-temporal data. Based on the above, a framework of STDW is composed of data layer, management layer and application layer. The functions of STDW should include data analysis besides data process and data storage. When users apply certain kind of data services, STDW identifies the right data by metadata management system, then start data processing tool to form a data product which serves the data mining and OLAP. All varieties of distributed databases (DDBs) make up data sources of STDW, including Digital Elevation Model (DEM), Diagnosis-Related Group (DRG), Data Locator Group (DLG), Data Objects Management (DOM), Place Name and other databases in existence. The management layer implements heterogeneous data processing, metadata management and spatio-temporal data storage. The application layer provides data products service, multidimensional data cube, data mining tools and on-line analytical process.

With the exponential increase in the urban population, urban transportation systems are confronted with numerous challenges. Traffic congestion is common, traffic accidents happen frequently, and traffic environments are deteriorating. To alleviate these issues and improve the efficiency of urban transportation, accurate traffic forecasting is crucial. In this study, we aim to provide a comprehensive overview of the overall architecture of traffic forecasting, covering aspects such as traffic data analysis, traffic data modeling, and traffic forecasting applications. We begin by introducing existing traffic forecasting surveys and preliminaries. Next, we delve into traffic data analysis from traffic data collection, traffic data formats, and traffic data characteristics. Additionally, we summarize traffic data modeling from spatial representation, temporal representation, and spatio-temporal representation. Furthermore, we discuss the application of traffic forecasting, including traffic flow forecasting, traffic speed forecasting, traffic demand forecasting, and other hybrid traffic forecasting. To support future research in this field, we also provide information on open datasets, source resources, challenges, and potential research directions. As far as we know, this paper represents the first comprehensive survey that focuses specifically on the overall architecture of traffic forecasting.

The urban intelligent traffic decision analysis system is a system designed to solve urban road traffic problems, improve traffic safety and comfort, and improve the level of traffic services. The application of big data mining technology can improve the ability of various types of road traffic information analysis and processing, expand the scope of application, and provide decision-making reference for the application of intelligent transportation systems, so as to ensure the smoothness and safety of roads. This article will introduce the concept of big data mining technology and intelligent transportation system, and analyze the architecture of urban traffic intelligent system based on big data technology and the design and implementation of mining system.

High-speed network poses challenges on heavy traffic measurement and analysis, which becomes more serious due to the scale of modern data centers. Cloud platform is an alternative to crack the nut because it gives concurrent considerations to both high-performance computing and high-extensible data storage. Therefore, we use Hadoop to construct a trial cloud platform to tackle the challenges derived from the heavy traffic, and perform network traffic analysis and target packet dissection. Our testbed evaluates the performance against different processing strategies, and shows the statistic results. According to the experimental results, the Hadoop-based platform can fit to process massive traffic data very well, and it's appropriate to manage today's data centers with cloud computing services.

Due to the limited amount of sensor infrastructure and its distribution restrictions, it is a challenge for the traffic related groups to estimate traffic conditions. The GPS log files of floating cars have fundamentally improved the quantity and quality of wide-range traffic data collection. To convert this data into useful traffic information, new traffic models and data processing algorithms must be developed. To conduct a comprehensive and accurate traffic flow analysis, this paper proposes a traffic flow analysis and prediction process based on GPS data of floating cars. For the key links involved in the process, new models and algorithms are implemented. Based on the actual data of a city, we validate the effectiveness of the design process, and provide a reliable solution for urban traffic flow analysis.

Nowadays, congestion and accidents are creating major risks to cities, including delays, higher fuel usage, and compromised safety. Effective traffic modelling and accident data analysis are critical for identifying high risk identifying accident-prone locations, understanding the causes of accidents and creating focused actions to enhance traffic flow and safety. GIS is an effective tool for integrating, analysing and visualizing different geographical data relevant to transportation networks such as, traffic flow, infrastructure, and safety. It enables geographical analysis and visualization of accident hotspots by integrating accident data, road conditions, traffic numbers, and environmental factors. The use of GIS in traffic modelling and accident data analysis provides considerable benefits in urban transportation planning and management. The aim of the paper is to provide an overview of the application of GIS in traffic modelling and accidental data analysis, highlighting the methodologies, advancements, and challenges in this field. The review shall provide a comprehensive assessment of the current state of traffic modelling and accidental data analysis using GIS. It will highlight the significant contributions of GIS technology, identify key research gaps, and offer insights into future directions for enhancing transportation planning and decision-making processes.

Netflow-based network traffic analysis is one of today's mainstream network traffic monitoring and analysis techniques. However, port-based traffic identification that employed in Netflow-based protocol analysis is inaccurate. Netflow cannot provide enough details for analysis of site access behaviour. In order to overcome the above limitation, we propose an enhanced Netflow data collection method combining packet capturing and flow technique for accurate application identification. Then we design and implement an enhanced Netflow data collection system, which lays a data foundation for traffic analysis. Finally, the feasibility and effectiveness of the system are verified.

This paper proposes a method of applying a machine learning algorithm in order to improve the recognition rate of the video-based traffic information system. After applying the Error Backpropagation learning neural network algorithm to the traffic information it will be used for image recognition results. The training data is generated from the traffic information system, the noise of the generated data is removed by Gaussian smoothing. In this paper, we develop a machine learning based Traffic Condition analysis system was able to get an improved recognition rate than conventional vision-based system.