A parallel low rank matrix optimization method for recovering internet traffic network data via link flow measurement

In real-world traffic data, missing data is inevitable for traffic collection problem and some other reasons. Therefore, it is increasingly critical to complete the whole traffic data. To make full use of hidden spatial-temporal structures of the Internet traffic data, we extend to traffic matrix (user, server) to a three-way tensor (user, server, time). This paper proposed a novel algorithm to recover the missing data, named TSVT (Tucker decomposition and tensor singular value thresholding). We exploit the proposed the Tucker decomposition and low rank tensor completion (LRTC). Based on LRTC, we extend the known singular value thresholding (SVT) to the tensor case, and combine Tucker decomposition to optimize the performance of the algorithm. Based on Internet traffic data, we model two different tensors, and apply our TSVT to infer the unobserved data of the traffic. Furthermore, we apply TSVT algorithm to our Internet data. Numerical experiments on Synthetic data verify our method and the application for completing DPI traffic data proves the effectiveness of the method.

Privacy preserving data mining of sequential patterns for network traffic data

Detecting anomalous traffic is a crucial task of managing networks. Many anomaly detection algorithms have been proposed recently. However, constrained by their matrix-based traffic data model, existing algorithms often suffer from low detection accuracy. To fully utilize the multi-dimensional information hidden in the traffic data, this paper takes an initiative to investigate the potential and methodologies of performing tensor factorization for more accurate Internet anomaly detection. Only considering the low-rank linearity features hidden in the data, current tensor factorization techniques would result in low anomaly detection accuracy. We propose a novel Graph-based Tensor Recovery model (Graph-TR) to well explore both low rank linearity features as well as the non-linear proximity information hidden in the traffic data for better anomaly detection. We encode the non-linear proximity information of the traffic data by constructing nearest neighbor graphs and incorporate this information into the tensor factorization using the graph Laplacian. Moreover, to facilitate the quick building of neighbor graph, we propose a nearest neighbor searching algorithm with the simple locality-sensitive hashing (LSH). We have conducted extensive experiments using Internet traffic trace data Abilene and GEANT. Compared with the state of art algorithms on matrix-based anomaly detection and tensor recovery approach, our Graph-Trcan achieve significantly lower False Positive Rate and higher True Positive Rate.

Monitoring the performance of a large network would involve a high measurement cost. To reduce the overhead, sparse network monitoring techniques may be applied to select paths or time intervals to take the measurements, while the remaining monitoring data can be inferred leveraging the spatial-temporal correlations among data. The quality of missing data recovery, however, highly relies on the specific inference technique adopted. Tensor completion is a promising technique for more accurate missing data inference by exploiting the multi-dimensional data structure. However, data processing for higher dimensional tensors involves a large amount of computation, which prevents conventional tensor completion algorithms from practical application in the presence of large amount of data. This work takes the initiative to investigate the potential and methodologies of performing parallel processing for high-speed and high accuracy tensor completion over Graphics Processing Units (GPUs). We propose a GPU-accelerated parallel Tensor Completion scheme (GPU-TC) for accurate and fast recovery of missing data. To improve the data recovery accuracy and speed, we propose three novel techniques to well exploit the tensor factorization structure and the GPU features: grid-based tensor partition, independent task assignment based on Fisher-Yates shuffle, sphere facilitated and memory-correlated scheduling. We have conducted extensive experiments using network traffic trace data to compare the proposed GPU-TC with the state of art tensor completion algorithms and matrix-based algorithms. The experimental results demonstrate that GPU-TC can achieve significantly better performance in terms of two relative error ratio metrics and computation time.

Digital crimes are a part of modern life but evidence of these crimes can be captured in network traffic data logs. Analysing these logs is a difficult process, this is especially true as the format that different attacks can take can vary tremendously and may be unknown at the time of the analysis. The main objective of the field of network forensics consists of gathering evidence of illegal acts from a networking infrastructure. Therefore, software tools, and techniques, that can help with these digital investigations are in great demand. In this paper, an approach to analysing and visualising network traffic data based upon the use of self-organising maps (SOM) is presented. The self-organising map has been widely used in clustering tasks in the literature; it can enable network clusters to be created and visualised in a manner that makes them immediately more intuitive and understandable and can be performed on high-dimensional input data, transforming this into a much lower dimensional space. In order to show the usefulness of this approach, the self-organising map has been applied to traffic data, for use as a tool in network forensics. Moreover, the proposed SOM takes into account the qualitative features that are present in the traffic data, in addition to the quantitative features. The traffic data was was clustered and visualised and the results were then analysed. The results demonstrate that this technique can be used to aid in the comprehension of digital forensics and to facilitate the search for anomalous behaviour in the network environment.

Traffic sensor networks allow convenient collection of travel data that are of great significance for intelligent transportation systems (ITSs). However, the universality of missing data impedes the application of ITS and thus accurate missing data recovery is indispensable in practice. Typically, the global low-rankness and local spatiotemporal smoothness exist in underlying traffic tensor data. In light of this, this article proposes an improved low-rank tensor completion (LRTC) model by exploiting abundant structural information from incomplete tensors. Specifically, a logarithm power composite (LPC)-norm is first proposed as a nonconvex substitute of the rank function, leading to a flexible characterization of tensor multidimensional correlation. Then, a joint structural regression (JSR) model is presented to simultaneously leverage the intrinsic temporal continuity and profile similarity of traffic data. By doing so, we construct a novel nonconvex LRTC model by integrating the global low-rankness and fine-grained spatiotemporal structure that are complementary to each other. To solve the proposed model, following the optimization framework of the alternating direction method of multipliers (ADMMs), we develop an efficient iterative algorithm where each step can be solved in a closed form. Extensive experiments on four real-world traffic data are conducted to evaluate the effectiveness of the proposed approach. The results demonstrate that compared with other tensor completion methods, our model significantly improves the recovery performance.

Network operators are now facing bandwidth outages as well as a growing pressure to ensure good Quality of Service (QoS). An important practical issue for network service providers is to pay close attention to the load changes of network traffic, in particular, the stationary increase of load from a normal demand. Many network monitoring applications and performance analysis tools are based on the study of an aggregate measure of network traffic, e.g. number of packets in transit (NPT), which is a long-term univariate time series. To classify this type of network traffic data and detect any increase of network source load, we propose a dynamic principal component analysis (PCA) method, first to extract data features and then to detect a stationary load increase of network traffic. The proposed detection schemes are based on either the major or the minor principal components of network traffic data. To demonstrate the applications of the proposed feature extraction method and the detection schemes, we applied them to network traffic data simulated from the packet switching network (PSN) model. Additionally, we propose a combined detection scheme that uses both the major and the minor principal components. The proposed detection schemes, based on dynamic PCA, show enhanced performance in detecting an increase of network load for the simulated network traffic data. These results offer a new feature extraction method based on dynamic PCA that creates additional feature variables for event detection in a univariate time series.

The traffic data of the intelligent network with the Space-Integrated Ground Information Network have the characteristics of robust traffic burst, large traffic data, and high volatility compared with the ground traffic. We propose a Gated Temporal Convolution Network (GTCN) algorithm to solve the above problems, the algorithm segments the traffic time series into equal length according to the autocorrelation of the network traffic, and creates the feature sequence by extracting the segmented features using Temporal Convolution Network (TCN), to learn the burst and fluctuation of the intelligent network traffic of the Space-Integrated Ground Information Intelligent Network and to improve the processing efficiency of the traffic time series. Then, the feature sequences of each subsection data are stored, screened, and retained through the gated unit. Finally, the network traffic data are accurately predicted using the characteristics of the sequences stored by the TCN and the gating unit. The simulation experiments showed that, compared with the comparison algorithms, the accuracy of the proposed algorithm is improved by at least 58.72%, the training speed of the algorithm is improved by 84.05%, and the algorithm execution speed is improved by 71.92%, which meet the requirements of the accuracy and timeliness of the traffic prediction model of the Space-Integrated Ground Information Intelligent Network.

As machine learning (ML)-based traffic classification develops, Internet traffic data is published in public to serve as test data. Although the IP addresses therein are anonymized, it is given explicitly which data belongs to an identical user. Then using the information, an adversary can identify a user from the anonymized users. The paper first gives a k-anonymity method to reduce the probability of information leak to P/k, where P is the probability of information leak without k-anonymity. Assume the number of the flows belonging to an IP address follows Normal distribution, the information loss is shown (μ2+σ2)/(kμ2+σ2), where μ and σ are respectively the mean and the variance of the Normal distribution. Later, random noise is added to further reduce the probability of information leak to P/k2, with an expected distortion rate of approximately 2d+log k-log|X|, where d is the number of dimensions and |X| is the number of the vectors. At last, real-world Internet traffic data is used to evaluate the utility of the anonymized traffic data. According to the experimental results, the k-anonymized noised data can be clustered with an overall accuracy rate close to the state-of-the-art results for non-anonymized traffic data.

Location based services (LBS) are capturing global spotlights increasingly, in which users tend to pick up navigation service as their favorites. This paper examines the dynamic routing problems based on real-time traffic information in LBS. At first, the overall structure of dynamic navigation system is illustrated, which is composed of four ingredients: traffic data collection, digital road map, static routing and dynamic re-routing. Traffic data collection is the task of traffic information center (TIC), where various sources of data are fused into link travel times that are used as criteria to choose candidates in path determining. Digital road map includes digital road network and historical traffic data, where the topology relationship of road network is elementarily explored and the method to represent statistical traffic data is referred to. Routing with historical traffic data and dynamic re-routing with live traffic data are two main functions of dynamic navigation service. The flow of dynamic re-routing algorithm is exhibited. Lastly, a first-in-first-out (FIFO) time-depend shortest path algorithm is discussed for dynamic routing, which adopts the adaptive A^* algorithm based on binary heap priority queue and RB-tree.

The loss of traffic state data is a common problem in intelligent transportation system. To improve the imputation accuracy and robustness of road traffic data, a novel generative adversarial network for the imputation of road network traffic data is proposed in this paper, i.e., GAE-GAN-LSTM. The spatiotemporal characteristics were extracted using a improved graph auto-encoder (GAE), followed by a generative adversarial network (GAN) to generate the complete spatiotemporal characteristics on the basis of the missing features. The internal structure of the generator was a long short-term memory network (LSTM), and the internal structure of the discriminator was a fully connected neural network (FCN). Finally, the traffic state data could be recovered by the decoder of GAE. The experimental results revealed that the performance of the proposed method was better than that of the other methods at any data loss ratio considered. The main innovations of the proposed method include two aspects. One, an improved GAE for the imputation of road network data was presented by redefining the loss function of GAE, which could effectively extract the potential spatiotemporal features of a road network. Two, the GAN was used to generate the spatiotemporal characteristics of the traffic state data by using the strong data generation ability of GAN.KeywordsData imputationGraph auto-encoderGenerative adversarial networkRoad traffic state data

Data missing remains a difficult and important problem in the transportation information system, which seriously restricts the application of the intelligent transportation system (ITS), dominatingly on traffic monitoring, e.g., traffic data collection, traffic state estimation, and traffic control. Numerous traffic data imputation methods had been proposed in the last decade. However, lacking of sufficient temporal variation characteristic analysis as well as spatial correlation measurements leads to limited completion precision, and poses a major challenge for an ITS. Leveraging the low-rank nature and the spatial-temporal correlation of traffic network data, this paper proposes a novel approach to reconstruct the missing traffic data based on low-rank matrix factorization, which elaborates the potential implications of the traffic matrix by decomposed factor matrices. To further exploit the temporal evolvement characteristics and the spatial similarity of road links, we design a time-series constraint and an adaptive Laplacian regularization spatial constraint to explore the local relationship with road links. The experimental results on six real-world traffic data sets show that our approach outperforms the other methods and can successfully reconstruct the road traffic data precisely for various structural loss modes.

Underwater wireless sensor networks (UWSNs) have been used increasingly for critical tasks such as environment surveillance and underwater exploration. Nevertheless, their peculiar working environment, which entails their use of acoustic communication, high message latency, low bandwidth, mobility, stringent energy resources, and noisy communication channels, renders them susceptible to complex cyber threats including Sybil, Denial of Service (DoS), and traffic analysis attacks. This reality is a major drawback to the use of traditional intrusion detection systems used in other wireless communication networks. This manuscript addresses these issues through the presentation of an optimized cyberattack detection framework in UWSNs using an equivariant quantum convolutional neural network integrated with flamingo jellyfish search optimization (CD‐UWSN‐EQCNN‐FJSO). In this approach, network traffic data from the NSL‐KDD dataset are normalized using Bayesian boundary trend filtering (BBTF) to handle noise and uncertainty. Bitterling fish optimization (BFO) is then applied for feature selection, with further statistical feature extraction made by the double probability integral transform (DPIT). Here, the designed equivariant quantum convolutional neural network (EQCNN) is well leverage equivariance properties to perform robust detection under dynamic underwater network conditions, while the flamingo jellyfish search optimization (FJFO) approach dynamically fine‐tunes the weights within the network for improved detection accuracy and lower false alarm rates. The experimental results indicate that the proposed CD‐UWSNs‐EQCNN‐FJSO approach is able to provide much 27.15%, 26.09%, and 28.10% higher accuracy and 29.03%, 25.23%, and 29.1% higher recall capabilities, as well as much lower rates of 20.09%, 22.24%, and 20.01% lower false positive rate than other UWSN‐based cyber security solutions available in the existing methods.

With the rapid development of large-scale complex networks and proliferation of various social network applications, the amount of network traffic data generated is increasing tremendously, and efficient anomaly detection on those massive network traffic data is crucial to many network applications, such as malware detection, load balancing, network intrusion detection. Although there are many methods around for network traffic anomaly detection, they are all designed for single machine, failing to deal with the case that the network traffic data are so large that it is prohibitive for a single computer to store and process the data. To solve these problems, we propose a parallel algorithm based on Isolation Forest and Spark for network traffic anomaly detection. We combine the advantages of Isolation Forest algorithm in network traffic anomaly detection and big data processing capability of Spark technology. Meanwhile, we apply the idea of parallelization to the process of modeling and evaluation. In the calculation process, by assigning tasks to multiple compute nodes, Isolation Forest and Spark can efficiently perform anomaly detection and evaluation process. By this way, we can also solve the problem of computation bottleneck on single machine. Extensive experiments on real world datasets show that our Isolation Forest and Spark is efficient and scales well for anomaly detection on large network traffic data.

PPFSCADA: Privacy preserving framework for SCADA data publishing