Graph based Tensor Recovery for Accurate Internet Anomaly Detection

A parallel low rank matrix optimization method for recovering internet traffic network data via link flow measurement

Anomalous traffic detection is a vital task in advanced Internet supervision and maintenance. To detect anomalies accurately, various data representations, such as vectors, matrices, and tensors, have been adopted to model traffic data. Among them, tensor-based methods outperform others due to their capability of capturing comprehensive correlations between complex network traffic. However, existing tensor-based algorithms remain certain shortcomings, such as working offline, cannot timely detect traffic anomalies, and high computation costs. To conquer the aforementioned deficiencies, we propose a novel <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sequence tensor recovery</i> (STR) algorithm in this paper, which utilizes the results of historical tensor decomposition to achieve quick and accurate anomaly detection with low consumption when traffic data series arrive. Furthermore, we propose a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">dynamic sequence tensor recovery</i> (DSTR) algorithm to improve anomaly detection accuracy by better capturing the variation over time of the comprehensive correlation of traffic data hidden in the tensor structure. The experimental results on two real traffic traces, Abilene and G <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\grave{E}$</tex-math></inline-formula> ANT, indicate the proposed STR and DSTR algorithms are superior to the state-of-the-art algorithms in terms of accuracy and computation cost.

Traffic flow prediction is an important part of intelligent transportation systems (ITS). However, sensor failures or the transmission distortion often occur in the process of data acquisition, which will inevitably cause the loss or abnormality of traffic flow data transmitted to the edge server. In this situation, it is necessary to share traffic flow data among different platforms. However, existing traffic flow prediction methods are facing two challenges in the process of traffic flow data sharing. First, user privacy is often leaked in the process of sharing traffic data on various platforms. Moreover, with the continuous updating of data, the efficiency and scalability of data sharing between different platforms will become lower and lower. In view of the above challenges, in this paper, we propose a novel prediction method for the missing traffic flow data caused by abnormal sensors, named ASMVP_{distr-LSH} based on distributed locality-sensitive hashing (LSH) technique. At last, a case study is presented to illustrate the feasibility and effectiveness of our approach ASMVP_{distr-LSH}.

In real-world traffic data, missing data is inevitable for traffic collection problem and some other reasons. Therefore, it is increasingly critical to complete the whole traffic data. To make full use of hidden spatial-temporal structures of the Internet traffic data, we extend to traffic matrix (user, server) to a three-way tensor (user, server, time). This paper proposed a novel algorithm to recover the missing data, named TSVT (Tucker decomposition and tensor singular value thresholding). We exploit the proposed the Tucker decomposition and low rank tensor completion (LRTC). Based on LRTC, we extend the known singular value thresholding (SVT) to the tensor case, and combine Tucker decomposition to optimize the performance of the algorithm. Based on Internet traffic data, we model two different tensors, and apply our TSVT to infer the unobserved data of the traffic. Furthermore, we apply TSVT algorithm to our Internet data. Numerical experiments on Synthetic data verify our method and the application for completing DPI traffic data proves the effectiveness of the method.

Outlier detection is one important research area of data mining, which plays key roles in data preprocessing, equipment fault diagnosis, credit fraud detection, traffic incident detection etc. This paper is devoted to a new outlier detection algorithm based on the degree of sharpness. The proposed algorithm takes a new way to solve the outlier detection problem, which employs a measure in image processing, degree of sharpness, to detect the outliers. Compared to the classical outlier detection methods with statistical learning, the proposed algorithm has no iterative processes. It generates a smooth curve to describe the overall distribution of the data firstly, and then computes the sharpness of degree for each data point. Finally, the outliers are recognized as they have larger values of the degree of sharpness. Also, some practical applications on traffic big data are presented to prove the effectiveness of the proposed algorithm.

Last decades have witnessed the monumental growth of Inter-net applications, and the network security fields have attracted increasing interests as it has become a necessity due to proliferation of information technologies in everyday life. Net-work anomaly detection aims to detect the potential anomalous behavior in traffic data, which is critical for detecting and reacting to the presence of attacks in the network. In this pa-per, we present our solution for the ICASSP 2021 Network Anomaly Detection Challenge (NAD) challenge. Firstly, we present our feature extraction method in detecting network anomalies, as efficiently extracting the discriminate features from the traffic data over time can be greatly helpful to improve the anomaly detection performance. Secondly, both gradient boosting decision trees and TabTransformer-Based classification model are trained for the multi-class classification task. The TabTransformer is based on self-attention and can transform the categorical feature' embeddings into robust contextual embeddings, thus, can achieve higher prediction accuracy. Finally, we ensemble the results to improve the performance further. To demonstrate the effectiveness of our approach, extensive experiments have been conducted on the NAD datasets. Our approach achieves the score of 0.625 and ranked as 2nd place in the final leaderboard.

Locality-sensitive hashing (LSH) is the basis of many algorithms that use a probabilistic approach to find nearest neighbors. We describe an algorithm for optimizing the parameters and use of LSH. Prior work ignores these issues or suggests a search for the best parameters. We start with two histograms: one that characterizes the distributions of distances to a point's nearest neighbors and the second that characterizes the distance between a query and any point in the data set. Given a desired performance level (the chance of finding the true nearest neighbor) and a simple computational cost model, we return the LSH parameters that allow an LSH index to meet the performance goal and have the minimum computational cost. We can also use this analysis to connect LSH to deterministic nearest-neighbor algorithms such as $k$ - $d$ trees and thus start to unify the two approaches.

As machine learning (ML)-based traffic classification develops, Internet traffic data is published in public to serve as test data. Although the IP addresses therein are anonymized, it is given explicitly which data belongs to an identical user. Then using the information, an adversary can identify a user from the anonymized users. The paper first gives a k-anonymity method to reduce the probability of information leak to P/k, where P is the probability of information leak without k-anonymity. Assume the number of the flows belonging to an IP address follows Normal distribution, the information loss is shown (μ2+σ2)/(kμ2+σ2), where μ and σ are respectively the mean and the variance of the Normal distribution. Later, random noise is added to further reduce the probability of information leak to P/k2, with an expected distortion rate of approximately 2d+log k-log|X|, where d is the number of dimensions and |X| is the number of the vectors. At last, real-world Internet traffic data is used to evaluate the utility of the anonymized traffic data. According to the experimental results, the k-anonymized noised data can be clustered with an overall accuracy rate close to the state-of-the-art results for non-anonymized traffic data.

This paper proposes a new multi-channel network traffic anomaly detection method combined with the idea of multi-scale decomposition and multi-channel detection theory. It can be learned that anomalies could change the characteristics of traffic data at different scales. Traditional anomaly detection methods usually work on each scale independently thus mainly focused on temporally correlated traffic. With the fully exploration on internal frequency-time correlations within multiple scales, this method first obtained the multi-scale decomposition of original traffic data using Ensemble Empirical Mode Decomposition (EEMD), then it is combined with a multi-channel Generalized Likelihood Ratio Test (GLRT) detector, for anomaly detection and decision-making. It can be verified with experiments that this method performs better than other traditional methods, thus gives a new sight on the anomaly detection with different types of traffic data.

Modal frequencies are widely used as damage-sensitive features in vibration-based unsupervised anomaly detection for full-scale civil structures. However, environmental variations introduce nonlinear and non-Gaussian behaviors in modal frequencies, characterized by multicentricity and substantial variance differences. These behaviors inevitably mask damage-induced frequency shifts, leading to increased false-detection rates and degraded real-time performance. To address this, this paper presents a novel unsupervised framework for early-stage anomaly detection that integrates exponential slow feature analysis (SFA) with just-in-time learning-aided standard K -nearest neighbor metrics. First, nonlinear modal features with minimum temporal variation are extracted from modal frequencies via exponential SFA for dimensionality reduction. Afterward, under the standard measurement criteria, the Euclidean squared distances between each sample in the entire training set and its nearest neighbors are computed. The standard distance for each neighbor is then modeled, with the number of neighbors automatically selected through Bayesian optimization and cross-validation. The primary purpose of this step is to ensure that the nearest neighbor distances between samples under different environmental conditions and damage conditions including abrupt anomalies or gradual degradations, are measured on a consistent scale, thereby minimizing environmental variability. After that, a robust detection indicator is derived from the mean of squared nearest neighbor distances and combined with kernel density estimation to determine an online detection threshold for outliers. The results indicate the proposed method effectively mitigates strong environmental variability without prior environmental measurements and accurately identifies anomalies in bridge modal frequencies with negligible false-positive and false-negative errors.

Outlier detection is a key technique in data ming and machine learning fields. The deviating characters of outliers make huge detrimental effects on the learning tasks. A lot of algorithms are therefore proposed to handle outliers from different perspectives, such as distance, density, angle and so on. Among these approaches, the density-based methods achieve better performance, but also suffer from huge time complexity. Recently, in order to accelerate the speed and improve the performance, the subsampling ensemble method attracts much attention, which has a reasonable theoretical interpretation and high performance. However, existing work only gives the partial picture of outlier detection via row-sampling, the effective portfolio of bi-sampling is still void. In light of this, we propose the general outlier detection framework via bi-sampling, Bi-Sampling Outlier Detection (BSOD) and provide the effective portfolios of the row and column-sampling ratios in a theoretical way. In addition, the benefits of BSOD are fully illustrated in terms of ensemble diversity and divide-and-conquer. Further we employ LOF within BSOD as BI-LOF to conduct extensive experiments. In general, on 30 synthetic and 17 real-world data sets we thoroughly explore the characteristics of BI-LOF with different numbers of instances, features, nearest neighbors, validate the theoretical analysis of BSOD condition on synthetic data sets, and show obvious advantages over other state-of-the-art algorithms in terms of low and high dimensional real-world data sets. And finally we use BI-LOF to conduct image outlier detection and show high quality and stableness of BI-LOF.

In recent years, various types of Intelligent Transportation Systems(ITSs) have produced a large amount of structured traffic flow data. Using these structured data and deep learning models to predict urban short-term traffic flow is an important focus of current research. In order to solve the problem of low accuracy of Long-Short Term Memory(LSTM) model in urban traffic flow prediction, this paper proposes an LSTM model based on attention mechanism. The proposed model can learn the importance of each past value to the current value from the long sequence of traffic data at the past moment, which makes it possible to extract more valuable features. Constructed a dataset using the traffic data in the core section of Wuhan for experiments, and the performance of the improved model is compared with the original LSTM model. The results show that when the input data sequence increases from 16 to 64, the MAPE of our proposed model is reduced by 3.76%, while the MAPE of the LSTM model is reduced by 1.51%, which proves the effectiveness of our proposed method.

This paper explores the immunological model and implements it in the domain of intrusion detection on computer networks. The main objective of the paper is to monitor, log the network traffic and apply detection algorithms for detecting intrusions within the network. The proposed model mimics the natural Immune System (IS) by considering both of its layers, innate immune system and adaptive immune system respectively. The current work proposes Statistical Modeling based Anomaly Detection (SMAD) as the first layer of Intrusion Detection System (IDS). It works as the Innate Immune System (IIS) interface and captures the initial traffic of a network to find out the first-hand vulnerability. The second layer, Adaptive Immune-based Anomaly Detection (AIAD) has been considered for determining the features of the suspicious network packets for detection of anomaly. It imitates the adaptive immune system by taking into consideration the activation of the T-cells and the B-cells. It captures relevant features from header and payload portions for effective detection of intrusion. Experiments have been conducted on both the real-time network traffic and the standard datasets KDD99 and UNSW-NB15 for intrusion detection. The SMAD model yields as high as 96.04% true positive rate and around 97% true positive rate using real-time traffic and standard data sets. Highly suspicious traffic detected in the SMAD model is further tested for vulnerability in the AIAD model. Results show significant true positive rate, closer to almost 99% of accurately detecting the file-based and user-based anomalies for both the real-time traffic and standard data sets.

Anomaly detection is a critical issue in Network Intrusion Detection Systems (NIDSs). Most anomaly based NIDSs employ supervised algorithms, whose performances highly depend on attack-free training data. However, this kind of training data is difficult to obtain in real world network environment. Moreover, with changing network environment or services, patterns of normal traffic will be changed. This leads to high false positive rate of supervised NIDSs. Unsupervised outlier detection can overcome the drawbacks of supervised anomaly detection. Therefore, we apply one of the efficient data mining algorithms called random forests algorithm in anomaly based NIDSs. Without attack-free training data, random forests algorithm can detect outliers in datasets of network traffic. In this paper, we discuss our framework of anomaly based network intrusion detection. In the framework, patterns of network services are built by random forests algorithm over traffic data. Intrusions are detected by determining outliers related to the built patterns. We present the modification on the outlier detection algorithm of random forests. We also report our experimental results over the KDD'99 dataset. The results show that the proposed approach is comparable to previously reported unsupervised anomaly detection approaches evaluated over the KDD' 99 dataset.

Anomaly in geoacoustic emission is an important earthquake precursor. Current geoacoustic anomaly detection methods are limited by their low signal-to-noise ratio, low intensity, sample imbalance, and low accuracy. Therefore, this paper proposes a clone selection algorithm optimized one-class support vector machine method (CSA-OCSVM) for geoacoustic anomaly detection. First, the interquartile range (IQR), cubic spline interpolation, and time window are designed to amplify the geoacoustic signal intensity and energy change rules to reduce the interference of geoacoustic signal noise and intensity. Secondly, to address the imbalance of positive and negative samples in geoacoustic anomaly detection, a one-class support vector machine is introduced for anomaly detection. Meanwhile, in view of the optimization capabilities of the clone selection algorithm, it is adopted to optimize the hyperparameters of OCSVM to improve its detection accuracy. Finally, the proposed model is applied to geoacoustic data anomaly detection in nine different datasets, which are derived from our self-developed acoustic electromagnetic to AI (AETA) system, to verify its effectiveness. By designing comparative experiments with IQR, genetic algorithm OCSVM (GA-OCSVM), particle swarm optimization OCSVM (PSO-OCSVM), and evaluating the performance of the true positive rate (TPR) and false positive rate (FPR), the experimental results depict that the proposed model is superior to the existing state-of-the-art geoacoustic anomaly detection approaches.