Detecting and Identifying Network Anomalies by Component Analysis

The paper proposes an approach to security information processing in order to detect computer attacks and network anomalies based on big data technologies. The main contribution of the work is in the development, implementation and investigation of the proposed combined framework for processing security data using parallel computing environment and measuring the performance of the implemented system for detection of network attacks and anomalies. The research goal is to increase the performance of attack detection (under the given requirements for accuracy of solutions) compared to the traditional IDS application. The implemented approach is built using the open source systems Snort and Spark. The paper discusses the capabilities and performance assessment of parallel data processing in order to detect computer attacks and network anomalies, as well as key principles of working with big data. The presented main results of an experimental performance evaluation of the applied approach confirm its high efficiency for analyzing network traffic and security events.

The complexity of the Internet and the volume of network traffic have dramatically increased in the last few years, making it more challenging to design scalable Network Traffic Monitoring and Analysis (NTMA) systems. Critical NTMA applications such as the detection of network attacks and anomalies require fast mechanisms for on-line analysis of thousands of events per second, as well as efficient techniques for off-line analysis of massive historical data. The high-dimensionality of network data provided by current network monitoring systems opens the door to the massive application of machine learning approaches to improve the detection and classification of network attacks and anomalies, but this higher dimensionality comes with an extra data processing overhead. In this paper we present Big-DAMA, a big data analytics framework (BDAF) for NTMA applications. Big-DAMA is a flexible BDAF, capable to analyze and store big amounts of both structured and unstructured heterogeneous data sources, with both stream and batch processing capabilities. Big-DAMA uses off-the-shelf big data storage and processing engines to offer both stream data processing and batch processing capabilities, decomposing separate engines for stream, batch and query, following a Data Stream Warehouse (DSW) paradigm. Big-DAMA implements several algorithms for anomaly detection and network security using supervised and unsupervised machine learning (ML) models, using off-the-shelf ML libraries. We apply Big-DAMA to the detection of different types of network attacks and anomalies, benchmarking multiple supervised ML models. Evaluations are conducted on top of real network measurements collected at the WIDE backbone network, using the well-known MAWILab dataset for attacks labeling. Big-DAMA can speed up computations by a factor of 10 when compared to a standard Apache Spark cluster, and can be easily deployed in cloud environments, using hardware virtualization technology.

Recently, the concern about network management and network security has inspired many research topics, among which, research on detecting and identifying anomalies has attracted a lot of interest. In this paper, we propose an anomaly identification method based on traffic flow analysis. The result of the research proves that by assembling some features of traffic flows and calculating some specific extended metrics, we can successfully detect and identify network anomalies with high accuracy. Besides, the method also proves its efficiency with the ability to quantify network anomalies and locate relevant network nodes. The research validity has been verified by both simulation data and real network data.

The detection of anomalies in network traffic is a crucial issue affecting the security of Internet users. A statistical network anomaly detection algorithm is a promising way of detecting such anomalies, however, it has to be given appropriate parameters for accurate detection and identification. In general, it is very difficult to obtain appropriate parameter settings a priori, because network traffic is not stable in time or space. Thus, although many anomaly detection methods have been proposed, there has been little discussion about their parameter tunings. In this paper, we investigate an automatic and dynamic parameter tuning of a statistical network traffic anomaly detection method. In particular, we clarify whether one can consistently use the best parameter fixed for a certain instance; this choice clearly depends on the macroscopic and dynamic behavior of Internet traffic anomalies. We ascertain the appropriate learning period for setting a parameter of an anomaly detection algorithm based on a sketch and multi-scale gamma-function model by using real network traces measured in a trans-Pacific link over a period of six months. The main results of our study are as follows: (1) Without learning, the best parameter varies day by day. (2) With a longer learning period, the best parameter setting is affected by significant data during the learning period. (3) The appropriate period of the learning is about 3 days. (4) The performance degradation from introducing dynamic parameter tuning is 17% in the best case.

The importance of the ability to predict trends in social media has been growing rapidly in the past few years with the growing dominance of social media in our everyday’s life. Whereas many works focus on the detection of anomalies in networks, there exist little theoretical work on the prediction of the likelihood of anomalous network pattern to globally spread and become “trends”. In this work we present an analytic model for the social diffusion dynamics of spreading network patterns. Our proposed method is based on information diffusion models, and is capable of predicting future trends based on the analysis of past social interactions between the community’s members. We present an analytic lower bound for the probability that emerging trends would successfully spread through the network. We demonstrate our model using two comprehensive social datasets — the Friends and Family experiment that was held in MIT for over a year, where the complete activity of 140 users was analyzed, and a financial dataset containing the complete activities of over 1.5 million members of the eToro social trading community.

Signal processing techniques are used to analyze and detect network anomalies because of their ability to detect new and unknown intrusions. The paper proposes a method of modeling network signals for the detection of network anomalies, which combines wavelet approximation and the theory of system identification. To characterize the behavior of network traffic, fifteen functions are provided, which are used as input signals within the system. At the same time, it is assumed that security violations within the network can be detected by checking abnormal patterns of system functioning according to audit data.
 Despite the fact that machine learning methods have achieved significant results in detecting network anomalies, they still face the difficulty of using the implemented algorithms, in the presence of differences in the behavior of the training data and test data, which in turn leads to inefficient performance of the algorithms. This effect is exacerbated by the limitation of algorithms to detect previously unknown types of attacks due to the large number of false positives.
 The paper develops a new method of modeling network signals for detecting anomalies in networks using wavelet analysis. In particular, the general architecture of the approach consists of three components: feature analysis, modeling of normal network traffic based on wavelet approximation and prediction using ARX model, intrusion or non-intrusion decision making
 The result is evaluated using the DARPA intrusion detection dataset, which performs a comprehensive analysis of the intrusions in the dataset. Evaluation results show that this approach provides a high level of detection of both instances and types of attacks.

Internet of Things (IoT) is a technology that allows various physical devices to connect to each other and exchange data via the internet network. The application of IoT is increasingly widespread in various sectors such as smart homes, manufacturing industries, smart agriculture, and healthcare. However, along with the increasing number of devices and the volume of data traffic sent, the potential risk of cybersecurity threats is also increasing. The large number of IoT devices that have limited computing capabilities makes the system more vulnerable to various attacks, including intrusion, exploitation of system weaknesses, and Distributed Denial of Service (DDoS) attacks. Therefore, early detection of anomalies in network traffic is a crucial aspect to maintain the security and stability of IoT systems. This study aims to develop and implement a Support Vector Machine (SVM)-based architecture as a classification method in an anomaly detection system on an IoT network. SVM was chosen because of its ability to handle high-dimensional data and non-linear classification effectively. The methodology used includes the process of extracting features from IoT network traffic datasets, data normalization, model training using the SVM algorithm, and evaluating model performance in distinguishing between normal and anomalous traffic. Thus, the implementation of SVM architecture can be an effective and efficient solution in intrusion detection systems for IoT networks. This research also opens up opportunities for the development of more adaptive security systems by integrating machine learning-based detection models into large-scale IoT infrastructures.

Last decades have witnessed the monumental growth of Inter-net applications, and the network security fields have attracted increasing interests as it has become a necessity due to proliferation of information technologies in everyday life. Net-work anomaly detection aims to detect the potential anomalous behavior in traffic data, which is critical for detecting and reacting to the presence of attacks in the network. In this pa-per, we present our solution for the ICASSP 2021 Network Anomaly Detection Challenge (NAD) challenge. Firstly, we present our feature extraction method in detecting network anomalies, as efficiently extracting the discriminate features from the traffic data over time can be greatly helpful to improve the anomaly detection performance. Secondly, both gradient boosting decision trees and TabTransformer-Based classification model are trained for the multi-class classification task. The TabTransformer is based on self-attention and can transform the categorical feature' embeddings into robust contextual embeddings, thus, can achieve higher prediction accuracy. Finally, we ensemble the results to improve the performance further. To demonstrate the effectiveness of our approach, extensive experiments have been conducted on the NAD datasets. Our approach achieves the score of 0.625 and ranked as 2nd place in the final leaderboard.

We propose an unsupervised learning based anomaly detection framework for identifying cells experiencing performance degradation due to mobility problems, in LTE networks. Handover failure rate is used as a performance metric, whereas the mobility problems considered include too-early and too-late handovers. In order to enable unsupervised learning, the framework leverages existing datasets in commercial LTE networks (e.g. performance management counters, configuration management data, geographical locations, and inventory data etc). To this end, the first step is data pre-processing, followed by feature extraction based on principal component analysis and clustering. For implementation, we use real data from an operational commercial LTE network. Results show that clustering is highly effective in understanding and identifying mobility related anomalous behaviour, and provides actionable insights for automation and self-optimization, paving the way for efficient mobility robustness optimization, which is an important self-optimization use-case for contemporary 4G/5G networks.

In this era of digital transformation, the importance of network anomaly detection has been amplified to safeguard the security and integrity of various vital applications. These applications include the protection of critical infrastructures, prevention of cyber-attacks, and upkeep of network performance, among others. Our study presents a thorough evaluation of an array of machine learning algorithms for the proficient detection of network anomalies. These algorithms include, but are not limited to, Random Forest, Multi-Layer Perceptron (MLP), and Support Vector Machine (SVM). We illustrate an in-depth comparison between these chosen algorithms, analyzing their performance metrics, strengths, and weaknesses, with a specific focus on their practical applicability and influence on network security methodologies. The investigation into these machine learning algorithms exhibits potential advantages and constraints of employing such methodologies for network anomaly detection. We further shed light on the determinants that affect their performance in diverse scenarios. Based on the exhaustive analysis, we provide suggestive guidelines for choosing the most suitable algorithm depending on specific use cases or requirements. This study thus serves as a comprehensive guide to understanding the role and impact of machine learning in the critical field of network anomaly detection.

This paper explores the potential of machine-mastering models to automate network Anomaly Detection (NAD) through Time series analysis. We employ a two-level method wherein the primary degree entails function selection thru foremost component analysis (PCA), accompanied by gadget mastering (ML) model choice from more than a few supervised studying algorithms. The second stage evaluates the overall performance of the numerous selected ML models and optimizes theirhyperparameters when necessary. Our experiments demonstrate that ML-driven computerized network Anomaly Detection can provide accurate and well-timed detection of network anomalies with little supervision and parameter tuning attempts. The outcomes of our experiments display that Random Forests and Support Vector Machines (SVMs) carry out first-rate some of the model’s grid searches, demonstrating aggressive accuracy and precision ratings from an anomaly detection perspective. We also intensely evaluate the consequences and provide insightful discussion on the possibilities and challenges surrounding using ML for automatic community Anomaly Detection.

The article deals with detection of network anomalies. Network anomalies include everything that is quite different from the normal operation. For detection of anomalies were used machine learning systems. Machine learning can be considered as a support or a limited type of artificial intelligence. A machine learning system usually starts with some knowledge and a corresponding knowledge organization so that it can interpret, analyse, and test the knowledge acquired. There are several machine learning techniques available. We tested Decision tree learning and Bayesian networks. The open source data-mining framework WEKA was the tool we used for testing the classify, cluster, association algorithms and for visualization of our results. The WEKA is a collection of machine learning algorithms for data mining tasks.

Critical industrial infrastructures are currently facing increasing cyberspace threats in their underlying information and communication systems. The advanced monitoring, control, and management functionalities of the industrial systems firmly rely on the reliable and secure operations of the industrial control system (ICS) network. This article characterizes the ICS network traffic and presents a scalable and efficient solution for real-time ICS network traffic anomaly detection, considering various forms of ICS anomaly events. The events due to the cyberattacks, malicious operating behaviors, and network anomalies can be effectively detected without sophisticated computational requirements and retrieval of communication protocols. The proposed hybrid statistical-machine learning model integrates a seasonal autoregressive integration moving average (SARIMA)-based dynamic threshold model and a long short-term memory (LSTM) model to jointly identify the abnormal traffic patterns with low false omission rates. The proposed solution is extensively evaluated at a realistic ICS cyber–physical system (CPS) testbed, and the numerical results confirm its high detection accuracy and low computational complexity. Note to Practitioners—This article was motivated by the challenge of real-time anomaly detection in industrial cyber–physical systems (CPSs). The existing industrial control system (ICS) network anomaly detection solutions are generally carried out based on a single model based on the historian database and cannot dynamically classify the abnormal conditions in a real-time fashion. A novel hybrid statistical-machine learning model is developed that integrates a seasonal autoregressive integration moving average (SARIMA)-based dynamic threshold model and a long short-term memory (LSTM) model to jointly identify the anomalous events through traffic pattern analysis. The proposed anomaly detection solution can efficiently provide accurate detection for cyberattacks, malicious operating behaviors, and network anomalies while meeting the real-time requirements of ICS networks. The proposed solution can be deployed in the realistic ICS CPSs, e.g., the power generation system, gas pipeline systems, and urban railway transportation systems. The preliminary numerical results obtained from the ICS-CPS testbed suggested that it can provide high detection accuracy with low computational complexity and, hence, can be adopted with minimal deployment hurdles.

The motivation behind exploring anomaly detection in self-organizing networks lies in the evolving landscape of telecommunications and network management. Conventional methods for identifying network anomalies often struggle to adapt to the dynamic and complex nature of modern self-organizing networks. The problem addressed in this research is the efficacy of anomaly detection methods in self-organizing networks (SONs) within the context of telecommunications and network management. As SONs become increasingly prevalent to meet the demands of modern, highly dynamic wireless communication systems, the need for robust anomaly detection mechanisms is paramount. Conventional anomaly detection approaches in SONs are often based on predefined rules and thresholds, which may struggle to adapt to the intricate and rapidly evolving network behaviors. These methods can result in false alarms, missed anomalies, and inefficient resource allocation. Furthermore, emerging SONs incorporate a multitude of diverse technologies, including 5G, IoT, and edge computing, compounding the complexity of anomaly detection. Contemporary machine learning techniques hold promise in addressing these challenges by enabling the automatic and adaptive detection of anomalies, leveraging the abundance of data generated in SONs. However, the suitability, performance, and scalability of these methods in dynamic and large-scale SON environments remain critical concerns. This research aims to compare and evaluate conventional anomaly detection methods against contemporary machine learning approaches in SONs to assess their accuracy, efficiency, and adaptability. The goal is to provide insights into the most effective anomaly detection strategies, ultimately enhancing network stability, minimizing downtime, and ensuring the secure and efficient operation of modern telecommunications systems.

Ensuring a secure network environment is crucial, especially with the increasing number of threats and attacks on digital systems. Implementing effective security measures, such as anomaly detection can help detect any abnormal traffic patterns. Several statistical and machine learning approaches are used to detect network anomalies including robust statistical methods. Robust methods can help identify abnormal traffic patterns and distinguish them from normal traffic accurately. In this study, a robust Principal Component Analysis (PCA) method called ROBPCA which is known for its extensive use in the literature of chemometrics and genetics is utilized for detecting network anomalies and compared with another robust PCA method called PCAGRID. The anomaly detection performances of these methods are evaluated by injecting synthetic traffic volume into a well-known traffic matrix. According to the application results, when the normal subspace is contaminated with large anomalies the ROBPCA method provides much better performance in detecting anomalies.