An Enhanced CUSUM Algorithm for Anomaly Detection

A large amount of work has been done on the KDD 99 dataset, most of which include the use of a hybrid anomaly and misuse detection model done in parallel with each other. In order to further classify the intrusions, our approach to network intrusion detection includes the use of two different anomaly detection models followed by misuse detection applied to the combined output obtained from the previous step. The end goal of this is to verify the anomalies detected by the anomaly detection algorithm and clarify whether they are actually intrusions or random outliers from the trained normal (and thus to try and reduce the number of false positives). We aim to detect a pattern in this novel intrusion technique itself, and not the handling of such intrusions. The intrusions were detected to a very high degree of accuracy.

Traditional Intrusion Detection Systems (IDS) can be complemented by an Anomaly Detection Algorithm (ADA) to also identify unknown attacks. We argue that, as each ADA has its own strengths and weaknesses, it might be beneficial to rely on multiple ADAs to obtain deeper insights. ADAs are very resource intensive; thus, real-time detection with multiple algorithms is even more challenging in high-speed networks. To handle such high data rates, we developed a controlled load allocation scheme that adaptively allocates multiple ADAs on a multi-core system. The key idea of this concept is to utilize as many algorithms as possible without causing random packet drops, which is the typical system behavior in overload situations. We developed a proof of concept anomaly detection framework with a sample set of ADAs. Our experiments confirm that the detection performance can substantially benefit from using multiple algorithms and that the developed framework is also able to cope with high packet rates.

In the field of launch vehicles, high launch density, heavy work intensity and large data volume have brought the traditional automatic anomaly detection with incomplete coverage, and complex design of criterion, which influencing performance evaluation of launch vehicle. In order to fully mine the data variation law implied in the massive telemetry data, the intelligent anomaly detection algorithm is proposed and designed as a beneficial supplement to the traditional anomaly detection algorithm, and improve the anomaly detection coverage and efficiency. With the telemetry data generated by the liquid launch vehicle taken as the research object, and the intelligent anomaly detection algorithm is compared with the traditional algorithm. It is shown that the algorithm proposed adapts to the anomaly detection scenarios with frequency anomaly and frame loss, which may be difficult for traditional algorithms to design anomaly detection criteria, and the average anomaly detection performance is improved by nearly 30% and the variance is reduced by 85%, providing a reference for improving anomaly detection system and applying intelligent anomaly detection algorithms in aerospace.

Anomaly detection in big data is a key problem in the big data analytics domain. In this paper, the definitions of anomaly detection and big data were presented. Due to the sampling and storage burden and the inadequacy of privacy protection of anomaly detection based on uncompressed data, compressive sensing theory was introduced and used in the anomaly detection algorithm. The anomaly detection criterion based on wavelet packet transform and statistic process control theory was deduced. The proposed anomaly detection technique was used for through-wall human detection to demonstrate the effectiveness. The experiments for detecting humans behind a brick wall and gypsum based on ultra-wideband radar signal were carried out. The results showed that the proposed anomaly detection algorithm could effectively detect the existence of a human being through compressed signals and uncompressed data.

Anomaly detection in streaming data is an important task for many real-world applications, such as network security, fraud detection, and system monitoring. However, streaming data often exhibit concept drift, which means that the data distribution changes over time. This poses a significant challenge for many anomaly detection algorithms, as they need to adapt to the evolving data to maintain high detection accuracy. Existing streaming anomaly detection algorithms lack a unified evaluation framework that validly assesses their performance and robustness under different types of concept drifts and anomalies. In this paper, we conduct a systematic technical review of the state-of-the-art methods for anomaly detection in streaming data. We propose a new data generator, called SCAR (Streaming data generator with Customizable Anomalies and concept dRifts), that can synthesize streaming data based on synthetic and real-world datasets from different domains. Furthermore, we adapt four static anomaly detection models to the streaming setting using a generic reconstruction strategy as baselines, and then compare them systematically with 9 existing streaming anomaly detection algorithms on 76 synthesized datasets that have various types of anomalies and concept drifts. The challenges and future research directions for anomaly detection in streaming data are also presented.

In this article, we propose an online and unsupervised anomaly detection algorithm for streaming data using an array of sliding windows and the probability density-based descriptors (PDDs) (based on these windows). This algorithm mainly consists of three steps: 1) we use a main sliding window over streaming data and segment this window into an array of nonoverlapping subwindows; 2) we propose the PDDs with dimension reduction, based on the kernel density estimation, to estimate the probability density of data in each subwindow; and 3) we design the distance-based anomaly detection rule to determine whether the current observation is anomalous. The experimental results and performances are presented based on the Numenta anomaly benchmark. Compared with the anomaly detection algorithm using the hierarchical temporal memory proposed by Numenta (which outperforms a wide range of other anomaly detection algorithms), our algorithm can perform better in many cases, that is, with higher detection rates and earlier detection for contextual anomalies and concept drifts.

The detection of anomalies in network traffic is a crucial issue affecting the security of Internet users. A statistical network anomaly detection algorithm is a promising way of detecting such anomalies, however, it has to be given appropriate parameters for accurate detection and identification. In general, it is very difficult to obtain appropriate parameter settings a priori, because network traffic is not stable in time or space. Thus, although many anomaly detection methods have been proposed, there has been little discussion about their parameter tunings. In this paper, we investigate an automatic and dynamic parameter tuning of a statistical network traffic anomaly detection method. In particular, we clarify whether one can consistently use the best parameter fixed for a certain instance; this choice clearly depends on the macroscopic and dynamic behavior of Internet traffic anomalies. We ascertain the appropriate learning period for setting a parameter of an anomaly detection algorithm based on a sketch and multi-scale gamma-function model by using real network traces measured in a trans-Pacific link over a period of six months. The main results of our study are as follows: (1) Without learning, the best parameter varies day by day. (2) With a longer learning period, the best parameter setting is affected by significant data during the learning period. (3) The appropriate period of the learning is about 3 days. (4) The performance degradation from introducing dynamic parameter tuning is 17% in the best case.

Network anomaly detection has become the promising aspect of intrusion detection. The existing anomaly detection models depict the detection profiles with a static way, which lack good adaptability and interoperability. Furthermore, the detection rate is low, so they are difficult to be deployed the realtime detection under the high-speed network environment. In this paper, the excellent mechanisms of self-learning and adaptability in the human immune system are referred and a dynamic anomaly detection algorithm with immune negative selection, named as DADAI, is proposed. The concepts and formal definitions of antigen, antibody, and memory cells in the network security domain are given; the dynamic clonal principle of antibody is integrated; the mechanism of immune vaccination is discussed, and the dynamic evolvement formulations of detection profiles are established (including the detection profiles' dynamic generation and extinction, dynamic learning, dynamic transformation, and dynamic self-organization), which will accomplish that the detection profiles dynamically synchronize with the real network environment. Both our theoretical analysis and experimental results show that DADAI is a good solution to network anomaly detection, which increase the veracity and timeliness on anomaly detection problem.

Anomaly detection in sensor time series is a crucial aspect for raw data cleaning in gas turbine industry. In addition to efficiency, a successful methodology for industrial applications should be also characterized by ease of implementation and operation. To this purpose, a comprehensive and straightforward approach for Detection, Classification and Integrated Diagnostics of Gas Turbine Sensors (named DCIDS) is proposed in this paper. The tool consists of two main algorithms, i.e. the Anomaly Detection Algorithm (ADA) and the Anomaly Classification Algorithm (ACA). The ADA identifies anomalies according to three different levels of filtering based on gross physics threshold application, inter-sensor statistical analysis (sensor voting) and single-sensor statistical analysis. Anomalies in the time series are identified by the ADA, together with their characteristics, which are analyzed by the ACA to perform their classification. Fault classes discriminate among anomalies according to their time correlation, magnitude and number of sensors in which an anomaly is contemporarily identified. Results of anomaly identification and classification can subsequently be used for sensor diagnostic purposes. The performance of the tool is assessed in this paper by analyzing two temperature time series with redundant sensors taken on a Siemens gas turbine in operation. The results show that the DICDS is able to identify and classify different types of anomalies. In particular, in the first dataset, two severely incoherent sensors are identified and their anomalies are correctly classified. In the second dataset, the DCIDS tool proves to be capable of identifying and classifying clustered spikes of different magnitudes.

In the resilient and reliable electrical power system, the condition of high voltage insulation plays a crucial role. In the field of high voltage insulation integrity, the partial discharge (PD) inception and development trends are essential for assessment criteria in diagnostics systems. The observed trend to employ more and more sophisticated algorithms with machine learning features and artificial intelligence (AI) elements is observed everywhere. The classification and identification of features in PD images is perceived as a critical requirement for an effective high voltage insulation diagnosis. In this context, techniques allowing for anomaly detection, trends observation, and feature extraction in partial discharge patterns are important. In this paper, the application of few algorithms belonging to image processing, machine learning and optical flow is presented. The feature extraction refers to image segmentation and detection of coherent forms in the images. The anomaly detection algorithms can trigger early detection of the trend changes or the appearance of a new discharge form, and hence are suitable for PD monitoring applications. Anomaly detection can also handle transients and disturbances that appear in the PD image as an indication of an abnormal state. The future monitoring systems should be equipped with trend evolution algorithms. In this context, two examples of insulation aging and application of PD-based monitoring are shown. The first one refers to deep convolutional neural networks used for classification of deterioration stages in high voltage insulation. The latter one demonstrates application of optical flow approach for motion detection in partial discharge images. The motivation for the research was the strive to machine-controlled pattern analysis, leading towards intelligent PD-based diagnostics.

Anomaly detection of time series plays an important role in reliability systems engineering. However, in practical application, there is no precisely defined boundary between normal and anomalous behaviors in different application scenarios. Therefore, different anomaly detection algorithms and processes ought to be adopted for time series in different situation. Although such strategy improve the accuracy of anomaly detection, it takes a lot of time for practitioners to configure various algorithms to millions of series, which greatly increases the development and maintenance cost of anomaly detection processes. In this paper, we propose CRATOS which is a self-adapt algorithms that extract features from time series, and then cluster series with similar features into one group. For each group we utilize evolutionary algorithm to search the best anomaly detection methods and processes. Our methods can significantly reduce the cost of development and maintenance of anomaly detection. According to experiments, our clustering methods achieves the state-of-art results. The accuracy of the anomaly detection algorithms in this paper is 85.1%.

Exploiting fuzzy rough entropy to detect anomalies

Detecting anomalies during the operation of a network is an important aspect of network management and security. Recent development of high-performance embedded processing systems allow traffic monitoring and anomaly detection in real-time. In this paper, we show how such processing capabilities can be used to run several different anomaly detection algorithms in parallel on thousands of different traffic subclasses. The main challenge in this context is to manage and aggregate the vast amount of data generated by these processes. We propose (1) a novel aggregation process that uses continuous anomaly information (rather than binary outputs) from existing algorithms and (2) an anomaly tree representation to illustrate the state of all traffic subclasses. Aggregated anomaly detection results show a lower false positive and false negative rate than any single anomaly detection algorithm.

Cloud technology has brought great convenience to enterprises as well as customers. System logs record notable events and are becoming valuable resources to track and investigate system status. Detecting anomaly from logs as fast as possible can improve the quality of service significantly. Although many machine learning algorithms (e.g., SVM, Logistic Regression) have high detection accuracy, we find that they assume data are clean and might have high training time. Facing these challenges, in this paper, we propose Robust Online Evolving Anomaly Detection (ROEAD) framework which adopts Robust Feature Extractor (RFE) to remove the effects of noise and Online Evolving Anomaly Detection (OEAD) to dynamic update parameters. We propose Online Evolving SVM (OES) algorithm as the example of online anomaly detection methods. We analyze the performance of OES in theory and prove the performance difference between OES and the best hypothesis tends to zero as time goes infinity. We compare the performance of ROEAD against state-of-the-art anomaly detection algorithms using public log datasets. The results demonstrate that ROEAD is able to remove the effects of noise and OES can improve the detection accuracy by more than 40%.

Real-time anomaly detection based on long short-Term memory and Gaussian Mixture Model