Statistical wavelet-based anomaly detection in big data with compressive sensing

The definitions of anomaly detection and big data were presented. Due to the sampling and storage burden of anomaly detection in big data, compressive sensing theory was introduced and used in anomaly detection algorithm. The anomaly detection criterion based on wavelet packet transform and statistic process control theory was deduced. The anomaly detection method was used for through wall human detection. The experiments for detecting human behind Brick wall based on UWB radar signal was carried out. The results showed that the proposed anomaly detection algorithm could effectively detect the existence of human being through compressed signals.KeywordsAnomaly detectionBig dataThrough wall human detectionCompressive sensing

Outlier detection (or anomaly detection) is a fundamental task in data mining. Outliers are data that deviate from the norm and outlier detection is often compared to “finding a needle in a haystack.” However, the outliers may generate high value if they are found, value in terms of cost savings, improved efficiency, compute time savings, fraud reduction and failure prevention. Detection can identify faults before they escalate with potentially catastrophic consequences. Big Data refers to large, dynamic collections of data. These vast and complex data appear problematic for traditional outlier detection methods to process but, Big Data provides considerable opportunity to uncover new outliers and data relationships. This chapter highlights some of the research issues for outlier detection in Big Data and covers the solutions used and research directions taken along with an analysis of some current outlier detection approaches for Big Data applications.

The use of the big data analytics (BDA) platform is increasingly becoming prevalent in the data sciences. However, BDA processes consume resources and time excessively. Automating BDA processes is a cognitive approach to the BDA domain, which is most impaired by its heavy consumption of time and resources. However, the BDA workflow is highly dependent on diversified constraints because of the high variability, veracity, and volume of data processing to accomplish highly influential and sophisticated requirements. The workflow has to pass rigorous and diverse data mining steps, each step contains several tasks and these tasks made by many sub tasks, that it must be accomplished to progress to the next step in the workflow. This increases the available solution space for BDA processes. The intelligent heuristic approach is needed to address the domain-specific concerns which are large solution space, and awareness of constraints are caused the BDA planning. Therefore, we propose to use GraphPlan-based dynamic workflow generation for the BDA domain. Our empirical studies prove that the proposed sophisticated method satisfied planning requirements and outperformed a related planning technique.

A large amount of work has been done on the KDD 99 dataset, most of which include the use of a hybrid anomaly and misuse detection model done in parallel with each other. In order to further classify the intrusions, our approach to network intrusion detection includes the use of two different anomaly detection models followed by misuse detection applied to the combined output obtained from the previous step. The end goal of this is to verify the anomalies detected by the anomaly detection algorithm and clarify whether they are actually intrusions or random outliers from the trained normal (and thus to try and reduce the number of false positives). We aim to detect a pattern in this novel intrusion technique itself, and not the handling of such intrusions. The intrusions were detected to a very high degree of accuracy.

In this work, we present the first distributed solution for the Local Outlier Factor (LOF) method -- a popular outlier detection technique shown to be very effective for datasets with skewed distributions. As datasets increase radically in size, highly scalable LOF algorithms leveraging modern distributed infrastructures are required. This poses significant challenges due to the complexity of the LOF definition, and a lack of access to the entire dataset at any individual compute machine. Our solution features a distributed LOF pipeline framework, called DLOF. Each stage of the LOF computation is conducted in a fully distributed fashion by leveraging our invariant observation for intermediate value management. Furthermore, we propose a data assignment strategy which ensures that each machine is self-sufficient in all stages of the LOF pipeline, while minimizing the number of data replicas. Based on the convergence property derived from analyzing this strategy in the context of real world datasets, we introduce a number of data-driven optimization strategies. These strategies not only minimize the computation costs within each stage, but also eliminate unnecessary communication costs by aggressively pushing the LOF computation into the early stages of the DLOF pipeline. Our comprehensive experimental study using both real and synthetic datasets confirms the efficiency and scalability of our approach to terabyte level data.

This paper compares two anomaly detection methods, comparing the Z-score statistical technique with autoen-coders for big datasets, which are crucial for industries like manufacturing, energy, and transportation to maintain smooth operations and avoid costly disruptions. Autoencoders outperformed Z-score statistical technique in anomaly detection on big datasets, achieving higher precision (0.94), F1-score (0.97), and recall (1.00) compared to Z-score statistical technique. This highlights autoencoders' superior ability to accurately identify anomalies, making them more effective for robust anomaly detection in complex data environments.

In the field of launch vehicles, high launch density, heavy work intensity and large data volume have brought the traditional automatic anomaly detection with incomplete coverage, and complex design of criterion, which influencing performance evaluation of launch vehicle. In order to fully mine the data variation law implied in the massive telemetry data, the intelligent anomaly detection algorithm is proposed and designed as a beneficial supplement to the traditional anomaly detection algorithm, and improve the anomaly detection coverage and efficiency. With the telemetry data generated by the liquid launch vehicle taken as the research object, and the intelligent anomaly detection algorithm is compared with the traditional algorithm. It is shown that the algorithm proposed adapts to the anomaly detection scenarios with frequency anomaly and frame loss, which may be difficult for traditional algorithms to design anomaly detection criteria, and the average anomaly detection performance is improved by nearly 30% and the variance is reduced by 85%, providing a reference for improving anomaly detection system and applying intelligent anomaly detection algorithms in aerospace.

Traditional Intrusion Detection Systems (IDS) can be complemented by an Anomaly Detection Algorithm (ADA) to also identify unknown attacks. We argue that, as each ADA has its own strengths and weaknesses, it might be beneficial to rely on multiple ADAs to obtain deeper insights. ADAs are very resource intensive; thus, real-time detection with multiple algorithms is even more challenging in high-speed networks. To handle such high data rates, we developed a controlled load allocation scheme that adaptively allocates multiple ADAs on a multi-core system. The key idea of this concept is to utilize as many algorithms as possible without causing random packet drops, which is the typical system behavior in overload situations. We developed a proof of concept anomaly detection framework with a sample set of ADAs. Our experiments confirm that the detection performance can substantially benefit from using multiple algorithms and that the developed framework is also able to cope with high packet rates.

Anomaly detection in streaming data is an important task for many real-world applications, such as network security, fraud detection, and system monitoring. However, streaming data often exhibit concept drift, which means that the data distribution changes over time. This poses a significant challenge for many anomaly detection algorithms, as they need to adapt to the evolving data to maintain high detection accuracy. Existing streaming anomaly detection algorithms lack a unified evaluation framework that validly assesses their performance and robustness under different types of concept drifts and anomalies. In this paper, we conduct a systematic technical review of the state-of-the-art methods for anomaly detection in streaming data. We propose a new data generator, called SCAR (Streaming data generator with Customizable Anomalies and concept dRifts), that can synthesize streaming data based on synthetic and real-world datasets from different domains. Furthermore, we adapt four static anomaly detection models to the streaming setting using a generic reconstruction strategy as baselines, and then compare them systematically with 9 existing streaming anomaly detection algorithms on 76 synthesized datasets that have various types of anomalies and concept drifts. The challenges and future research directions for anomaly detection in streaming data are also presented.

With the increasingly severe network security situation, advanced network traffic anomaly detection techniques are urgently needed. This paper provides a comprehensive survey of the research status and latest progress in the field of network anomaly detection. Firstly, we introduce the basic concepts, common methods, and challenges of network traffic analysis, which lays the foundation for anomaly detection. Then, we systematically summarize the mainstream techniques in the anomaly detection field, including statistical methods, machine learning methods, deep learning methods, and behavior analysis methods, analyzing their basic principles, representative works, advantages and disadvantages, and applicable scenarios. Next, we focus on discussing the hybrid methods in the anomaly detection field, elaborating on the motivations, common strategies, and representative works of hybrid methods, and pointing out that hybrid methods are an important development direction for anomaly detection. In addition, the paper also summarizes the application effects of several types of methods in practical network security tasks and makes a quantitative comparison in tabular form. Finally, we prospect the future development trends of network anomaly detection techniques, proposing goals such as intelligentization, automation, federalization, and interpretability, while analyzing the challenges faced by anomaly detection, including data heterogeneity, complexity of security threats, model robustness, privacy protection, and interpretability. We argue that network anomaly detection requires interdisciplinary integration, strengthening of security big data governance, and a shift from passive defense to active immunity. As the strategic position of cyberspace security becomes increasingly prominent, driven by disruptive technologies such as big data, artificial intelligence, and blockchain, network anomaly detection will surely usher in new development opportunities and challenges.

Anomaly detection has been extensively studied in the last two decades. Features are usually selected or created at first for characterizing behaviours of networks, users or systems, and then anomaly detection algorithms are developed and applied. However, such traditional network anomaly detection systems detect a limited number of attack types and report a huge number of false alarms due to lack of proper features and inherent weakness of anomaly detection algorithms. In order to minimize the number of false alerts and maximize the detection accuracy, we propose in this chapter an enhanced CUSUM algorithm for network anomaly detection, modelling various features from different sources and reporting alerts according to some decision strategies. The experimental evaluation with the 1999 DARPA intrusion detection evaluation dataset shows an empirical study of applying the proposed enhanced CUSUM algorithm for detecting successfully some network attacks.

Network anomaly detection has become the promising aspect of intrusion detection. The existing anomaly detection models depict the detection profiles with a static way, which lack good adaptability and interoperability. Furthermore, the detection rate is low, so they are difficult to be deployed the realtime detection under the high-speed network environment. In this paper, the excellent mechanisms of self-learning and adaptability in the human immune system are referred and a dynamic anomaly detection algorithm with immune negative selection, named as DADAI, is proposed. The concepts and formal definitions of antigen, antibody, and memory cells in the network security domain are given; the dynamic clonal principle of antibody is integrated; the mechanism of immune vaccination is discussed, and the dynamic evolvement formulations of detection profiles are established (including the detection profiles' dynamic generation and extinction, dynamic learning, dynamic transformation, and dynamic self-organization), which will accomplish that the detection profiles dynamically synchronize with the real network environment. Both our theoretical analysis and experimental results show that DADAI is a good solution to network anomaly detection, which increase the veracity and timeliness on anomaly detection problem.

In this article, we propose an online and unsupervised anomaly detection algorithm for streaming data using an array of sliding windows and the probability density-based descriptors (PDDs) (based on these windows). This algorithm mainly consists of three steps: 1) we use a main sliding window over streaming data and segment this window into an array of nonoverlapping subwindows; 2) we propose the PDDs with dimension reduction, based on the kernel density estimation, to estimate the probability density of data in each subwindow; and 3) we design the distance-based anomaly detection rule to determine whether the current observation is anomalous. The experimental results and performances are presented based on the Numenta anomaly benchmark. Compared with the anomaly detection algorithm using the hierarchical temporal memory proposed by Numenta (which outperforms a wide range of other anomaly detection algorithms), our algorithm can perform better in many cases, that is, with higher detection rates and earlier detection for contextual anomalies and concept drifts.

The detection of anomalies in network traffic is a crucial issue affecting the security of Internet users. A statistical network anomaly detection algorithm is a promising way of detecting such anomalies, however, it has to be given appropriate parameters for accurate detection and identification. In general, it is very difficult to obtain appropriate parameter settings a priori, because network traffic is not stable in time or space. Thus, although many anomaly detection methods have been proposed, there has been little discussion about their parameter tunings. In this paper, we investigate an automatic and dynamic parameter tuning of a statistical network traffic anomaly detection method. In particular, we clarify whether one can consistently use the best parameter fixed for a certain instance; this choice clearly depends on the macroscopic and dynamic behavior of Internet traffic anomalies. We ascertain the appropriate learning period for setting a parameter of an anomaly detection algorithm based on a sketch and multi-scale gamma-function model by using real network traces measured in a trans-Pacific link over a period of six months. The main results of our study are as follows: (1) Without learning, the best parameter varies day by day. (2) With a longer learning period, the best parameter setting is affected by significant data during the learning period. (3) The appropriate period of the learning is about 3 days. (4) The performance degradation from introducing dynamic parameter tuning is 17% in the best case.

SummaryIn recent years, wireless sensor networks are pervasive and are generating tons of data every second. Performing outlier detection to detect faulty sensors from such a large amount of data becomes a challenging task. Most of the existing techniques for outlier detection in wireless sensor networks concentrate only on contents of the data source without considering correlation among different data attributes. Moreover, these methods are not scalable to big data. To address these 2 limitations, this paper proposes an outlier detection approach based on correlation and dynamic SMO (sequential minimal optimization) regression that is scalable to big data. Initially, correlation is used to find out strongly correlated attributes and then the point anomalous nodes are detected using dynamic SMO regression. For fast processing of big data, Hadoop MapReduce framework is used. The experimental analysis demonstrates that the proposed approach efficiently detects the point and contextual anomalies and reduces the number of false alarms. For experiments, real data of sensors used in body sensor networks are taken from Physionet database.