An Incremental Learning-Based Anomaly Detection Model for High-Dimensional Time Series

Many local outlier detection algorithms have been proposed inspired by the idea of local outlier factor (LOF). However, they often have low detection performance and are sensitive to neighborhood size because there is a major defect in their calculation formulas of outlier degree and the kNN (k-nearest neighbors) method is widely used to quantify a neighborhood of an instance. To address these issues, we define a novel nearest neighbors tree (NNT) to measure a neighborhood of an instance. Meanwhile, we propose a local structure outlier factor (LSOF), which score each local structure instead of each data point and report the top-scored local structures as anomalous local structures, where outliers and groups of outliers are easily divided according to characteristics of the NNT. Our experimental results demonstrate that the competitive behavior of our method on both synthetic and real-world datasets.

Outlier detection is an important task in data mining with numerous applications, including credit card fraud detection, video surveillance, etc. A recent work on outlier detection has introduced a novel notion of local outlier in which the degree to which an object is outlying is dependent on the density of its local neighborhood, and each object can be assigned a Local Outlier Factor (LOF) which represents the likelihood of that object being an outlier. Although the concept of local outliers is a useful one, the computation of LOF values for every data objects requires a large number of κ-nearest neighbors searches and can be computationally expensive. Since most objects are usually not outliers, it is useful to provide users with the option of finding only n most outstanding local outliers, i.e., the top-n data objects which are most likely to be local outliers according to their LOFs. However, if the pruning is not done carefully, finding top-n outliers could result in the same amount of computation as finding LOF for all objects. In this paper, we propose a novel method to efficiently find the top-n local outliers in large databases. The concept of "micro-cluster" is introduced to compress the data. An efficient micro-cluster-based local outlier mining algorithm is designed based on this concept. As our algorithm can be adversely affected by the overlapping in the micro-clusters, we proposed a meaningful cut-plane solution for overlapping data. The formal analysis and experiments show that this method can achieve good performance in finding the most outstanding local outliers.

Many real-world time series data exhibit outliers and noise like abnormal machine operation, emergency monitoring etc. Anomaly detection is the most important way to deal with it. During all the solutions, Prophet is an effective model for large-scale time series with complicated patterns in anomaly detection and forecasting scenarios. However, Prophet model is easy to fall into overfitting at special time points, which leads to large errors in some behaviors. In this paper we propose a robust complex univariate time series anomaly detection framework RobustProphet based on forecasting. Specifically, in the step of forecasting, linear regression function is adopted to fuse Prophet and XGBoost model which use L2 regularization to resolve the overfitting problem and optimize the prediction results. Once the final forecasting results are generated, we offer a nonparametric dynamic thresholding approach to evaluate residuals to detect anomalies in original time series. We made a comparative study between our proposed model and other classical anomaly detection methods such as Local Outlier Factor (LOF), Isolation Forest on datasets from Yahoo benchmark datasets. The experimental results show that our proposed anomaly detection model has better precision and higher AUC.

[1] A recent article by Rosenlof and Reid [2008] (hereinafter referred to as RR08) examines the relationship in the tropics between long-term cooling in the stratosphere and warming of the sea surface, with special emphasis in the western Pacific. In the stratosphere, radiosonde observations seem to show that most of the cooling took place since the mid-1990s, apparently coincident to a considerable extent with an increase in the rate of warming of sea surface temperatures (SSTs). A link was proposed by RR08 whereby the increase in SSTs leads to enhanced convection, which is responsible, dynamically, for cooling in the stratosphere. [2] The stratospheric analyses of RR08 rely considerably on tropical radiosonde observations, particularly those from several island stations in the western tropical Pacific. There is extensive evidence that the long-term variation in worldwide radiosonde temperatures is affected critically by inhomogeneities introduced through changes in instruments and measurement practices [e.g., Gaffen, 1994; Gaffen et al., 2000; Lanzante et al . , 2003a, 2003b; Climate Change Science Program (CCSP), 2006], a fact which is acknowledged by RR08. However, they minimize the importance of any such inhomogeneities in the six stations located in the ‘‘warm pool’’ region which they employ [RR08, paragraph 10]: ‘‘However, for our analysis, and in particular for changes noted post-1990, these discontinuities are not sufficient to preclude the use of these stations.’’ The purpose of this comment is to demonstrate that in fact such discontinuities are present, post-1990, and critically affect the evaluation of stratospheric temperatures made by RR08. [3] In their Figures 2, 3, and 5, RR08 display temperature time series for several levels in the lower stratosphere for the four stations located in the heart of the western Pacific warm pool. They note that while most of the earlier part of the records show little long-term trend, a pronounced cooling is seen for about the last 10 years of the record ( 1995–2005). Furthermore, they note an abrupt rise in temperature in early 1999 that lasts for about 2 years. As further evidence to bolster their findings, they note [RR08, paragraph 12] the similarity of the time series from the different stations: ‘‘. . . the overall similarity of the time series in Figure 2 shows that the assumption of homogeneity over the warm pool region is reasonable.’’ Unfortunately, there is considerable evidence to dispute the veracity of these claims. [4] The apparent substantial cooling during the final 10 years of the record can be explained largely by a change in radiosonde instrument type, specifically the transition from Viz to Vaisala, hereafter referred to as the VVT. Abrupt cooling upon introduction of Vaisala radiosondes was noted by Parker et al. [1997] for stations in Australia and New Zealand in the mid to late 1980s, and for U.S. controlled tropical Pacific stations beginning in 1995 [Stendel et al., 2000]. More generally, there has been a systematic worldwide change to Vaisala radiosondes since the 1980s with an expected artificial cooling, especially in the stratosphere [Lanzante et al., 2003a; Christy et al., 2003; CCSP, 2006] (see also Integrated Global Radiosonde Archive (IGRA) station history metadata available at http:// www1.ncdc.noaa.gov/pub/data/igra/igra-metadata.txt). [5] The IGRA metadata indicate that the VVT occurs in November or December 1995 for five of the six stations shown in Figure 1 of RR08 (Lihue, Truk, Koror, Yap, and Pago Pago). The literature cited above suggests that the major part of the drop in stratospheric temperature after 1995 (as seen quite clearly in Figure 5 of RR08) is almost certainly artificial. To further illustrate this point, Figure 1a shows several time series of stratospheric temperatures at Koror. The thick black curve is based on the IGRA data [Durre et al., 2006], which were used by RR08. Note that the IGRA data, although of very high quality (i.e., they have been carefully checked for consistency in time and in the vertical, random errors and outliers, as well as duplicate soundings, etc.), have not been adjusted to account for artificial inhomogeneities. A number of teams have produced adjusted radiosonde temperature time series, each using a different homogenization method. Also shown in Figure 1a are corresponding adjusted time series (HadAT2) [Thorne et al., 2005], RATPAC [Free et al., 2005], IUK [Sherwood et al., 2008], RAOBCORE 1.4 [Haimberger et al., 2008], and RICH [Haimberger et al., 2008] produced by these teams. In comparing the IGRA with the various adjusted time series in Figure 1a it is seen that the adjusted versions all show much less long-term cooling, as indicated by the accompanying trend lines. A closer look at Figure 1a reveals that most of the difference is attributable to the transition across 1995. The adjusted time series shows a JOURNAL OF GEOPHYSICAL RESEARCH, VOL. 114, D12104, doi:10.1029/2008JD010542, 2009 Click Here for Full Article

After the local outlier factor was first proposed, there is a large family of local outlier detection approaches derived from it. Since the existing approaches only focus on the extent of overall separation between an object and its neighbors, and ignore the degree of dispersion between them, the precision of these approaches will be affected by various degrees in the scattered datasets. In addition, the outlier data occupy a relatively small amount in the dataset, but the existing approaches need to perform local outlier factor calculation on all data during the outlier detection, which greatly reduces the efficiency of the algorithms. In this paper, we redefine a local outlier factor called local deviation coefficient (LDC) by taking full advantage of the distribution of the object and its neighbors. And then, we propose a safe non-outlier objects elimination approach named as rough clustering based on multi-level queries (RCMLQ) to preprocess the datasets to eliminate the non-outlier objects to the utmost. Finally, an efficient local outlier detection approach named as efficient density-based local outlier detection for scattered data (E2DLOS) is proposed based on the LDC and RCMLQ. The RCMLQ greatly reduces the amount of data that needs to be quantified for local outlier factor and the LDC is more sensitive to the degree of anomaly of the scattered datasets, and so the E2DLOS improves the existing local outlier detection approaches in time efficiency and detection accuracy. Experiments show that the LDC can better reflect the true abnormal situations of the data for the scattered datasets. And the RCMLQ can be used in parallel with the traditional methods of improving the efficiency of the nearest neighbor search, which can further improve the efficiency of the E2DLOS algorithm by about 16%.

The automated detection of sequential anomalies in time series is an essential task for many applications, such as the monitoring of technical systems, fraud detection in high-frequency trading, or the early detection of disease symptoms. All these applications require the detection to find all sequential anomalies possibly fast on potentially very large time series. In other words, the detection needs to be effective, efficient and scalable w.r.t. the input size. Series2Graph is an effective solution based on graph embeddings that are robust against re-occurring anomalies and can discover sequential anomalies of arbitrary length and works without training data. Yet, Series2Graph is no t scalable due to its single-threaded approach; it cannot, in particular, process arbitrarily large sequences due to the memory constraints of a single machine. In this paper, we propose our distributed anomaly detection system, short DADS, which is an efficient and scalable adaptation of Series2Graph. Based on the actor programming model, DADS distributes the input time sequence, intermediate state and the computation to all processors of a cluster in a way that minimizes communication costs and synchronization barriers. Our evaluation shows that DADS is orders of magnitude faster than S2G, scales almost linearly with the number of processors in the cluster and can process much larger input sequences due to its scale-out property.

Outlier detection is the process of detecting the data objects which are grossly different from or inconsistent with the remaining set of data. Some of the important applications in the field of data mining are fraud detection, customer behavior analysis, and intrusion detection. There are number of good research algorithms for detecting outliers if the entire data is available and algorithms can operate in more than single passes to achieve the required results. Among the existing methods, LOF (Local outlier Factor) a density based method is very efficient in detecting all forms of outliers. LOF algorithm can not be directly applied to the datastream as the large number of nearest neighbor searches, LOF computation and lrd (local reachability distances) can make it highly inefficient for datastream. In this paper we propose a cluster based partitioning algorithm which can divide the stream in safe region and candidate regions. In Second phase apply LOF algorithm over these partitions separately with some slight enhancement for LOF computation over candidate region to achieve accurate results for finding most outstanding outliers. Several experiments on different dataset confirm that our technique can find better outliers with low computational cost than the direct LOF or compared to the other enhancements proposed for LOF.

The article considers the problem of detecting anomalies in the time series of media content metrics obtained from Telegram channels. The task of detecting anomalies is relevant in the context of combating disinformation and analyzing the dynamics of content distribution in social networks. Anomalies in metrics, such as the number of views, shares, comments, and reactions, may indicate manipulative actions, including the use of bots, falsification of reach, or the spread of disinformation. A dataset obtained through the official Telegram API was used for the analysis. A feature of the data is the lack of retrospective values of metrics, which complicates the analysis of their dynamics. This problem was partially solved by collecting data at fixed time intervals after the publication of each post. The collected data was grouped by the channels of origin of the posts and time intervals after publication to ensure the accuracy of the analysis. Since the data were unlabeled, manual processing was used to remove outliers and ensure the reliability of the modeling. The article analyzes the functionality of five Python libraries for detecting anomalies in time series: PyOD, TODS, PySAD, Darts, and Prophet. Their compliance with the requirements was assessed, in particular, with respect to working with time series, processing incomplete data, real-time support, seasonality, and computational efficiency. A comparison was made based on tables and graphs that demonstrate the results of using each library. In particular, the PyOD library is a well-known tool for detecting anomalies, but does not support direct work with time series. TODS has the potential to detect anomalies in streaming data, but its development has been discontinued. PySAD specializes in streaming data analysis, but requires a fixed frequency of input data, which limits its application. The Darts library offers a wide set of algorithms for time series analysis, but requires pre-filling of missing values, which creates an additional load on the model. The best results were achieved using the Prophet model, which is able to work with irregular time series without the need for additional data augmentation. The experiments conducted showed that Prophet provides the best balance between forecast accuracy and computational efficiency. The choice of the amount of historical data for modeling is crucial, since excessive data increases processing time, and insufficient data reduces forecast accuracy.The results of the study indicate that none of the considered libraries is a universal solution for all tasks.However, Prophet showed the greatest potential for detecting anomalies in the time series of Telegram metrics, which makes it the best candidate for further development and adaptation in media content monitoring tasks.

The financial time series is inherently nonlinear and hence cannot be efficiently predicted by using linear statistical methods such as regression. Hence, intelligent predictor has been developed and reported which is suitable for nonlinear time series. But such predictors require that the past financial data are available at the location of the predictor which is not the case in many real-life situations. Hence, when the financial data are available at different places and a single intelligent predictor needs to be developed, the task becomes challenging. In the current work, this problem has been addressed and solved using a low-complexity artificial neural network and employing incremental and diffusion learning strategies. In the current study, distributed prediction of three different types of time series such as exchange rates, stock indices and net asset values has been carried using incremental and diffusion-based learning strategies. The results of different days ahead prediction of two proposed low computational complexity-based functional link artificial neural network are compared with those obtained by conventional intelligent method. The results of simulation-based experiments reveal similar or improved prediction performance of the proposed distributed predictors compared to conventional one. In addition, saving in band width, memory and power are achieved in this method.

Since the Local Outlier Factor (LOF) was first proposed, there is a large family of approaches that is derived from it. For the reason that the existing local outliers detection approaches only focus on the extent of overall separation between an object and its neighbors, and do not pay attention to the degree of dispersion between them, the precision of these approaches will be seriously affected in the scattered data sets for outlier detection. In this paper, we redefine the local outliers by combining the degree of dispersion of the object and its neighbors, and propose a new local outlier detection approach (N2DLOF). Compared to conventional approaches, the outliers obtained by N2DLOF are more sensitive to the degree of anomaly of the scattered data sets. Experiments show that our approach has a significant improvement on outlier detection precision in the case of scattered datasets with similar time complexity. In short, we extend the ecosystem of the local outlier detection approaches from a new perspective.

In this work, we present the first distributed solution for the Local Outlier Factor (LOF) method -- a popular outlier detection technique shown to be very effective for datasets with skewed distributions. As datasets increase radically in size, highly scalable LOF algorithms leveraging modern distributed infrastructures are required. This poses significant challenges due to the complexity of the LOF definition, and a lack of access to the entire dataset at any individual compute machine. Our solution features a distributed LOF pipeline framework, called DLOF. Each stage of the LOF computation is conducted in a fully distributed fashion by leveraging our invariant observation for intermediate value management. Furthermore, we propose a data assignment strategy which ensures that each machine is self-sufficient in all stages of the LOF pipeline, while minimizing the number of data replicas. Based on the convergence property derived from analyzing this strategy in the context of real world datasets, we introduce a number of data-driven optimization strategies. These strategies not only minimize the computation costs within each stage, but also eliminate unnecessary communication costs by aggressively pushing the LOF computation into the early stages of the DLOF pipeline. Our comprehensive experimental study using both real and synthetic datasets confirms the efficiency and scalability of our approach to terabyte level data.

Anomalies in time series, also called “discord,” are the abnormal subsequences. The occurrence of anomalies in time series may indicate that some faults or disease will occur soon. Therefore, development of novel computational approaches for anomaly detection (discord search) in time series is of great significance for state monitoring and early warning of real-time system. Previous studies show that many algorithms were successfully developed and were used for anomaly classification, e.g., health monitoring, traffic detection, and intrusion detection. However, the anomaly detection of time series was not well studied. In this paper, we proposed a long short-term memory- (LSTM-) based anomaly detection method (LSTMAD) for discord search from univariate time series data. LSTMAD learns the structural features from normal (nonanomalous) training data and then performs anomaly detection via a statistical strategy based on the prediction error for observed data. In our experimental evaluation using public ECG datasets and real-world datasets, LSTMAD detects anomalies more accurately than other existing approaches in comparison.

The problem of detecting anomalies in time series has attracted much attention recently due to its numerous applications. In this paper, we proposed a novel framework based on nonlinear methods. Firstly, we use time-dependent URP (Unthreholded Recurrence Plots) to represent time series, so as to capture the nonlinear characteristic of time series. Secondly, we present ELM-AE (Extreme Learning Machine Auto-Encoder) based algorithm to learn the main features of URPs. Finally, we compute residual errors as anomaly score for each time series points. This framework can perform well on complex nonlinear systems using unlabeled datasets.

Current ICH guidelines, e.g.ICH E6 (R3), advocate a risk-based statistical review of clinical trial data to identify anomalies. The open-source R package, clinical trial anomaly spotter (CTAS) has been developed by Bayer and the Intercompany Quality Analytics (IMPALA) consortium, helps detect inconsistencies in subject time series data at both site and subject levels, facilitating timely intervention. CTAS analyzes time series of equal length. Each subject-level time series is summarized as six optional scalars: mean, standard deviation, range, relative unique value count, autocorrelation and local outlier factor. To detect site-level anomalies, sites can be scored using 3 different scoring methods. The performance of the CTAS algorithm was tested using simulations, artificially introducing site anomalies of various types and degrees into clinical trial data sets. We found that CTAS can reliably detect site anomalies depending on the degree of the anomaly introduced. Less complex anomalies such as mean were easier to detect than complex outlier such as local outlier factor. The three scoring methods differed in their ability to detect anomalous sites with a small number of patients and their false positive rates. CTAS is a valuable tool for timely detection of outliers in clinical data, suitable for integration into risk-based strategies. Choosing the appropriate site anomaly scoring method is crucial for handling sites with fewer subjects effectively.

The ability to accurately and consistently discover anomalies in time series is important in many applications. Fields such as finance (fraud detection), information security (intrusion detection), healthcare, and others all benefit from anomaly detection. Intuitively, anomalies in time series are time points or sequences of time points that deviate from normal behavior characterized by periodic oscillations and long-term trends. For example, the typical activity on e-commerce websites exhibits weekly periodicity and grows steadily before holidays. Similarly, domestic usage of electricity exhibits daily and weekly oscillations combined with long-term season-dependent trends. How can we accurately detect anomalies in such domains while simultaneously learning a model for normal behavior? We propose a robust offline unsupervised framework for anomaly detection in seasonal multivariate time series, called AURORA. A key innovation in our framework is a general background behavior model that unifies periodicity and long-term trends. To this end, we leverage a Ramanujan periodic dictionary and a spline-based dictionary to capture both seasonal and trend patterns. We conduct experiments on both synthetic and real-world datasets and demonstrate the effectiveness of our method. AURORA has significant advantages over existing models for anomaly detection, including high accuracy (AUC of up to 0.98), interpretability of recovered normal behavior (100% accuracy in period detection), and the ability to detect both point and contextual anomalies. In addition, AURORA is orders of magnitude faster than baselines.