An improved methodology for outlier detection in dynamic datasets

Robust Detection and Estimation in Dynamic Systems and Statistical Signal Processing: Intersections, Parallel Paths and Applications

The tourist data is very interesting to be studied because the Indonesian tourism sector is an activator of the national economic which is potential to push higher economic growth in the future. Therefore, the forecast about tourist data is very needed for tourism business. The tourist data tend to fluctuate caused by many factors that affect the number of tourists extremely in an area, such as disasters, government regulation, social stability, violence and terrorism. That the extreme data can be assessed using intervention analysis and outlier detection. Intervention model is a time series model that can be used to forecast data consist of intervention of internal and external factors. In the intervention model, there are two kinds of intervention function, i.e., step and pulse functions. Step function is a form of intervention occurred in period of time while the pulse function is a form of intervention occurred only in a certain time. For the outlier detection, there are four types, such as additive outlier (AO), innovational outlier (IO), level shift (LS) and temporary change (TC). As an empirical studies was conducted by the domestic tourists data in Yogyakarta from January 2006 until December 2010 who staying on five-star hotels and motel throughout Yogyakarta. Based on the result of this research, known that the intervention occurred on January 2010 using the pulse function with MSE value 1172. Meanwhile based on the outliers detection, known any five outliers but only four outliers that significant included to the intervention model with MSE value 523,7167. So, the intervention model and outlier detection are chosen as a the best model based on the smallest MSE criterion. Keywords: Domestic tourists, intervention model, pulse function, outlier detection

In recent years, anomaly detection techniques in time-series data have been widely used in manufacturing, cybersecurity, and other fields. Meanwhile, various anomaly detection models based on generative adversarial networks (GAN) are gradually used in time-series anomaly detection tasks. However, there are problems of unstable generator training, missed detection of anomalous data, and inconsistency between the discriminator's discriminant and the anomaly detection target in GAN networks. Aiming at the above problems, the paper proposes a DUAL-ADGAN (Dual Anomaly Detection Generative Adversarial Networks) model for the detection of anomalous data in time series. First, the Wasserstein distance satisfying the Lipschitz constraint is used as the loss function of the data reconstruction module, which improves the stability of the traditional GAN network training. Second, by adding a data prediction module to the DUAL-ADGAN model, the distinction between abnormal and normal samples is increased, and the rate of missing abnormal data in the model is reduced. Third, by introducing the Fence-GAN loss function, the discriminator is aligned with the anomaly detection target, which effectively reduces the anomaly data false detection rate of the DUAL-ADGAN model. Finally, anomaly scores derived from the DUAL-ADGAN model are compared with dynamic thresholds to detect anomalies. The experimental results show that the average F1 of the DUAL-ADGAN model is 0.881, which is better than the other nine baseline models. The conclusions demonstrate that the DUAL-ADGAN model proposed in the paper is more stable in training while effectively solving the problems of anomaly miss detection and discriminator inconsistency with the anomaly detection target in the anomaly detection task.

The need for detection, identification and prediction of anomalies emerged just a short time ago. This task brings multiple challenges that are mainly associated with recording and detection of novel attacks and influences. It draws attention of experts in network security and diagnostics of information and industrial systems. Detection of anomalies in dynamic dataflows largely determines the efficiency of management of computer network information security within information and industrial systems. At the same time, the technology of dynamic dataflow prediction is also very important for building systems intended to detect anomalies in data protection within industrial control systems. Any automated systems are based on available computer capabilities and advances in management theory, mathematical modeling and optimizations methods. Processes that occur in industrial automated systems and that are reflected in dataflows being observed constitute complex multi-component processes, thus making it more complicated to predict such processes. In this case, complex multi-component time series data should be predicted at different (short-term, medium-term and long-term) time scales. A neural network architecture matching the structure of the multi-component time series being predicted should be built for generating a multi-component prediction. It is proposed to decompose the complex multi-component time series into several basic components using the digital signal processing technology, i. e. to perform a preliminary structural analysis of multi-component time series within the observed range of all time series that reflect operation of the industrial control system. Separate predictions with different time horizon are formed for each basic component of the multi-component time series using the available neural network architecture and machine learning taking into account dynamic characteristics of the above components. Anomalies in the observed range of multiple time series that reflect operation of the industrial control system are detected (identified) through component-wise comparison of each component (resulted from the above preliminary digital processing) of any time series within the observed range of all time series, with each prediction of the relevant component of the above time series within the observed range of all time series. This approach that implies component-wise comparison will allow to detect anomalies within the range of observed time series of the industrial control system separately by their different dynamic characteristics, and thus will improve the efficiency of management of information security within information and industrial systems.

The prevalence of streaming data across various sectors poses significant challenges for real-time anomaly detection due to its volume, velocity, and variability. Traditional data processing methods often need to be improved for such dynamic environments, necessitating robust, scalable, and efficient real-time analysis systems. This study compares two advanced machine learning approaches—LSTM autoencoders and Matrix Profile algorithms—to identify the most effective method for anomaly detection in streaming environments using the NYC taxi dataset. Existing literature on anomaly detection in streaming data highlights various methodologies, including statistical tests, window-based techniques, and machine learning models. Traditional methods like the Generalized ESD test have been adapted for streaming data but often require a full historical dataset to function effectively. In contrast, machine learning approaches, particularly those using LSTM networks, are noted for their ability to learn complex patterns and dependencies, offering promising results in real-time applications. In a comparative analysis, LSTM autoencoders significantly outperformed other methods, achieving an F1-score of 0.22 for anomaly detection, notably higher than other techniques. This model demonstrated superior capability in capturing temporal dependencies and complex data patterns, making it highly effective for the dynamic and varied data in the NYC taxi dataset. The LSTM autoencoder's advanced pattern recognition and anomaly detection capabilities confirm its suitability for complex, high-velocity streaming data environments. Future research should explore the integration of LSTM autoencoders with other machine-learning techniques to enhance further the accuracy, scalability, and efficiency of anomaly detection systems. This study advances our understanding of scalable machine-learning approaches and underscores the critical importance of selecting appropriate models based on the specific characteristics and challenges of the data involved.

Anomaly detection in time series data is a critical task with numerous applications across various domains.This study presents a comprehensive empirical review that explores the methodologies, evaluation metrics, benchmark datasets, novel techniques, and real-world case studies in anomaly detection.The study begins by examining the different methodologies employed in anomaly detection, including statistical-based methods, machine learning-based approaches, and deep learning-based techniques.Evaluation metrics such as precision, Recall, F1-score, ROC curve, and AUC are discussed, along with commonly used benchmark datasets that serve as standards for evaluating anomaly detection algorithms.Novel techniques and algorithms for anomaly detection in time series data are critically analyzed, including time series decomposition and reconstruction methods, transfer learning and domain adaptation techniques, online and streaming anomaly detection approaches, and ensemble methods.These techniques' strengths, limitations, and potential applications are discussed in detail.Real-world case studies showcase the practical applications of anomaly detection in different domains.These case studies include anomaly detection in network traffic data, energy consumption patterns, and medical sensor data.The mathematical approaches and algorithms employed in these case studies are examined to provide insights into the specific methodologies used for anomaly detection.Future research directions and challenges in anomaly detection are discussed, highlighting the importance of explainable AI and interpretable models, incorporating domain knowledge and context awareness, privacy-preserving techniques, and integrating anomaly detection with other data analysis techniques.This empirical review contributes to the field by comprehensively analyzing anomaly detection techniques in time series data.The study's findings offer valuable insights into the strengths and limitations of different methodologies and highlight emerging trends in the field.Furthermore, future research directions and challenges provide a roadmap for advancing anomaly detection in time series data analysis.

Anomaly detection in videos is challenging due to the scarcity and variance in positive samples. Current anomaly detection methods can be categorized into reconstruction models and future frame prediction-based models. However, reconstruction models might be exceptionally adapted to abnormal events due to the learning capacity and generalization ability of deep neural networks, whereas prediction-based methods can be sensitive to noise. In this study, we propose an anomaly detection model based on the latent feature space, which combines advantages from both sides. We argue that the constraints in the latent feature space can promote reconstruction; moreover, the optical flow is also considered to introduce temporal information. We use SPyNet for accurate and efficient optical flow estimation. We extensively validate our method on the UCSD Ped1, UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets. The results demonstrated the feasibility of the proposed method and the benefit of utilizing information in the latent feature space.

Most data collected in real-world scenarios is contaminated, adversely affecting the performance of existing anomaly detection methods that typically rely on clean training data to learn normal patterns. While some approaches have made progress in addressing data contamination, they still face the following challenges: 1) Most methods use pixel-level constraints between the contaminated and reconstructed data as the training loss, and the contaminated data is preferentially learned due to the larger error; 2) Current anomaly detection methods based on frequency-domain do not fully utilize the complex spectral characteristics nor the cross-domain consistency of time series data, which limits the further improvement of the anomaly detection performance. This paper proposes a filter-enhanced reconstruction network with domain-adversarial training for multivariate time series anomaly detection (FERDA), which enables the model to extract normal patterns and high-level semantic information more accurately by training with the error of filtered and reconstructed data. FERDA designs a dual-domain contamination filtering module consisting of the multi-head memory module and the complex linear network. The module is trained to learn normal patterns and cross-domain information through adversarial learning of intra-domain semantic consistency and inter-domain distinguishability. Moreover, a two-stage contamination suppression strategy is proposed. The contamination degree of data is quantified by the difference between pre- and post-filtering, and the impact of contamination is further mitigated by two-stage training with different losses. The proposed method outperforms 23 state-of-the-art baselines in experiments on five benchmark datasets across different domains.

In the actual maintenance of manufacturing enterprises, abnormal changes in after-sale parts demand data often make the inventory strategies unreasonable. Due to the intermittent and small-scale characteristics of demand sequences, it is difficult to accurately identify the anomalies in such sequences using current anomaly detection algorithms. To solve this problem, this paper proposes an unsupervised anomaly detection method for intermittent time series. First, a new abnormal fluctuation similarity matrix is built by calculating the squared coefficient of variation and the maximum information coefficient from the macroscopic granularity. The abnormal fluctuation sequence can then be adaptively screened by using agglomerative hierarchical clustering. Second, the demand change feature and interval feature of the abnormal sequence are constructed and fed into the support vector data description model to perform hypersphere training. Then, the unsupervised abnormal point location detection is realized at the micro-granularity level from the abnormal sequence. Comparative experiments are carried out on the actual demand data of after-sale parts of two large manufacturing enterprises. The results show that, compared with the current representative anomaly detection methods, the proposed approach can effectively identify the abnormal fluctuation position in the intermittent sequence of small samples, and also obtain better detection results.

An extension of Ferguson's (Fourth Berkeley Symposium on Probability and Mathematical Statistics, 1961, Volume 1) univariate normal results and Schwager and Margolin's (1982) multivariate normal results for detection of outliers is made to the multivariate elliptically symmetric case with mean slippage. The main result can be viewed as a robustness property of the use of Mardia's multivariate kurtosis statistic as a locally optimum test statistic to detect outliers against nonnormal multivariate distributions.

The increasing prevalence of vision-threatening retinal diseases poses a growing challenge on healthcare systems worldwide, underscoring the need for effective strategies in disease diagnosis and management. Unsupervised artificial intelligence (AI) approaches, such as anomaly detection, offer the potential to identify established and novel retinal imaging biomarkers. They reduce reliance on expert-driven annotations, mitigate bias, and overcome the limitations of predefined disease categories. Moreover, when integrated into clinical workflows, they have the potential to enhance practical applicability by supporting efficient screening, longitudinal disease monitoring, and objective assessment of treatment response. This review examines the role of anomaly detection for discovering clinically relevant retinal imaging biomarkers, by categorizing biomarkers according to their underlying pathophysiology, summarizing the spectrum of anomaly detection methods applied in the field, and highlighting biomarkers that have been identified through these approaches. Current anomaly detection methods favor the identification of established biomarkers characterized by strong intensity contrasts and well-defined structural boundaries, while more subtle abnormalities remain difficult to capture. Future research should prioritize integrating large language models (LLMs), foundation methods, multimodality, and few-shot anomaly detection, and increase explainability to advance the clinical application of anomaly detection for retinal imaging biomarker discovery in ophthalmology.

Anomaly in geoacoustic emission is an important earthquake precursor. Current geoacoustic anomaly detection methods are limited by their low signal-to-noise ratio, low intensity, sample imbalance, and low accuracy. Therefore, this paper proposes a clone selection algorithm optimized one-class support vector machine method (CSA-OCSVM) for geoacoustic anomaly detection. First, the interquartile range (IQR), cubic spline interpolation, and time window are designed to amplify the geoacoustic signal intensity and energy change rules to reduce the interference of geoacoustic signal noise and intensity. Secondly, to address the imbalance of positive and negative samples in geoacoustic anomaly detection, a one-class support vector machine is introduced for anomaly detection. Meanwhile, in view of the optimization capabilities of the clone selection algorithm, it is adopted to optimize the hyperparameters of OCSVM to improve its detection accuracy. Finally, the proposed model is applied to geoacoustic data anomaly detection in nine different datasets, which are derived from our self-developed acoustic electromagnetic to AI (AETA) system, to verify its effectiveness. By designing comparative experiments with IQR, genetic algorithm OCSVM (GA-OCSVM), particle swarm optimization OCSVM (PSO-OCSVM), and evaluating the performance of the true positive rate (TPR) and false positive rate (FPR), the experimental results depict that the proposed model is superior to the existing state-of-the-art geoacoustic anomaly detection approaches.

Multivariate outlier detection and remediation in geochemical databases

Nowadays, more and more unmanned systems are adopting time series anomaly detection algorithms as the primary tool for KPI monitoring. However, many anomalous data in real environments often have complex features such as large data fluctuations, irregular data fluctuations and large data drift. Traditional univariate or multivariate anomaly detection algorithms can hardly meet the demand for anomaly detection of time series with many features and large fluctuations. Therefore, we propose a multivariate anomaly detection framework adapted to complex environments, which can effectively reduce the complexity of time series data and improve the overall performance of anomaly detection algorithms in complex real systems. First, data preprocessing is performed on the time series before anomaly detection to reduce the impact of temporal bias of different data items in multivariate time series data. Second, to intuitively analyze the anomalous fluctuation characteristics of the time series, we use Spectral Residual to process the significance of the time series. Then, we combine the Spectral Residual processed data and Logistic Regression algorithm to construct anomaly classifiers for anomaly detection of real-time data in complex real systems. Finally, by experimenting with a prototype system on a private cloud platform, our approach obtains superior results under multiple anomaly patterns compared with traditional univariate and multivariate anomaly detection algorithms.KeywordsUnmanned systemsMultivariate anomaly detectionSpectral residualConcept drift

Abstract. We consider an autoregressive moving‐average (ARMA) time series where the observations are perturbed by two kinds of outliers: an additive outlier (AO) or an innovation outlier (IO). Abraham and Yatawara [Journal of Time Series Analysis(1988) Vol. 9, pp. 109–19] investigate a sequential test which successively detects and identifies the outlier type. In this article, we propose an extension of this test, called ‘modified sequential test’, which performs the two procedures simultaneously and coherently. The asymptotic distribution of the test statistic is calculated under the null hypothesis that no outlier is present. Comparison of the two test procedures using simulation experiments shows that the proposed test gives a better power especially in the case of an IO.