Unsupervised Anomaly Detection for Intermittent Sequences Based on Multi-Granularity Abnormal Pattern Mining

Safety enhancement is a major goal of the aviation industry owing to the predicted increase in air travel. There is also the need to prevent fatalities, increase reliability and reduce monetary costs suffered as a result of delays and accidents that still occur. Accidents today are complex as a result of many causal factors acting alone but more often as a combination with other contributing factors. In tackling this trend, proactive measures have been put in place to find hazardous combinations that occur during flights in order to mitigate them before accidents occur. Flight Anomaly Detection (AD) methods are aimed at highlighting abnormal occurrences of a flight, that are different from the norm. As an improvement on the current state-of-the-art method, previous works have proposed different AD techniques for detection of previously unknown flight risks such as component faults, aircraft operational inefficiencies and some abnormal crew behaviour. However, current AD methods individually have limitations that prevent them from detecting certain significant anomalies in flight data. This paper surveys current flight AD approaches, their strengths and limitations as well as brings to light the benefits of a hybrid AD method to extend previous work and find safety-critical events, particularly those related to abnormal crew activity: a class of events known to amount for a substantial number of accidents/incidents today. It also highlights another emerging AD application opportunity, its challenges and how AD is beneficial in addressing them.

The increasing prevalence of vision-threatening retinal diseases poses a growing challenge on healthcare systems worldwide, underscoring the need for effective strategies in disease diagnosis and management. Unsupervised artificial intelligence (AI) approaches, such as anomaly detection, offer the potential to identify established and novel retinal imaging biomarkers. They reduce reliance on expert-driven annotations, mitigate bias, and overcome the limitations of predefined disease categories. Moreover, when integrated into clinical workflows, they have the potential to enhance practical applicability by supporting efficient screening, longitudinal disease monitoring, and objective assessment of treatment response. This review examines the role of anomaly detection for discovering clinically relevant retinal imaging biomarkers, by categorizing biomarkers according to their underlying pathophysiology, summarizing the spectrum of anomaly detection methods applied in the field, and highlighting biomarkers that have been identified through these approaches. Current anomaly detection methods favor the identification of established biomarkers characterized by strong intensity contrasts and well-defined structural boundaries, while more subtle abnormalities remain difficult to capture. Future research should prioritize integrating large language models (LLMs), foundation methods, multimodality, and few-shot anomaly detection, and increase explainability to advance the clinical application of anomaly detection for retinal imaging biomarker discovery in ophthalmology.

Anomaly detection using improved deep SVDD model with data structure preservation

This study focuses on unsupervised anomaly detection for structural systems using autoencoders. Since collecting abnormal data is difficult while normal data from actual bridges can be more easily obtained, we propose an unsupervised learning-based anomaly detection method that utilizes the differences between normal and abnormal structural states through autoencoders. The proposed approach employs two models: AE1, which is trained on known normal data and evaluates the reconstruction error for the data to be diagnosed, and AE2, which is trained on the data to be diagnosed and calculates the reconstruction error for known normal data. Anomalies are detected based on the reconstruction errors from each model. Additionally, a combined bridge anomaly detection method called BL-AE, which integrates AE1 and AE2, is proposed. Additionally, the improvement of AE1’s detection performance for actual bridges whose normal state varies due to seasonal changes is investigated. The results showed that learning with overlapping (duplicated) data improves anomaly detection accuracy. Furthermore, we evaluated the anomaly detection performance of BL-AE using artificially generated amplitude anomalies. As a result, Both the AUC and the actual positive rate at 10% false positive rate were improved.

Identifying unusual or anomalous patterns in an underlying dataset is an important but challenging task in many applications. The focus of the unsupervised anomaly detection literature has mostly been on vectorised data. However, many applications are more naturally described using higher-order tensor representations. Approaches that vectorise tensorial data can destroy the structural information encoded in the high-dimensional space, and lead to the problem of the curse of dimensionality. In this paper we present the first unsupervised tensorial anomaly detection method, along with a randomised version of our method. Our anomaly detection method, the One-class Support Tensor Machine (1STM), is a generalisation of conventional one-class Support Vector Machines to higher-order spaces. 1STM preserves the multiway structure of tensor data, while achieving significant improvement in accuracy and efficiency over conventional vectorised methods. We then leverage the theory of nonlinear random projections to propose the Randomised 1STM (R1STM). Our empirical analysis on several real and synthetic datasets shows that our R1STM algorithm delivers comparable or better accuracy to a state-of-the-art deep learning method and traditional kernelised approaches for anomaly detection, while being approximately 100 times faster in training and testing.

Volunteered Geographic Information (VGI), defined as geographic information contributed voluntarily by individuals, has grown exponentially with the aid of ubiquitous GPS-enabled technologies. VGI projects have generated a large amount of geographic data, providing a new data source for scientific research. However, many scientists are concerned about the quality of VGI data for research, given the lack of rigorous and systematic quality control procedures. This study contributes to the improvement of quality control procedures by proposing a Cross-Volunteer Referencing Anomaly Detection (CVRAD) method to filter anomalous data, using the crowdsourced Safecast radiation data as a case study. The anomaly detection method is validated using two data sets: (1) an official radiation survey data set collected by the KURAMA car-borne system, (2) a set of anomalous Safecast measurements filtered by Safecast moderators. The validation results show that the proposed CVRAD method outperformed the 1.5 IQR benchmark method in minimizing the overall measurement error and detecting abnormal imports of Safecast measurements, thus demonstrating the effectiveness of the proposed method in improving the overall accuracy of crowdsourced radiation measurements.

Anomaly in geoacoustic emission is an important earthquake precursor. Current geoacoustic anomaly detection methods are limited by their low signal-to-noise ratio, low intensity, sample imbalance, and low accuracy. Therefore, this paper proposes a clone selection algorithm optimized one-class support vector machine method (CSA-OCSVM) for geoacoustic anomaly detection. First, the interquartile range (IQR), cubic spline interpolation, and time window are designed to amplify the geoacoustic signal intensity and energy change rules to reduce the interference of geoacoustic signal noise and intensity. Secondly, to address the imbalance of positive and negative samples in geoacoustic anomaly detection, a one-class support vector machine is introduced for anomaly detection. Meanwhile, in view of the optimization capabilities of the clone selection algorithm, it is adopted to optimize the hyperparameters of OCSVM to improve its detection accuracy. Finally, the proposed model is applied to geoacoustic data anomaly detection in nine different datasets, which are derived from our self-developed acoustic electromagnetic to AI (AETA) system, to verify its effectiveness. By designing comparative experiments with IQR, genetic algorithm OCSVM (GA-OCSVM), particle swarm optimization OCSVM (PSO-OCSVM), and evaluating the performance of the true positive rate (TPR) and false positive rate (FPR), the experimental results depict that the proposed model is superior to the existing state-of-the-art geoacoustic anomaly detection approaches.

Fiber-optic communication systems serve as the backbone of modern data communication networks, with increasing demands on their reliability and robustness in various emerging applications. A key challenge in ensuring reliable fiber-optic transmission lies in addressing fiber anomalies, which can cause signal degradation, service disruptions, and even system failures. However, the current anomaly detection method is too complex to be implemented in deployed networks or consumes too much time during detection. This paper proposes a simple and effective fiber anomaly detection method for C+L-band fiber-optic communication systems, leveraging the spectral tilt induced by the stimulated Raman scattering (SRS) effect. The method reconstructs the spectral tilt along an anomalous fiber link by analyzing the input and output power profile, easily obtainable from optical channel monitors (OCMs), enabling anomaly localization and loss quantification through forward and backward spectral tilt comparison. The performance and robustness of the proposed method are studied and discussed in actual scenarios such as erbium-doped fiber amplifier gain fluctuations, OCM measurement errors, parameter inaccuracies, and modeling inaccuracies. In addition to being able to detect a single anomaly, the process of using the proposed method to achieve multiple anomaly detection links is also discussed according to the chronological order of anomalies. Experimental validation on a multi-span C+L-band transmission system achieves a maximum localization error of 2.06 km and a maximum loss estimation error of 0.14 dB within 2 s. This work underscores the potential of exploiting inherent physical phenomena like SRS for fast anomaly detection in deployed optical networks.

Anomaly detection in the fan system of a thermal power plant monitored by continuous and two-valued variables

The prevalence of networked sensors and actuators in many real-world systems such as smart buildings, factories, power plants, and data centers generate substantial amounts of multivariate time series data for these systems. The rich sensor data can be continuously monitored for intrusion events through anomaly detection. However, conventional threshold-based anomaly detection methods are inadequate due to the dynamic complexities of these systems, while supervised machine learning methods are unable to exploit the large amounts of data due to the lack of labeled data. On the other hand, current unsupervised machine learning approaches have not fully exploited the spatial-temporal correlation and other dependencies amongst the multiple variables (sensors/actuators) in the system for detecting anomalies. In this work, we propose an unsupervised multivariate anomaly detection method based on Generative Adversarial Networks (GANs). Instead of treating each data stream independently, our proposed MAD-GAN framework considers the entire variable set concurrently to capture the latent interactions amongst the variables. We also fully exploit both the generator and discriminator produced by the GAN, using a novel anomaly score called DR-score to detect anomalies by discrimination and reconstruction. We have tested our proposed MAD-GAN using two recent datasets collected from real-world CPS: the Secure Water Treatment (SWaT) and the Water Distribution (WADI) datasets. Our experimental results showed that the proposed MAD-GAN is effective in reporting anomalies caused by various cyber-intrusions compared in these complex real-world systems.

Intrusion detection is crucial in computer network security issues; therefore, this work is aimed at maximizing network security protection and its improvement by proposing various preventive techniques. Outlier detection and semisupervised clustering algorithms based on shared nearest neighbors are proposed in this work to address intrusion detection by converting it into a problem of mining outliers using the network behavior dataset. The algorithm uses shared nearest neighbors as similarity, judges whether it is an outlier according to the number of nearest neighbors of a data point, and performs semisupervised clustering on the dataset where outliers are deleted. In the process of semisupervised clustering, vast prior knowledge is added, and the dataset is clustered according to the principle of graph segmentation. The novelty of the proposed algorithm lies in outlier detection while effectively avoiding the dependence on parameters, thus eliminating the influence of outliers on clustering. This article uses real datasets: lypmphography and glass for simulation purposes. The simulation results show that the algorithm proposed in this paper can effectively detect outliers and has a good clustering effect. Furthermore, the experimentation reveals that the outlier detection‐based SCA‐SNN algorithm has the best practical effect on the dataset without outliers, clearly validating the clustering performance of the outlier detection‐based SCA‐SNN algorithm. Furthermore, compared to the other state‐of‐the‐art anomaly detection method, it was revealed that the anomaly detection technology based on outlier mining does not require a training process. Thus, they overcome the current anomaly detection problems caused due to incomplete normal patterns in training samples.

Most data collected in real-world scenarios is contaminated, adversely affecting the performance of existing anomaly detection methods that typically rely on clean training data to learn normal patterns. While some approaches have made progress in addressing data contamination, they still face the following challenges: 1) Most methods use pixel-level constraints between the contaminated and reconstructed data as the training loss, and the contaminated data is preferentially learned due to the larger error; 2) Current anomaly detection methods based on frequency-domain do not fully utilize the complex spectral characteristics nor the cross-domain consistency of time series data, which limits the further improvement of the anomaly detection performance. This paper proposes a filter-enhanced reconstruction network with domain-adversarial training for multivariate time series anomaly detection (FERDA), which enables the model to extract normal patterns and high-level semantic information more accurately by training with the error of filtered and reconstructed data. FERDA designs a dual-domain contamination filtering module consisting of the multi-head memory module and the complex linear network. The module is trained to learn normal patterns and cross-domain information through adversarial learning of intra-domain semantic consistency and inter-domain distinguishability. Moreover, a two-stage contamination suppression strategy is proposed. The contamination degree of data is quantified by the difference between pre- and post-filtering, and the impact of contamination is further mitigated by two-stage training with different losses. The proposed method outperforms 23 state-of-the-art baselines in experiments on five benchmark datasets across different domains.

Evolving anomaly detection for network streaming data

To ensure the performance of large-scale datacenters, operators need to monitor up to tens of millions of various-type KPIs, e.g., CPU utilization, memory utilization. For each KPI, it is crucial but challenging to detect outliers that deviate from its historical patterns or the patterns of other KPIs in the same period. In this work, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">OutSpot</i> , an unsupervised outlier detection framework that integrates hierarchical agglomerative clustering (HAC) with conditional variational autoencoder (CVAE), which significantly improves computational efficiency and comprehensively learns the above two patterns. Additionally, two simple yet effective techniques, soft threshold and median filter, are applied to precisely determine outlier KPIs. Using two real-world datasets collected from the datacenters owned by a top-tier global short video service provider and a top-tier domestic operator,respectively. It demonstrates that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">OutSpot</i> achieves the best F1 score of 0.95 and 0.91, AUC of 0.99 and 0.99 on the two datasets, significantly outperforming seven baseline outlier detection methods.

Keep DRÆMing: Discriminative 3D anomaly detection through anomaly simulation