Evolving anomaly detection for network streaming data

The data collected by sensors is streaming data in the Internet of Things(IoT). Although existing deep learningbased anomaly detection methods generally perform well on static data, they struggle to respond timely to streaming data after distribution changes. However, streaming data suffers from conceptual drift due to the highly dynamic nature of IoT. In network security, concept drift-oriented anomaly detection is a crucial task, because it can adjust the model to adapt to the latest data, and detect attacks in time. Existing streaming anomaly detection methods are confronted with some challenges, including of the latency of model updates, the uneven importance of new data, and the self-poisoning due to model self-updates. To tackle the above challenges, we propose a knowledge distillation based adaptive anomaly detection model towards concept-drift, ADTCD. ADTCD transfers the knowledge of the teacher model to the student model and only updates the student model to reduce the delay. We construct an algorithm of dynamically adjusting model parameters, which dynamically adjusts model weights through local inference on new samples, in order to improving the models responsiveness to new distribution data, meanwhile solving the problem of uneven importance of new data. In additions, we adopt a one-class support vector-based outlier removal method to tackle the self-poisoning problem. In comprehensive experiments on seven high-dimensional datasets, ADTCD achieves an AUC improvement of 12.46% compared to the state-of-the-art streaming anomaly detection methods. Our future direction will focus on exploring the concept drift problem using methods beyond autoencoders.

In recent years, anomaly detection techniques in time-series data have been widely used in manufacturing, cybersecurity, and other fields. Meanwhile, various anomaly detection models based on generative adversarial networks (GAN) are gradually used in time-series anomaly detection tasks. However, there are problems of unstable generator training, missed detection of anomalous data, and inconsistency between the discriminator's discriminant and the anomaly detection target in GAN networks. Aiming at the above problems, the paper proposes a DUAL-ADGAN (Dual Anomaly Detection Generative Adversarial Networks) model for the detection of anomalous data in time series. First, the Wasserstein distance satisfying the Lipschitz constraint is used as the loss function of the data reconstruction module, which improves the stability of the traditional GAN network training. Second, by adding a data prediction module to the DUAL-ADGAN model, the distinction between abnormal and normal samples is increased, and the rate of missing abnormal data in the model is reduced. Third, by introducing the Fence-GAN loss function, the discriminator is aligned with the anomaly detection target, which effectively reduces the anomaly data false detection rate of the DUAL-ADGAN model. Finally, anomaly scores derived from the DUAL-ADGAN model are compared with dynamic thresholds to detect anomalies. The experimental results show that the average F1 of the DUAL-ADGAN model is 0.881, which is better than the other nine baseline models. The conclusions demonstrate that the DUAL-ADGAN model proposed in the paper is more stable in training while effectively solving the problems of anomaly miss detection and discriminator inconsistency with the anomaly detection target in the anomaly detection task.

Integration of Information and Communication Technology (ICT) in modern smart grids (SGs) offers many advantages including the use of renewables and an effective way to protect, control and monitor the energy transmission and distribution. To reach an optimal operation of future energy systems, availability, integrity and confidentiality of data should be guaranteed. Research on the cyber-physical security of electrical substations based on IEC 61850 is still at an early stage. In the present work, we first model the network traffic data in electrical substations, then, we present a statistical Anomaly Detection (AD) method to detect Denial of Service (DoS) attacks against the Generic Object Oriented Substation Event (GOOSE) network communication. According to interpretations on the self-similarity and the Long-Range Dependency (LRD) of the data, an Auto-Regressive Fractionally Integrated Moving Average (ARFIMA) model was shown to describe well the GOOSE communication in the substation process network. Based on this ARFIMA-model and in view of cyber-physical security, an effective model-based AD method is developed and analyzed. Two variants of the statistical AD considering statistical hypothesis testing based on the Generalized Likelihood Ratio Test (GLRT) and the cumulative sum (CUSUM) are presented to detect flooding attacks that might affect the availability of the data. Our work presents a novel AD method, with two different variants, tailored to the specific features of the GOOSE traffic in IEC 61850 substations. The statistical AD is capable of detecting anomalies at unknown change times under the realistic assumption of unknown model parameters. The performance of both variants of the AD method is validated and assessed using data collected from a simulation case study. We perform several Monte-Carlo simulations under different noise variances. The detection delay is provided for each detector and it represents the number of discrete time samples after which an anomaly is detected. In fact, our statistical AD method with both variants (CUSUM and GLRT) has around half the false positive rate and a smaller detection delay when compared with two of the closest works found in the literature. Our AD approach based on the GLRT detector has the smallest false positive rate among all considered approaches. Whereas, our AD approach based on the CUSUM test has the lowest false negative rate thus the best detection rate. Depending on the requirements as well as the costs of false alarms or missed anomalies, both variants of our statistical detection method can be used and are further analyzed using composite detection metrics.

An integrated energy system (IES) is vulnerable to network attacks due to the coupling features of multi-energy systems, as well as the deep integration between a physical system and an information system. The anomaly detection of the time-series data in an IES is a key problem to defend against network attacks and ensure the cyber-physical security of IES. Aiming at false data injection attacks (FDIAs) on IES, this paper proposes an anomaly detection method for time-series data in a cyber-physical integrated energy system based on time-frequency feature prediction. The time-frequency features of the time-series data are extracted based on three time-frequency transform methods (DWT, EMD, and EWT). Then the extracted time-frequency features are input to the autoencoder (AE) to capture the hidden features and nonlinear structure of the original data in the frequency domain. The time-domain data within the detected time period are predicted by applying regression prediction on the top-layer features of AE. Considering the uncertainty of regression prediction, kernel density estimation (KDE) is used to estimate the probability density function of prediction error and the interval of the predicted data is estimated accordingly. The estimated lower boundary value of the predicted data is selected as the attack judgment threshold for anomaly detection. The results of the case study verify the advantages of the proposed method in reducing the false positive rate and improving the anomaly detection accuracy.

Volunteered Geographic Information (VGI), defined as geographic information contributed voluntarily by individuals, has grown exponentially with the aid of ubiquitous GPS-enabled technologies. VGI projects have generated a large amount of geographic data, providing a new data source for scientific research. However, many scientists are concerned about the quality of VGI data for research, given the lack of rigorous and systematic quality control procedures. This study contributes to the improvement of quality control procedures by proposing a Cross-Volunteer Referencing Anomaly Detection (CVRAD) method to filter anomalous data, using the crowdsourced Safecast radiation data as a case study. The anomaly detection method is validated using two data sets: (1) an official radiation survey data set collected by the KURAMA car-borne system, (2) a set of anomalous Safecast measurements filtered by Safecast moderators. The validation results show that the proposed CVRAD method outperformed the 1.5 IQR benchmark method in minimizing the overall measurement error and detecting abnormal imports of Safecast measurements, thus demonstrating the effectiveness of the proposed method in improving the overall accuracy of crowdsourced radiation measurements.

This study focuses on unsupervised anomaly detection for structural systems using autoencoders. Since collecting abnormal data is difficult while normal data from actual bridges can be more easily obtained, we propose an unsupervised learning-based anomaly detection method that utilizes the differences between normal and abnormal structural states through autoencoders. The proposed approach employs two models: AE1, which is trained on known normal data and evaluates the reconstruction error for the data to be diagnosed, and AE2, which is trained on the data to be diagnosed and calculates the reconstruction error for known normal data. Anomalies are detected based on the reconstruction errors from each model. Additionally, a combined bridge anomaly detection method called BL-AE, which integrates AE1 and AE2, is proposed. Additionally, the improvement of AE1’s detection performance for actual bridges whose normal state varies due to seasonal changes is investigated. The results showed that learning with overlapping (duplicated) data improves anomaly detection accuracy. Furthermore, we evaluated the anomaly detection performance of BL-AE using artificially generated amplitude anomalies. As a result, Both the AUC and the actual positive rate at 10% false positive rate were improved.

The increasing prevalence of vision-threatening retinal diseases poses a growing challenge on healthcare systems worldwide, underscoring the need for effective strategies in disease diagnosis and management. Unsupervised artificial intelligence (AI) approaches, such as anomaly detection, offer the potential to identify established and novel retinal imaging biomarkers. They reduce reliance on expert-driven annotations, mitigate bias, and overcome the limitations of predefined disease categories. Moreover, when integrated into clinical workflows, they have the potential to enhance practical applicability by supporting efficient screening, longitudinal disease monitoring, and objective assessment of treatment response. This review examines the role of anomaly detection for discovering clinically relevant retinal imaging biomarkers, by categorizing biomarkers according to their underlying pathophysiology, summarizing the spectrum of anomaly detection methods applied in the field, and highlighting biomarkers that have been identified through these approaches. Current anomaly detection methods favor the identification of established biomarkers characterized by strong intensity contrasts and well-defined structural boundaries, while more subtle abnormalities remain difficult to capture. Future research should prioritize integrating large language models (LLMs), foundation methods, multimodality, and few-shot anomaly detection, and increase explainability to advance the clinical application of anomaly detection for retinal imaging biomarker discovery in ophthalmology.

The eternal enigma in prostatic biopsy access route.

formula omitted]S: Adaptive anomaly detection on sporadic data streams

Intrusion detection is crucial in computer network security issues; therefore, this work is aimed at maximizing network security protection and its improvement by proposing various preventive techniques. Outlier detection and semisupervised clustering algorithms based on shared nearest neighbors are proposed in this work to address intrusion detection by converting it into a problem of mining outliers using the network behavior dataset. The algorithm uses shared nearest neighbors as similarity, judges whether it is an outlier according to the number of nearest neighbors of a data point, and performs semisupervised clustering on the dataset where outliers are deleted. In the process of semisupervised clustering, vast prior knowledge is added, and the dataset is clustered according to the principle of graph segmentation. The novelty of the proposed algorithm lies in outlier detection while effectively avoiding the dependence on parameters, thus eliminating the influence of outliers on clustering. This article uses real datasets: lypmphography and glass for simulation purposes. The simulation results show that the algorithm proposed in this paper can effectively detect outliers and has a good clustering effect. Furthermore, the experimentation reveals that the outlier detection‐based SCA‐SNN algorithm has the best practical effect on the dataset without outliers, clearly validating the clustering performance of the outlier detection‐based SCA‐SNN algorithm. Furthermore, compared to the other state‐of‐the‐art anomaly detection method, it was revealed that the anomaly detection technology based on outlier mining does not require a training process. Thus, they overcome the current anomaly detection problems caused due to incomplete normal patterns in training samples.

Anomaly detection in streaming data is an important task for many real-world applications, such as network security, fraud detection, and system monitoring. However, streaming data often exhibit concept drift, which means that the data distribution changes over time. This poses a significant challenge for many anomaly detection algorithms, as they need to adapt to the evolving data to maintain high detection accuracy. Existing streaming anomaly detection algorithms lack a unified evaluation framework that validly assesses their performance and robustness under different types of concept drifts and anomalies. In this paper, we conduct a systematic technical review of the state-of-the-art methods for anomaly detection in streaming data. We propose a new data generator, called SCAR (Streaming data generator with Customizable Anomalies and concept dRifts), that can synthesize streaming data based on synthetic and real-world datasets from different domains. Furthermore, we adapt four static anomaly detection models to the streaming setting using a generic reconstruction strategy as baselines, and then compare them systematically with 9 existing streaming anomaly detection algorithms on 76 synthesized datasets that have various types of anomalies and concept drifts. The challenges and future research directions for anomaly detection in streaming data are also presented.

Anomaly detection is a classic and crucial problem in the field of artificial intelligence, which aims to find instances that deviate so much from the main distribution of data or differ from known instances. This paper explores how to combine advanced deep learning techniques with traditional probabilistic statistical methods for anomaly detection. We propose a very effective and concise semi-supervised anomaly detection method named “ORGAN-KDE” based on the orthogonal generative adversarial network (O-GAN) and kernel density estimation. In the training phase, we use the encoder of O-GAN to learn the latent representation of normal data, namely the code of normal data, and then use the kernel density estimation to solve the probability density function of code. The code of normal sample obtained through the trained encoder can get a larger probability value when passing through the trained kernel density estimator, while the code of anomalous sample can get a smaller probability value, so as to achieve the purpose of anomaly detection. Compared with other anomaly detection methods based on GAN, our method has a very simple network structure, and experiments have proved that it performs well on both structured datasets and image datasets.

BackgroundChemical exchange saturation transfer (CEST) magnetic resonance imaging (MRI) demonstrates significant potential for early disease detection, yet encounters challenges in lesion analysis due to interference from concomitant effects during imaging. Although machine learning-based unsupervised anomaly detection (UAD) methods offer feasible solutions for identifying subtle lesions in contaminated images, current UAD approaches primarily dependent on convolutional neural network (CNN) or Transformer architectures to process image-level data from conventional imaging modalities, exhibit limitations when applied to CEST data characterized by spectral information. This study aimed to propose a novel UAD framework tailored for CEST spectral data to overcome these limitations, enabling precise pixel-wise anomaly detection for early-stage lesions.MethodsThe proposed framework named Mamba-AE employs a multi-layers encoder-decoder architecture with stacked Mamba blocks as the core component. Each Mamba block integrates three key modules: selective state-space models (SSMs) to dynamically adjust parameters based on input sequences, enabling adaptive long-range dependency modeling of CEST spectral data; Gated Multi-Layer Perceptron with Sigmoid Linear Unit (SiLU) activation and residual connections to enhance non-linear feature learning and stability; Multi-Scale Feature Alignment that aligns hierarchical encoder-decoder features via cosine similarity, preserving physiological semantics across scales. The framework adopts a dual-domain reconstruction strategy: data-space reconstruction minimizes pixel-wise spectral errors through Huber loss, while feature-space reconstruction enforces consistency between paired encoder-decoder layer features via cosine similarity loss. Anomaly scores are generated by combining normalized residuals from data-space errors and discrepancies in feature-space alignment, enabling precise pixel-level lesion localization in early-stage CEST data. For quantitative evaluation of lesion detection performance, the area under the curve (AUC) and Dice similarity coefficient are employed as core metrics. The framework was trained exclusively on CEST spectra from healthy rat brain tissue and validated on an independent dataset from a rat model of transient focal cerebral ischemia induced by middle cerebral artery occlusion (MCAO) with reperfusion.ResultsThe proposed method has been validated on ischemic stroke rat datasets at 2, 6, and 24 hours post-occlusion. Mamba-AE demonstrated superior performance across all stages. At the critical 2-hour time point, it achieved the highest performance among all methods, with an AUC of 95.91% and a Dice score of 88.43% (sensitivity >89%, specificity >92%). The advantage remained substantial at 6 hours (AUC: 92.19%, Dice: 85.23%) and 24 hours (AUC: 94.64%, Dice: 87.79%), consistently outperforming other approaches. Furthermore, anomaly heatmaps revealed precise lesion localization capabilities, with spatial accuracy correlating strongly with histopathological ground truth measurements.ConclusionsMamba-AE provides a computationally efficient and robust solution for early disease detection in CEST MRI. Its integration of spectral feature learning and structural preservation highlights its potential for clinical applications requiring high sensitivity to subtle pathological changes.

Anomaly detection over multi-dimensional data stream has attracted considerable attention recently in various fields, such as network, finance and aerospace. In many cases, anomalies are composed of a sequence of multi-dimensional data, and it's necessary to detect this type of anomalies accurately and efficiently over data stream. Existing online methods of anomaly detection merely focus on the single-dimensional sequence. What's more, current studies about multi-dimensional sequence are mainly concentrated on static database. However, the anomaly detection for multi-dimensional sequence over data stream is much more difficult, due to the complexity of multidimensional sequence processing, the dynamic nature of data stream and the unbalance between normal and abnormal data. Facing these challenges, we propose an anomaly detection method for multi-dimensional sequence over data stream based on cost sensitive support vector machine (C-SVM) called ADMS. First, to improve the accuracy and efficiency, the ADMS transforms multi-dimensional sequences into feature vectors in a lossless way and prunes worthless features of these vectors. And then, the ADMS can detect abnormal sequences over dynamically imbalanced data stream by lively testing these vectors based on C-SVM. Experiments show that the false negative rate (FNR) of the ADMS is lower than 5%, the false positive rate (FPR) is lower than 7%, and the throughput is improved 42% by pruning worthless features. In addition, the AMDS performs well when there are concept drifts over the data stream.

Unsupervised outlier detection using neural network-based mixtures of probabilistic principal component analyzers for building chiller fault diagnosis