AI-Based Anomaly Detection for Cloud Logs and Distributed Traces

Anomaly detection in database transactions is critical for safeguarding sensitive information and ensuring the integrity of operations in industries like finance, healthcare, and e-commerce. Existing techniques, including rule-based, machine learning, and deep learning methods, face challenges such as high false positive rates, poor adaptability to evolving patterns, and limited scalability in imbalanced datasets. This research proposes a novel Reinforcement Learning (RL)-based anomaly detection system to address these limitations. The model employs a dynamic reward mechanism and anomaly scoring system to classify transactions accurately while reducing false positives. It leverages the Kaggle Anomaly Detection in Transactions Dataset and a synthetically generated dataset for training and evaluation. Experimental results show that the RL-based model outperforms traditional methods, achieving a precision of 95.2%, recall of 92.4%, and an AUC-ROC score of 97.2%, significantly higher than Autoencoders, Isolation Forest, and Support Vector Machines. The proposed model’s adaptability and robustness make it a scalable solution for real-time anomaly detection, addressing critical gaps in existing techniques. This study advances database security by offering a highly accurate, adaptive, and efficient system for detecting anomalies in complex transactional environments.

Conventional security methods deployed in power plants have difficulty detecting time delay attacks, since they do not alter network packets. However, these attacks can cause damage and instability in power systems, so detecting them is an urgent anomaly detection problem. Current state-of-the-art anomaly detection methods employ machine learning (ML) or statistical regression models in a supervised fashion, which require large amounts of labeled data for training. This data may be hard to practically obtain, so it is preferable to use unsupervised methods, which do not need labeled data. However, unsupervised anomaly detection solutions suffer from high false positive rates, especially under weak and moderate attacks. To improve on existing unsupervised solutions, we develop and present a dual-stage anomaly detection method using a Recurrent Variational Autoencoder (RVAE) and K-means clustering for detecting time delay attacks in power systems. We focus on samples including weak or moderate attacks which existing solutions cannot accurately detect, but can still harm the system if strategically targeted. We call these cases borderline samples, and use an additional clustering enhancement to more accurately classify them. Evaluation of the proposed approach on our power plant dataset demonstrates that our approach is effective in detecting time delay attacks, with 14.7% higher area under the ROC curve (AUC) than RVAE for borderline samples.

In recent years, anomaly detection techniques in time-series data have been widely used in manufacturing, cybersecurity, and other fields. Meanwhile, various anomaly detection models based on generative adversarial networks (GAN) are gradually used in time-series anomaly detection tasks. However, there are problems of unstable generator training, missed detection of anomalous data, and inconsistency between the discriminator's discriminant and the anomaly detection target in GAN networks. Aiming at the above problems, the paper proposes a DUAL-ADGAN (Dual Anomaly Detection Generative Adversarial Networks) model for the detection of anomalous data in time series. First, the Wasserstein distance satisfying the Lipschitz constraint is used as the loss function of the data reconstruction module, which improves the stability of the traditional GAN network training. Second, by adding a data prediction module to the DUAL-ADGAN model, the distinction between abnormal and normal samples is increased, and the rate of missing abnormal data in the model is reduced. Third, by introducing the Fence-GAN loss function, the discriminator is aligned with the anomaly detection target, which effectively reduces the anomaly data false detection rate of the DUAL-ADGAN model. Finally, anomaly scores derived from the DUAL-ADGAN model are compared with dynamic thresholds to detect anomalies. The experimental results show that the average F1 of the DUAL-ADGAN model is 0.881, which is better than the other nine baseline models. The conclusions demonstrate that the DUAL-ADGAN model proposed in the paper is more stable in training while effectively solving the problems of anomaly miss detection and discriminator inconsistency with the anomaly detection target in the anomaly detection task.

The rapid growth of Internet of Things (IoT) devices is accompanied by increasingly sophisticated security threats, including DDoS attacks, brute-force authentication attempts, and large-scale packet flooding. Traditional statistical methods for anomaly detection exhibit low robustness to noise and fail to account for the dynamic nature of IoT traffic. This results in a higher rate of false positives and reduced accuracy in attack identification. This paper proposes a hybrid approach to IoT traffic anomaly detection consisting of three stages: preliminary filtering of suspicious packets using a modified Z-score adjusted for sample size; adaptive probabilistic attack risk assessment based on a Bayesian classifier with a weighting function that amplifies the impact of significant deviations; final classification using an ensemble of models (Random Forest, SVM, and LSTM), which ensures robustness to noise and enables the identification of nonlinear dependencies in the data. Experimental evaluation on the UNSW-NB15 dataset, which includes both normal traffic and diverse attack scenarios, demonstrated that the proposed method achieved Precision = 89.1 %, Recall = 90.3 %, and F1-score = 89.9 %. The best results were observed in the analysis of message interval anomalies (up to 92 % accuracy), confirming the effectiveness of temporal features. The method outperformed classical algorithms (Rosner Test, Holt- Winters) and achieved comparable accuracy to autoencoder while requiring significantly fewer computational resources. The hybrid architecture enables adaptation to diverse attack types and reduces false alarms through the combination of statistical filtering and ensemble classification. Its noise resilience and low computational complexity make the method suitable for deployment in resource-constrained IoT environments. Future research directions include the integration of federated learning for decentralized anomaly detection and the use of self-adaptive neural architectures for predicting complex attack scenarios.

Intrusion detection system (IDS) has played an important role as a device to defend our networks from cyber attacks. However, since it is unable to detect unknown attacks, i.e., 0-day attacks, the ultimate challenge in intrusion detection field is how we can exactly identify such an attack by an automated manner. Over the past few years, several studies on solving these problems have been made on anomaly detection using unsupervised learning techniques such as clustering, one-class support vector machine (SVM), etc. Although they enable one to construct intrusion detection models at low cost and effort, and have capability to detect unforeseen attacks, they still have mainly two problems in intrusion detection: a low detection rate and a high false positive rate. In this paper, we propose a new anomaly detection method based on clustering and multiple one-class SVM in order to improve the detection rate while maintaining a low false positive rate. We evaluated our method using KDD Cup 1999 data set. Evaluation results show that our approach outperforms the existing algorithms reported in the literature; especially in detection of unknown attacks.

The prevalence of streaming data across various sectors poses significant challenges for real-time anomaly detection due to its volume, velocity, and variability. Traditional data processing methods often need to be improved for such dynamic environments, necessitating robust, scalable, and efficient real-time analysis systems. This study compares two advanced machine learning approaches—LSTM autoencoders and Matrix Profile algorithms—to identify the most effective method for anomaly detection in streaming environments using the NYC taxi dataset. Existing literature on anomaly detection in streaming data highlights various methodologies, including statistical tests, window-based techniques, and machine learning models. Traditional methods like the Generalized ESD test have been adapted for streaming data but often require a full historical dataset to function effectively. In contrast, machine learning approaches, particularly those using LSTM networks, are noted for their ability to learn complex patterns and dependencies, offering promising results in real-time applications. In a comparative analysis, LSTM autoencoders significantly outperformed other methods, achieving an F1-score of 0.22 for anomaly detection, notably higher than other techniques. This model demonstrated superior capability in capturing temporal dependencies and complex data patterns, making it highly effective for the dynamic and varied data in the NYC taxi dataset. The LSTM autoencoder's advanced pattern recognition and anomaly detection capabilities confirm its suitability for complex, high-velocity streaming data environments. Future research should explore the integration of LSTM autoencoders with other machine-learning techniques to enhance further the accuracy, scalability, and efficiency of anomaly detection systems. This study advances our understanding of scalable machine-learning approaches and underscores the critical importance of selecting appropriate models based on the specific characteristics and challenges of the data involved.

Relational Database Management Systems (RDBMSs) are foundational to operations across finance, healthcare, and government sectors. However, they remain susceptible to advanced threats such as polymorphic SQL injection, insider misuse, and timing-based data exfiltration.Traditional intrusion detection systems (IDSs), including signature-based and static machine learning models, often struggle to detect evolving and obfuscated attacks, resulting in high false-positive rates and limited adaptability.This paper presents the Triple Loop Learning (TLL) framework—a recursive, multi-tiered architecture that integrates supervised learning, reinforcement learning (RL), and meta-learning for real-time anomaly detection in SQL-based environments. To the best of our knowledge, TLL is the first framework to unify these three learning paradigms into a recursive, explainable, and low-latency anomaly detection system specifically tailored for structured SQL workloads.TLL comprises three interdependent loops: (i) an operational loop utilizing SQL-aware autoencoders for fine-grained anomaly detection, (ii) a tactical loop leveraging Deep Q-Networks (DQN) for adaptive threshold calibration, and (iii) a strategic loop based on Model-Agnostic Meta-Learning (MAML) for rapid cross-domain adaptation.TLL was evaluated on a composite dataset comprising the UNSW-NB15 benchmark, anonymized SQL logs from financial and healthcare institutions, and adversarial samples generated with MAD-GAN. The framework achieved an F1 score of 0.94, an AUC of 0.976, and an average inference latency of 84 ms.The results demonstrate that TLL enables robust, low-latency detection of zero-day and obfuscated SQL threats. Its explain-ability is enhanced through SHAP-based attributions, and its deployment is version-controlled and reproducible, ensuring traceability, compliance, and production readiness.

The active anomaly detection system of cyber physical power system (CPPS) can make up for the shortcomings of traditional isolation-based defense methods, to detect cyberattacks that breach the security boundary. The traditional detection method based on only communication side features may have high false positive rate and false negative rate. To address this defect, an abnormal traffic detection method is proposed from the perspective of cyber-physical combination, considering the influence of power system state on the communication network traffic. Firstly, based on communication behavior features, a data-driven anomaly detection method is proposed to realize the preliminary screening of abnormal flow. At the same time, the corresponding physical features of abnormal flow are extracted. Then cyberattack flows can be distinguished from the abnormal flows through an abnormal source tracing method based on bilateral characteristics. Based on the cosimulation platform of CPPS, it is verified that the proposed method can reduce the rate of false positive while ensuring a high detection accuracy against cyberattacks.

This study investigates the effectiveness of machine learning-based anomaly detection systems for cyber threat prevention, employing a quantitative research design with primary data collected from 400 cybersecurity professionals, IT administrators, and network security experts. The study investigated the implementation, efficacy, and constraints of machine learning (ML) algorithms by means of statistical examination, regression analysis, and the utilization of methodologies including decision trees, random forests, and support vector machines. A structured questionnaire was used to gather primary data for the study and sample size was 400. Findings reveal that 68% of organizations utilize ML for threat detection, with the financial sector leading at 75%. Despite promising adoption rates, challenges such as high false positive rates (54%), zero-day threat detection difficulties (41%), and data imbalance (60%) persist. Real-time learning and better integration with security infrastructure were highlighted as crucial for improving threat detection accuracy and system adaptability. While 72% of respondents viewed ML as effective, most emphasized the need for enhanced interoperability, false positive management, and incident response automation. These revelations highlight the progressive function of machine learning within the cybersecurity domain and the critical necessity for ongoing system enhancement in anticipation of forthcoming threat environments.

OHODIN is an online extension for data streams of the kNN-based ODIN anomaly detection approach. It provides a detection-threshold heuristic that is based on extreme value theory. In contrast to sophisticated anomaly and novelty detection approaches the decision-making process of ODIN is interpretable by humans, making it interesting for certain applications. However, it is limited in terms of the underlying detection method. In this article, we present an extension of the OHODIN to further detection techniques to reinforce OHODIN capability of online data streams anomaly detection. We introduce the algorithm modifications and an experimental evaluation with competing state-of-the-art anomaly detection approaches.

Although machine learning (ML) based solutions are ever-evolving for the attack defending paradigm, signatures of malicious network traffic are vital resources for intrusion detection systems (IDSs) and network forensic procedure, covering the lack of interpretability and stability for ML models. However, signature extraction is still a time and labor consuming task nowadays, resulting in possible increase of the attackers' dwell time. Existing automatic solutions rely too much on sequence similarity based and heuristic based methods, encountering performance degradation in large scale and dynamic network environment. In this paper, we present a novel method, called Clustering and Model Inference-based Rule Generation (CMIRGen), automatically generating token-set based signature rules for malicious traffic payloads to be inspected. CMIRGen leverages both optimized sequence similarity based and black-box model inference based methods to extract patterns from homogeneous and heterogeneous payloads respectively. Experimental evaluations have been conducted on several datasets and show the CMIRGen framework can extract discriminative signatures, presenting high recall rate and low false positive rate at the same time for malicious content recognition.

The rapid growth of network data has increased the complexity of detecting anomalies, which are crucial for ensuring the security and integrity of information systems. This study investigates the use of the Isolation Forest algorithm for anomaly detection in network traffic, utilizing the Luflow Network Intrusion Detection dataset, which contains 590,086 records with 16 features related to network activities. The methodology encompasses data preprocessing (cleaning, normalization, and feature scaling), feature selection (bytes in, bytes out, entropy, and duration), model training, and performance evaluation. The results demonstrate that Isolation Forest can effectively identify anomalies based on feature patterns, isolating suspicious data points without the need for labeled datasets. However, performance metrics, such as accuracy (42.92%), precision (14.37%), recall (2.87%), and F1-score (4.79%), reveal challenges such as high false-positive rates and low sensitivity to true anomalies. These findings highlight the potential of the algorithm for dynamic, high-dimensional datasets but also indicate the need for further improvements through hyperparameter tuning, feature engineering, and alternative approaches. This study contributes to the development of adaptive anomaly detection frameworks for network security and suggests future integration into real-time systems for proactive threat mitigation. The study's findings are particularly relevant for enhancing network security in environments such as corporate and governmental networks, where real-time anomaly detection is crucial.

Microservice architecture has been widely used by more and more developers in recent years. Accurate anomaly detection is crucial for system maintenance. Trace data can reflect the microservice dependency relationship and response time, which has been adopted for microservice anomaly detection now. However, due to the lack of unification modeling framework of response time and call path, the performance of anomaly detection degrades, and difficult to adapt to downstream tasks. To address the above issues, we propose BSDG, a trace anomaly detection method based on a dual graph convolutional neural network (dualGCN). First, BSDG extracts the microservice call dependencies, combing the learnable node attributes generated by Bi-directional Long Short-Term Memory(BiLSTM) to build an attribute dependency graph combined response time and call path. Then, a self-attention mapping graph is constructed and we use a dualGCN with mutual attention to generate effective feature embedding representation. Finally, BSDG adopts a multilayer perceptron with a new classification loss function to train the model in an end-to-end way for anomaly detection. The experimental results on public benchmarks show that the proposed BDSG outperforms baseline methods. We also conduct experiments on our constructed microservice trace dataset to validate the robustness of BSDG. Experiments show that the BSDG outperforms existing methods in microservice trace anomaly detection. KeywordsMicroservicesTracedualGCNAnomaly detection

Recently, artificial intelligence (AI) techniques have been used to describe the characteristics of information, as they help in the process of data mining (DM) to analyze data and reveal rules and patterns. In DM, anomaly detection is an important area that helps discover hidden behavior within the data that is most vulnerable to attack. It also helps detect network intrusion. Algorithms such as hybrid K-mean array and sequential minimal optimization (SMO) rating can be used to improve the accuracy of the anomaly detection rate. This paper presents an anomaly detection model based on the machine learning (ML) technique. ML improves the detection rate, reduces the false-positive alarm rate, and is capable of enhancing the accuracy of intrusion classification. This study used a dataset known as network security-knowledge and data discovery (NSL-KDD) lab to evaluate a proposed hybrid ML technology. K-mean cluster and SMO were used for classification. In the study, the performance of the proposed anomaly detection was tested, and results showed that the use of K-mean and SMO enhances the rate of positive detection besides reducing the rate of false alarms and achieving a high accuracy at the same time. Moreover, the proposed algorithm outperformed recent and close work related to using similar variables and the environment by 14.48% and decreased false alarm probability (FAP) by (12%) in addition to giving a higher accuracy by 97.4%. These outcomes are attributed to the common algorithm providing an appropriate number of detectors to be generated with an acceptable accurate detection and a trivial false alarm probability (FAP). The proposed hybrid algorithm could be considered for anomaly detection in future data mining systems, where processing in real-time is highly likely to be reduced dramatically. The justification is that the hybrid algorithm can provide appropriate detectors numbers that can be generated with an acceptable detection accuracy and trivial FAP. Given to the low FAP, it is highly expected to reduce the time of the preprocessing and processing compared with the other algorithms.

Anomaly detection is critical for network security, fraud detection, and system health monitoring applications. Traditional methods like statistical approaches and distance-based techniques often struggle with high-dimensional and complex data, leading to high false positive rates. This study addresses the challenge by investigating advanced AI-driven techniques to reduce false positives and enhance data security within cloud computing environments. This study employs deep learning models, integrates contextual data, and incorporates comprehensive security measures to enhance anomaly detection performance. Data from synthetic sources, such as the NSL-KDD dataset and real-world cloud environments, were utilized to capture user behavior logs, system states, and network traffic. Over 50 academic journals were reviewed, and 21 were selected based on inclusion criteria, such as relevance to AI-driven anomaly detection, empirical performance metrics, and the focus on cloud environments, and exclusion criteria that filtered out studies lacking empirical data or not specific to cloud-based systems. Methodologically, the research involves a comparative analysis of different AI techniques and their impact on false positive rates, accuracy, precision, and recall. The findings demonstrate that deep learning techniques significantly outperform traditional methods, achieving a lower false positive rate and higher accuracy. The results underscore the importance of contextual data and robust security protocols in reliable anomaly detection. This research fills a gap by thoroughly evaluating advanced AI techniques for reducing false positives in cloud environments. The study's significance lies in guiding the development of more effective anomaly detection systems, thereby enhancing security and reliability across various applications. Additionally, organizations should invest in continuously developing and integrating AI-driven anomaly detection systems with comprehensive security measures to improve their effectiveness the study suggests that further study be conducted with large datasets to evaluate the effectiveness of Hybrid anomaly detection systems in detecting and addressing false positives.