SALAD: Self-supervised Aggregation Learning for Anomaly Detection on X-Rays

In the last decade, Deep neural networks (DNNs) have been proven to outperform conventional machine learning models in supervised learning tasks. Most of these models are typically optimized by minimizing the well-known Cross-Entropy objective function. The latter, however, has a number of drawbacks, including poor margins and instability. Taking inspiration from the recent self-supervised Contrastive representation learning approaches, we introduce Supervised Contrastive learning framework for Textual representations (SuperConText) to address those issues.We pretrain a neural network by minimizing a novel fully-supervised contrastive loss. The goal is to increase both inter-class separability and intra-class compactness of the embeddings in the latent space. Examples belonging to the same class are regarded as positive pairs, while examples belonging to different classes are considered negatives. Further, we propose a simple yet effective method for selecting hard negatives during the training phase. In an extensive series of experiments, we study the impact of a number parameters on the quality of the learned representations (e.g. the batch size). Simulation results show that the proposed solution outperforms several competing approaches on various large-scale text classification benchmarks without requiring specialized architectures, data augmentations, memory banks, or additional unsupervised data. For instance, we achieve top-1 accuracy of 61.94% on the Amazon-F dataset, which is 3.54% above the best result obtained when using the cross-entropy with the same model architecture.

In recent years, the identification of abnormalities in attributed networks has become essential for applications including social media analysis, cybersecurity, and financial fraud detection. Unsupervised graph anomaly detection techniques seek to recognize infrequent and anomalous patterns in graph-structured data without the necessity of labelled instances. Conventional methods employing Graph Neural Networks (GNNs) frequently encounter difficulties, especially due to the transmission of noisy edges and the intrinsic intricacy of node interrelations. To overcome these restrictions, we introduce ANOGAT-Sparse-TL, an innovative hybrid framework that integrates graph sparsification and Graph Attention Networks (GAT) with autoencoder-based reconstruction for anomaly detection in attributed networks. The sparsification procedure removes extraneous edges and highlights significant node connections, thereby enhancing computational efficiency and improving anomaly detection efficacy. By including GAT, our model carefully allocates significance to pertinent neighboring nodes, yielding enhanced node embeddings. The autoencoder subsequently reconstructs these embeddings to detect abnormalities via reconstruction errors. Incorporating Tversky Loss in the reconstruction process further improves the robustness of the model by effectively addressing the imbalance between normal and anomalous data, prioritizing the detection of rare anomalies. This optimized loss function allows ANOGAT-Sparse-TL to focus on hard-to-reconstruct instances, which are typically indicative of anomalies, and reduces the impact of noisy data on the model's performance. ANOGAT-Sparse-TL effectively integrates attribute-based and structural anomalies, yielding comprehensive anomaly ratings. Comprehensive studies on the four real-world datasets indicate that our strategy surpasses current state-of-the-art methodologies, with enhanced performance. Moreover, the scalability of our methodology guarantees its relevance to extensive real-world networks, rendering it an adaptable option for diverse graph anomaly detection activities. ANOGAT-Sparse-TL, despite its complexity, maintains computational efficiency and provides substantial improvements in anomaly detection inside attributed networks. Future research may concentrate on enhancing interpretability and broadening generalizability to various network architectures.

Time series anomaly detection is an important research topic in the field of intelligent operation and maintenance. When software systems are frequently updated with continuous integration and deployment, the distribution of KPI data will also change, and the accuracy of anomaly detection models will inevitably decrease. To tackle this problem, we propose an active anomaly detection framework named Active-MTSAD suitable for multi-dimensional time series, combining unsupervised anomaly detection and active learning. The active learning module introduces three feedback strategies, namely denominator penalty, negative penalty, and metric learning, to learn new anomalous patterns under new data distribution. In metric learning, we consider the difference between normal and abnormal samples in reconstruction error and latent space. We conduct extensive experiments on a large-scale public dataset and a real-world dataset coming from Tencent. The experimental results show that Active-MTSAD can still achieve excellent performance in real scenarios where the distribution changes with only 0.2% of labels.

Deep neural networks excel in supervised learning tasks but are constrained by the need for extensive labeled data. Self-supervised learning emerges as a promising alternative, allowing models to learn without explicit labels. Information theory has shaped deep neural networks, particularly the information bottleneck principle. This principle optimizes the trade-off between compression and preserving relevant information, providing a foundation for efficient network design in supervised contexts. However, its precise role and adaptation in self-supervised learning remain unclear. In this work, we scrutinize various self-supervised learning approaches from an information-theoretic perspective, introducing a unified framework that encapsulates the self-supervised information-theoretic learning problem. This framework includes multiple encoders and decoders, suggesting that all existing work on self-supervised learning can be seen as specific instances. We aim to unify these approaches to understand their underlying principles better and address the main challenge: many works present different frameworks with differing theories that may seem contradictory. By weaving existing research into a cohesive narrative, we delve into contemporary self-supervised methodologies, spotlight potential research areas, and highlight inherent challenges. Moreover, we discuss how to estimate information-theoretic quantities and their associated empirical problems. Overall, this paper provides a comprehensive review of the intersection of information theory, self-supervised learning, and deep neural networks, aiming for a better understanding through our proposed unified approach.

Unsupervised anomaly localization aims to identify anomalous regions that deviate from normal sample patterns. Most recent methods perform feature matching or reconstruction for the target sample with pre‐trained deep neural networks. However, they still struggle to address challenging anomalies because the deep embeddings stored in the memory bank can be less powerful and informative. Specifically, prior methods often overly rely on the finite resources stored in the memory bank, which leads to low robustness to unseen targets. In this paper, we propose a novel subspace‐guided feature reconstruction framework to pursue adaptive feature approximation for anomaly localization. It first learns to construct low‐dimensional subspaces from the given nominal samples, and then learns to reconstruct the given deep target embedding by linearly combining the subspace basis vectors using the self‐expressive model. Our core is that, despite the limited resources in the memory bank, the out‐of‐bank features can be alternatively “mimicked” to adaptively model the target. Moreover, we propose a sampling method that leverages the sparsity of subspaces and allows the feature reconstruction to depend only on a small resource subset, contributing to less memory overhead. Extensive experiments on three benchmark datasets demonstrate that our approach generally achieves state‐of‐the‐art anomaly localization performance.

Do deep neural networks contribute to multivariate time series anomaly detection?

Considering several patients screened due to COVID-19 pandemic, computer-aided detection has strong potential in assisting clinical workflow efficiency and reducing the incidence of infections among radiologists and healthcare providers. Since many confirmed COVID-19 cases present radiological findings of pneumonia, radiologic examinations can be useful for fast detection. Therefore, chest radiography can be used to fast screen COVID-19 during the patient triage, thereby determining the priority of patient's care to help saturated medical facilities in a pandemic situation. In this paper, we propose a new learning scheme called self-supervised transfer learning for detecting COVID-19 from chest X-ray (CXR) images. We compared six self-supervised learning (SSL) methods (Cross, BYOL, SimSiam, SimCLR, PIRL-jigsaw, and PIRL-rotation) with the proposed method. Additionally, we compared six pretrained DCNNs (ResNet18, ResNet50, ResNet101, CheXNet, DenseNet201, and InceptionV3) with the proposed method. We provide quantitative evaluation on the largest open COVID-19 CXR dataset and qualitative results for visual inspection. Our method achieved a harmonic mean (HM) score of 0.985, AUC of 0.999, and four-class accuracy of 0.953. We also used the visualization technique Grad-CAM++ to generate visual explanations of different classes of CXR images with the proposed method to increase the interpretability. Our method shows that the knowledge learned from natural images using transfer learning is beneficial for SSL of the CXR images and boosts the performance of representation learning for COVID-19 detection. Our method promises to reduce the incidence of infections among radiologists and healthcare providers.

Time series anomaly detection is the process of identifying anomalies within time series data. The primary challenge of this task lies in the necessity for the model to comprehend the characteristics of time-independent and abnormal data patterns. In this study, a novel algorithm called adaptive memory broad learning system (AdaMemBLS) is proposed for time series anomaly detection. This algorithm leverages the rapid inference capabilities of the broad learning algorithm and the memory bank's capacity to differentiate between normal and abnormal data. Furthermore, an incremental algorithm based on multiple data augmentation techniques is introduced and applied to multiple ensemble learners, thereby enhancing the model's effectiveness in learning the characteristics of time series data. To bolster the model's anomaly detection capabilities, a more diverse ensemble approach and a discriminative anomaly score are recommended. Extensive experiments conducted on various real-world datasets demonstrate that the proposed method exhibits superior inference speed and more accurate anomaly detection compared to the existing competitors. A detailed experimental investigation is presented to elucidate the effectiveness of the proposed method and the underlying reasons for its efficacy.

Since deep neural networks can learn data representation from training data automatically, deep learning methods are widely used in the network anomaly detection. However, challenges of deep learning-based anomaly detection methods still exist, the major of which is the training data scarcity problem. In this paper, we propose a novel network anomaly detection method (NAFT) using federated learning and transfer learning to overcome the data scarcity problem. In the first learning stage, a people or organization \(O_t\), who intends to conduct a detection model for a specific attack, can join in the federated learning with a similar training task to learn basic knowledge from other participants’ training data. In the second learning stage, \(O_t\) uses the transfer learning method to reconstruct and re-train the model to further improve the detection performance on the specific task. Experiments conducted on the UNSW-NB15 dataset show that the proposed method can achieve a better anomaly detection performance than other baseline methods when training data is scarce.

The image reconstruction process in medical imaging can be treated as solving an inverse problem. The inverse problem is usually solved using time-consuming iterative algorithms with sparsity or other constraints. Recently, deep neural network (DNN)-based methods have been developed to accelerate the inverse-problem-solving process. However, these methods typically adopt supervised learning scheme, which requires ground truths, or labels of the solutions, for training. In many applications, it would be challenging or even impossible to obtain the ground truth, such as the tissue reflectivity function in ultrasound beamforming. In this study, a general framework based on self-supervised learning (SSL) scheme is proposed to train a DNN to solve the inverse problems. In this way, the measurements can be used as both the inputs and the labels during the training of DNN. The proposed SSL method is applied to four typical linear inverse problems for validation, i.e., plane wave ultrasound and photoacoustic image reconstructions, compressed sensing-based synthetic transmit aperture dataset recovery and deconvolution in ultrasound localization microscopy. Results show that, using the proposed framework, the trained DNN can achieve improved reconstruction accuracy with reduced computational time, compared with conventional methods.

The Coronavirus was detected in Wuhan, China in late 2019 and then led to a\npandemic with a rapid worldwide outbreak. The number of infected people has\nbeen swiftly increasing since then. Therefore, in this study, an attempt was\nmade to propose a new and efficient method for automatic diagnosis of Corona\ndisease from X-ray images using Deep Neural Networks (DNNs). In the proposed\nmethod, the DensNet169 was used to extract the features of the patients' Chest\nX-Ray (CXR) images. The extracted features were given to a feature selection\nalgorithm (i.e., ANOVA) to select a number of them. Finally, the selected\nfeatures were classified by LightGBM algorithm. The proposed approach was\nevaluated on the ChestX-ray8 dataset and reached 99.20% and 94.22% accuracies\nin the two-class (i.e., COVID-19 and No-findings) and multi-class (i.e.,\nCOVID-19, Pneumonia, and No-findings) classification problems, respectively.\n

Automatic seizure detection and classification using super-resolution superlet transform and deep neural network -A preprocessing-less method

The prevalence of streaming data across various sectors poses significant challenges for real-time anomaly detection due to its volume, velocity, and variability. Traditional data processing methods often need to be improved for such dynamic environments, necessitating robust, scalable, and efficient real-time analysis systems. This study compares two advanced machine learning approaches—LSTM autoencoders and Matrix Profile algorithms—to identify the most effective method for anomaly detection in streaming environments using the NYC taxi dataset. Existing literature on anomaly detection in streaming data highlights various methodologies, including statistical tests, window-based techniques, and machine learning models. Traditional methods like the Generalized ESD test have been adapted for streaming data but often require a full historical dataset to function effectively. In contrast, machine learning approaches, particularly those using LSTM networks, are noted for their ability to learn complex patterns and dependencies, offering promising results in real-time applications. In a comparative analysis, LSTM autoencoders significantly outperformed other methods, achieving an F1-score of 0.22 for anomaly detection, notably higher than other techniques. This model demonstrated superior capability in capturing temporal dependencies and complex data patterns, making it highly effective for the dynamic and varied data in the NYC taxi dataset. The LSTM autoencoder's advanced pattern recognition and anomaly detection capabilities confirm its suitability for complex, high-velocity streaming data environments. Future research should explore the integration of LSTM autoencoders with other machine-learning techniques to enhance further the accuracy, scalability, and efficiency of anomaly detection systems. This study advances our understanding of scalable machine-learning approaches and underscores the critical importance of selecting appropriate models based on the specific characteristics and challenges of the data involved.

Oil and Gas organizations are dependent on their IT infrastructure, which is a small part of their industrial automation infrastructure, to function effectively. The oil and gas (O&G) organizations industrial automation infrastructure landscape is complex. To perform focused and effective studies, Industrial systems infrastructure is divided into functional levels by The Instrumentation, Systems and Automation Society (ISA) Standard ANSI/ISA-95:2005. This research focuses on the ISA-95:2005 level-4 IT infrastructure to address network anomaly detection problem for ensuring the security and reliability of Oil and Gas resource planning, process planning and operations management. Anomaly detectors try to recognize patterns of anomalous behaviors from network traffic and their performance is heavily dependent on extraction time and quality of network traffic features or representations used to train the detector. Creating efficient representations from large volumes of network traffic to develop anomaly detection models is a time and resource intensive task. In this study we propose, implement and evaluate use of Deep learning to learn effective Network data representations from raw network traffic to develop data driven anomaly detection systems. Proposed methodology provides an automated and cost effective replacement of feature extraction which is otherwise a time and resource intensive task for developing data driven anomaly detectors. The ISCX-2012 dataset is used to represent ISA-95 level-4 network traffic because the O&G network traffic at this level is not much different than normal internet traffic. We trained four representation learning models using popular deep neural network architectures to extract deep representations from ISCX 2012 traffic flows. A total of sixty anomaly detectors were trained by authors using twelve conventional Machine Learning algorithms to compare the performance of aforementioned deep representations with that of a human-engineered handcrafted network data representation. The comparisons were performed using well known model evaluation parameters. Results showed that deep representations are a promising feature in engineering replacement to develop anomaly detection models for IT infrastructure security. In our future research, we intend to investigate the effectiveness of deep representations, extracted using ISA-95:2005 Level 2-3 traffic comprising of SCADA systems, for anomaly detection in critical O&G systems.

This study focuses on the application of neural networks for anomaly detection in microservice architectures. The increasing complexity of modern microservice systems makes traditional anomaly detection methods less effective, as they struggle to adapt to dynamic environments and large volumes of data. Neural networks offer a promising solution due to their ability to model complex patterns and detect subtle deviations that may indicate system failures or security threats. The aim of this work is to compare the effectiveness of different neural network-based approaches to anomaly detection, evaluate their strengths and weaknesses, and determine the most suitable method for practical implementation. To achieve this, several key tasks were outlined: an in-depth review of contemporary anomaly detection techniques applied to microservices; an analysis of neural network-based models, with a particular focus on autoencoders and recurrent neural networks (RNNs); a comparative assessment of these methods in terms of accuracy, computational efficiency, and robustness to noise; an exploration of hybrid models that combine different techniques to improve detection performance; and the formulation of recommendations for selecting the optimal approach based on specific application scenarios. This research examines the use of deep neural networks for detecting behavioral anomalies in microservice environments. Various architectures are tested and compared to identify trade-offs between detection accuracy, processing speed, and resistance to false positives. The findings indicate that deep neural networks, particularly autoencoders and recurrent models, achieve high precision in identifying anomalies, but their efficiency varies depending on the architecture and the nature of the dataset. The study highlights that while autoencoders effectively capture deviations in data distribution, recurrent networks excel in detecting temporal anomalies in sequential microservice interactions. The conclusions confirm that neural networks provide a powerful tool for anomaly detection in microservice monitoring, significantly outperforming conventional statistical approaches. However, challenges remain, including the need for adaptive models capable of learning in real time and optimizing computational resources. Further research should focus on hybrid approaches that integrate multiple neural architectures to enhance detection accuracy and scalability. The development of self-learning and auto-tuning models could also improve adaptability to evolving microservice behaviors, ensuring robust and efficient anomaly detection in complex distributed systems.