Convolutional autoencoder based on latent subspace projection for anomaly detection.

BackgroundUnsupervised clustering and outlier detection are important in medical research to understand the distributional composition of a collective of patients. A number of clustering methods exist, also for high-dimensional data after dimension reduction. Clustering and outlier detection may, however, become less robust or contradictory if multiple high-dimensional data sets per patient exist. Such a scenario is given when the focus is on 3-D data of multiple organs per patient, and a high-dimensional feature matrix per organ is extracted.MethodsWe use principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE) and multiple co-inertia analysis (MCIA) combined with bagplots to study the distribution of multi-organ 3-D data taken by computed tomography scans. After point-set registration of multiple organs from two public data sets, multiple hundred shape features are extracted per organ. While PCA and t-SNE can only be applied to each organ individually, MCIA can project the data of all organs into the same low-dimensional space.ResultsMCIA is the only approach, here, with which data of all organs can be projected into the same low-dimensional space. We studied how frequently (i.e., by how many organs) a patient was classified to belong to the inner or outer 50% of the population, or as an outlier. Outliers could only be detected with MCIA and PCA. MCIA and t-SNE were more robust in judging the distributional location of a patient in contrast to PCA.ConclusionsMCIA is more appropriate and robust in judging the distributional location of a patient in the case of multiple high-dimensional data sets per patient. It is still recommendable to apply PCA or t-SNE in parallel to MCIA to study the location of individual organs.

Outlier detection aims to find a data sample that is significantly different from other data samples. Various outlier detection methods have been proposed and have been shown to be able to detect anomalies in many practical problems. However, in high dimensional data, conventional outlier detection methods often behave unexpectedly due to a phenomenon called the curse of dimensionality. In this paper, we compare and analyze outlier detection performance in various experimental settings, focusing on text data with dimensions typically in the tens of thousands. Experimental setups were simulated to compare the performance of outlier detection methods in unsupervised versus semi-supervised mode and uni-modal versus multi-modal data distributions. The performance of outlier detection methods based on dimension reduction is compared, and a discussion on using k-NN distance in high dimensional data is also provided. Analysis through experimental comparison in various environments can provide insights into the application of outlier detection methods in high dimensional data.

Anomaly detection for core temperature has great significance in maintaining the safety of nuclear power plants. However, traditional auto-encoder-based anomaly detection methods might extract the latent space features with redundancy, which may lead to missing and false alarms. To address this problem, the idea of feature disentangling is introduced under the auto-encoder framework in this paper. First, a feature disentangling auto-encoder (DAE) is proposed where a latent space disentangling loss is designed to disentangle the features. We further propose an incrementally feature disentangling auto-encoder (IDAE), which is the improved version of DAE. In the IDAE model, an incremental feature generation strategy is developed, which enables the model to evaluate the disentangling degree to adaptively determine the feature dimension. Furthermore, an iterative training framework is designed, which focuses on the parameter training of the newly incremented feature, overcoming the difficulty of model training. Finally, we illustrate the effectiveness and superiority of the proposed method on a real nuclear reactor core temperature dataset. IDAE achieves average false alarm rates of 4.745% and 6.315%, respectively, using two monitoring statistics, and achieves average missing alarm rates of 6.4% and 2.9%, respectively, using two monitoring statistics, outperforming the other methods.

There are a large number of high-dimensional data in the Internet of Brain Things, and the data is reduced from the high-dimensional to low-dimensional to maintain the similarity between the data, thereby effectively ensuring the operating speed of the Internet of Brain Things. In the traditional method, the distributed dimensionality reduction reconstruction algorithm of the High dimensional data has poor dimensionality reduction effect and serious data distortion after reconstruction. A method for dimensionality reconstruction of high-dimensional data in the Internet of Brain Things is proposed. By using the algorithm of linear discriminant analysis, the projection matrix of high dimensional data is constructed to solve it. According to the solution results, using improved implicit variable model to establish a high dimensional large data dimensionality reduction model for the Internet of Brain Things. The fitness value of data after dimensionality reduction is calculated by quantum immune clonal algorithm, and the optimal individual and optimal solution are determined. The data reconfiguration is realized through the optimal solution marshalling. Experimental results show that the proposed algorithm can effectively improve the dimensionality reduction of High dimensional data in the Internet of Brain Things. After reconstruction, the reconstructed data retain accurate data information, the reliability of reconstructed data is high, and the computational complexity is not high, the need for small storage space, and the advantages of strong promotion ability.

Unsupervised learning-based anomaly detection using autoencoders has gained importance since anomalies behave differently than normal data when reconstructed from a well-regularized latent space. Existing research shows that retaining valuable properties of input data in latent space helps in the better reconstruction of unseen data. Moreover, real-world sensor data is skewed and non-Gaussian in nature rendering mean-based estimators unreliable for such cases. Reconstructionbased anomaly detection methods rely on Euclidean distance as the reconstruction error which does not consider useful correlation information in the latent space. In this work, we address some of the limitations of the Euclidean distance when used as a reconstruction error to detect anomalies (especially near anomalies) that have a similar distribution as the normal data in the feature space. We propose a latent dimension regularized autoencoder that leverages a robust form of the Mahalanobis distance (MD) to measure the latent space correlation to effectively detect near as well as far anomalies. We showcase that incorporating the correlation information in the form of robust MD in the latent space is quite helpful in separating both near and far anomalies in the reconstructed space.

Cyberattacks on computer networks are becoming increasingly sophisticated, particularly in Internet of Things (IoT) environments, where devices continuously generate large and complex amounts of data. Traditional security systems that rely on predefined rules or signatures often fail to detect new or evolving threats. Even deep learning methods, such as RNNs and CNNs, face challenges in handling dynamic traffic efficiently. To address these issues, this study introduces an autoencoder-based anomaly detection model that learns to identify abnormal network activities. The model was trained using Kaggle datasets containing both normal IoT traffic and malicious traffic from well-known botnets like Mirai and BASHLITE. By compressing network data into a latent space and reconstructing it, the model uses reconstruction error to detect unusual patterns that indicate anomalies. The experimental results were highly promising, achieving 99% accuracy, precision, recall, and F1-scores above 99%. Unlike previous studies that depend on simulated or cloud-based data, this research highlights the power of autoencoders for real-world IoT anomaly detection and lays a strong foundation for developing real-time intrusion detection systems.

In many image-related tasks, learning expressive and discriminative representations of images is essential, and deep learning has been studied for automating the learning of such representations. Some user-centric tasks, such as image recommendations, call for effective representations of not only images but also preferences and intents of users over images. Such representations are termed hybrid and addressed via a deep learning approach in this paper. We design a dual-net deep network, in which the two sub-networks map input images and preferences of users into a same latent semantic space, and then the distances between images and users in the latent space are calculated to make decisions. We further propose a comparative deep learning (CDL) method to train the deep network, using a pair of images compared against one user to learn the pattern of their relative distances. The CDL embraces much more training data than naive deep learning, and thus achieves superior performance than the latter, with no cost of increasing network complexity. Experimental results with real-world data sets for image recommendations have shown the proposed dual-net network and CDL greatly outperform other state-of-the-art image recommendation solutions.

Molecular gene-expression datasets consist of samples with tens of thousands of measured quantities (i.e., high dimensional data). However, lower-dimensional representations that retain the useful biological information do exist. We present a novel algorithm for such dimensionality reduction called Pathway Activity Score Learning (PASL). The major novelty of PASL is that the constructed features directly correspond to known molecular pathways (genesets in general) and can be interpreted as pathway activity scores. Hence, unlike PCA and similar methods, PASL’s latent space has a fairly straightforward biological interpretation. PASL is shown to outperform in predictive performance the state-of-the-art method (PLIER) on two collections of breast cancer and leukemia gene expression datasets. PASL is also trained on a large corpus of 50000 gene expression samples to construct a universal dictionary of features across different tissues and pathologies. The dictionary validated on 35643 held-out samples for reconstruction error. It is then applied on 165 held-out datasets spanning a diverse range of diseases. The AutoML tool JADBio is employed to show that the predictive information in the PASL-created feature space is retained after the transformation. The code is available at https://github.com/mensxmachina/PASL.

AI-generated content (AIGC) in the context of dermoscopic image analysis describes the application of artificial intelligence (AI) approaches to produce synthetic images for training and enriching machine learning (ML) methods. In dermatology, where dermoscopic images are essential in skin lesion detection, AIGC helps overcome the challenges of inadequate datasets and class imbalances. By employing advanced generative complex systems like Generative Adversarial Networks (GANs), AIGC will simulate different dermoscopic conditions by generating realistic-view synthetic images. These AI-generated images help to enhance the training dataset, contributing to the ML models with many widespread and varied sets of instances. This aids in increasing the ability of the model to precisely predict and generalize while handling new, unnoticed dermoscopic images. Deep learning (DL) methods, specifically convolutional neural networks (CNNs), can provide a massive breakthrough in an extensive range of computer vision (CV) tasks of complex systems, predominantly by exploiting large-scale annotated datasets. This paper develops a GAN-based Synthetic Medical Image Augmentation for the imbalanced Dermoscopic Image Analysis (GANSMIA-CDIA) method. The GANSMIA-CDIA technique’s primary target is exploiting the GAN model for synthetic image generation to handle class imbalance data problems. In addition, the GANSMIA-CDIA technique diagnoses the melanoma using the optimal DL model. The GANSMIA-CDIA technique applies a contrast enhancement process for the noise eradication process. The GANSMIA-CDIA technique follows a feature fusion method comprising different DL methods such as MobileNetV2, AlexNet, ResNet50, and Inceptionv3 to learn feature patterns in the preprocessed images. Meanwhile, the Social Network Search (SNS) technique is utilized for the hyperparameter tuning process. Also, the bidirectional long short-term memory (BiLSTM) technique is implemented to detect and classify melanoma. A series of simulations were performed on the standard dermoscopic image dataset to evaluate the performance of the GANSMIA-CDIA technique. The experimental values indicate the excellence of the GANSMIA-CDIA technique over existing techniques.

Salient object detection in complex scenes and environments is a challenging research topic. Most works focus on RGB-based salient object detection, which limits its performance of real-life applications when confronted with adverse conditions such as dark environments and complex backgrounds. Taking advantage of RGB and thermal infrared(RGBT) images becomes a new research direction for detecting salient objects in complex scenes, since the thermal infrared spectrum provides the complementary information and has been used in many computer vision tasks. However, current research for RGBT salient object detection is limited by the lack of a large-scale dataset and comprehensive benchmark. This work contributes such a RGBT image dataset named VT5000, including 5000 spatially aligned RGBT image pairs with ground truth annotations. VT5000 has 11 challenges collected in different scenes and environments for exploring the robustness of algorithms. With this dataset, we propose a powerful baseline approach, which extracts multilevel features of each modality and aggregates these features of all modalities with the attention mechanism for accurate RGBT salient object detection. To further solve the problem of blur boundaries of salient objects, we also use an edge loss to refine the boundaries. Extensive experiments show that the proposed baseline approach outperforms the state-of-the-art methods on VT5000 dataset and other two public datasets. In addition, we carry out a comprehensive analysis of different algorithms of RGBT salient object detection on VT5000 dataset, and then make several valuable conclusions and provide some potential research directions for RGBT salient object detection. Our new VT5000 dataset is made publicly available at https://github.com/lz118/RGBT-Salient-Object-Detection.

Incomplete mixed data-driven outlier detection based on local–global neighborhood information

The curse of dimensionality in high-dimensional data makes it difficult to capture the abnormality of data points in full data space. To deal with this problem, we propose an outlier detection model based on Variational Autoencoder and Genetic Algorithm for subspace outlier analysis of high-dimensional data (VAGA). The proposed VAGA model constructs a variational autoencoder (VAE) to preliminarily detect outliers. Then the genetic algorithm (GA) is used to search the abnormal subspace of the outliers obtained by the VAE layer to provide a basis for subspace outlier analysis. The subsequent clustering of the abnormal subspaces help filter out the false positives which are fed back to the VAE layer to adjust network weights. The comparative experiments performed on three public benchmark datasets show that the outlier detection results of the proposed VAGA model are highly interpretable and have better accuracy performance than the state-of-the-art outlier detection methods.

User Anomaly Detection, an important aspect of User Behaviour Analysis, is used to find anomalous user events from the event and network traffic log data. Traditional security mechanisms are not able to detect new/unknown types of anomalies as they do not incorporate contextual and behavioural aspects of the data for analysis. User Behaviour Anomaly Detection (UBAD) employs user behavioural patterns in context thereby achieving a higher detection rate. Log data employed for UBAD has multiple dimensions. With increase in dimensions (attributes), data gets sparse and the detection of anomalies becomes increasingly complex. For this reason, the single dimensional algorithms proposed for anomaly detection based on clustering, proximity and dimensional reduction do not work well on higher dimension data. OLAP based data analysis techniques provide efficient data slicing and aggregation operations crucial for multidimensional analytics along with multiscale visualisation for exploratory discovery. In this paper, an effective multidimensional process for UBAD is developed to detect anomalies using multi-dimensional statistical tests. An integral part of the process is the development of an OLAP Cube data model for event log data. The statistical efficiency of the UBAD process for different dimensions is investigated. On a real-life event log data, it is shown that the statistical efficiency of detection improves with the increased dimensionality of the tests: the true negative rate and the true positive rate show marked improvement. It is deduced that the computationally more expensive higher dimensional tests need to be employed in order to achieve better anomaly detection.

Hydrogenerators are strategic assets for power utilities. Their reliability and availability can lead to significant benefits. For decades, monitoring and diagnosis of hydrogenerators have been at the core of maintenance strategies. The main cause of hydrogenerator breakdown comes from failure of its high voltage stator, which is a major component of hydrogenerators. In our previous study, it was shown that more than 85% of stator failure mechanisms indicate the presence of Partial Discharges (PD) activity. PD are minute sparks that occur within voids inside high voltage insulation or in the air around the insulating system. Each PD event does not cause immediate failure, but it will slowly erode the insulation system and will lead to breakdown in years to decades. PD signal can be detected from the lead of the hydrogenerator while it is running, thus allowing for on-line diagnosis. Hydro-Québec has been collecting more than 33.000 unlabeled PD measurement files over the last decades including two types of measurement instruments. One of the instruments used is the Partial Discharge Analyzer (PDA). PDs have several different sources and each source is characterized by a specific signature. It is therefore essential to be able to automatically recognize the nature of the DPs in order to anticipate possible degradation. Expert rules to characterize these PD signals exist but these rules cannot be used as an automatic classification tool. Indeed, for certain ambiguous cases (conflict between classes) or the presence of several signatures at the same time (multi-classes), the expert's judgement remains essential.
 A Variationel AutoEncoder (VAE) is used for dimension reduction and projection into a 2D latent space to analyze the training data and then classify it. The 2D latent space of the VAE allows the data space to be restructured and reorganized to anticipate the performance of the classifier. The problem is how to optimize this latent space and thus obtain the best distribution of the different sources of PDs to maximize the chances of good responses from the classifier? To our knowledge, there is currently no method in the literature that clearly answers this question. What is certain, however, is that the optimization of the VAE learning is directly related to the quality of the learning base, i.e. a large size and a perfectly balanced database (all the cases are present and sufficiently represented). The objective of this paper is to compare the quality of the latent space obtained from the experts' rules with a latent space obtained directly from the input signal in an « End-to-End » approach.
 The first method concerns an original unsupervised deep learning method for PD recognition. Instead of labelling the PD measurement files by the expert for a supervised learning process, we use the rules developed by the experts of Hydro-Québec to create a feature vector from recognizable PD signatures. Indeed, labelling a sufficient quantity of signatures for a supervised approach is very time-consuming and therefore cannot be implemented. This is a common problem in the industry where more and more operational data is available but cannot be labelled by experts, who are busy with other tasks. Expert knowledge is then injected into a characteristic and feature vector. In the second method, several Generative Adversarial Networks (GANs) associated with several types of PDs are thus used to generate artificial signals in order to increase the size of the learning base and especially to balance it. A new latent space is thus obtained from the learning of the VAE exclusively carried out on data generated by the GANs.
 Validation tests based on the 33,000 measurements with metrics for evaluating the performances of the various latent spaces are used.

Anomaly detection in videos is challenging due to the scarcity and variance in positive samples. Current anomaly detection methods can be categorized into reconstruction models and future frame prediction-based models. However, reconstruction models might be exceptionally adapted to abnormal events due to the learning capacity and generalization ability of deep neural networks, whereas prediction-based methods can be sensitive to noise. In this study, we propose an anomaly detection model based on the latent feature space, which combines advantages from both sides. We argue that the constraints in the latent feature space can promote reconstruction; moreover, the optical flow is also considered to introduce temporal information. We use SPyNet for accurate and efficient optical flow estimation. We extensively validate our method on the UCSD Ped1, UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets. The results demonstrated the feasibility of the proposed method and the benefit of utilizing information in the latent feature space.