Physically-informed Bayesian feature optimization for semi-supervised industrial anomaly detection

Bayesian optimization has proven effective for optimizing expensive-to-evaluate functions in Chemical Engineering. However, valuable physical insights from domain experts are often overlooked. This article introduces a collaborative Bayesian optimization approach that re-integrates human input into the data-driven decision-making process. By combining high-throughput Bayesian optimization with discrete decision theory, experts can influence the selection of experiments via a discrete choice. We propose a multi-objective approach togenerate a set of high-utility and distinct solutions, from which the expert selects the desired solution for evaluation at each iteration. Our methodology maintains the advantages of Bayesian optimization while incorporating expert knowledge and improving accountability. The approach is demonstrated across various case studies, including bioprocess optimization and reactor geometry design, demonstrating that even with an uninformed practitioner, the algorithm recovers the regret of standard Bayesian optimization. By including continuous expert opinion, the proposed method enables faster convergence and improved accountability for Bayesian optimization in engineering systems.

Semi-supervised learning acts as an effective way to leverage massive unlabeled data. In this paper, we propose a novel training strategy, termed as Semi-supervised Contrastive Learning (SsCL), which combines the well-known contrastive loss in self-supervised learning with the cross entropy loss in semi-supervised learning, and jointly optimizes the two objectives in an end-to-end way. The highlight is that different from self-training based semi-supervised learning that conducts prediction and retraining over the same model weights, SsCL interchanges the predictions over the unlabeled data between the two branches, and thus formulates a co-calibration procedure, which we find is beneficial for better prediction and avoid being trapped in local minimum. Towards this goal, the contrastive loss branch models pairwise similarities among samples, using the pseudo labels generated from the cross entropy branch, and in turn calibrates the prediction distribution of the cross entropy branch with the contrastive similarity. We show that SsCL produces more discriminative representation and is beneficial to semi-supervised learning. Notably, on ImageNet with ResNet50 as the backbone, SsCL achieves 60.2% and 72.1% top-1 accuracy with 1% and 10% labeled samples, respectively, which significantly outperforms the baseline, and is better than previous semi-supervised and self-supervised methods.

In recent years, anomaly detection techniques in time-series data have been widely used in manufacturing, cybersecurity, and other fields. Meanwhile, various anomaly detection models based on generative adversarial networks (GAN) are gradually used in time-series anomaly detection tasks. However, there are problems of unstable generator training, missed detection of anomalous data, and inconsistency between the discriminator's discriminant and the anomaly detection target in GAN networks. Aiming at the above problems, the paper proposes a DUAL-ADGAN (Dual Anomaly Detection Generative Adversarial Networks) model for the detection of anomalous data in time series. First, the Wasserstein distance satisfying the Lipschitz constraint is used as the loss function of the data reconstruction module, which improves the stability of the traditional GAN network training. Second, by adding a data prediction module to the DUAL-ADGAN model, the distinction between abnormal and normal samples is increased, and the rate of missing abnormal data in the model is reduced. Third, by introducing the Fence-GAN loss function, the discriminator is aligned with the anomaly detection target, which effectively reduces the anomaly data false detection rate of the DUAL-ADGAN model. Finally, anomaly scores derived from the DUAL-ADGAN model are compared with dynamic thresholds to detect anomalies. The experimental results show that the average F1 of the DUAL-ADGAN model is 0.881, which is better than the other nine baseline models. The conclusions demonstrate that the DUAL-ADGAN model proposed in the paper is more stable in training while effectively solving the problems of anomaly miss detection and discriminator inconsistency with the anomaly detection target in the anomaly detection task.

In this paper, we present a novel interactive image retrieval technique using semi-supervised learning. Recently, Guan and Qiu (8, 9) have shown that by constructing a Bayesian Network where the nodes represent the (continuous) class membership scores and arcs represent the dependence relations of the data points, the (semi-supervised) classification problem can be formulated as a quadratic optimization problem; and by using the labeled data as linear constraints, the optimization problem yields a large, sparse system of linear equations which can be solved very efficiently using standard methods. In this work, we show that this semi-supervised learning method can be naturally adopted as a computational tool to incorporate users feedbacks for interactive image retrieval. We present experimental results to show the effectiveness of our new interactive image retrieval method. We also show that semi- supervised learning can have advantages over supervised and unsupervised learning in image retrieval applications.

With the rapid development of rail transit, the health status of trackside equipment is more and more crucial to the operation of rail transit. The use of high-precision anomaly detection methods can quickly detect the abnormal conditions of trackside equipment, ensuring the safety of rail transit. In this paper, a new framework of trackside equipment anomaly detection based on semi-supervised and multi-domain deep learning is proposed, which effectively improves the efficiency and accuracy of anomaly detection. Considering the inconspicuous changes in trackside equipment, including screw missing and cable corrosion, it is not feasible to directly use machine learning method for anomaly detection. Therefore, we first match the collected data to get the difference image. Secondly, considering the sparsity of the difference image, the incremental principal component analysis method is used to reduce the dimension of the difference image. Finally, in view of the limited number of training samples and the high cost of manual labeling, we combine semi-supervised deep learning and multi-domain deep learning to effectively train the anomaly detection model. Experimental results show that the proposed method has higher detection accuracy than the traditional artificial anomaly detection and anomaly detection based on shallow image alignment methods.

Evolving anomaly detection for network streaming data

Automated diagnosis of infant hip dysplasia is heavily affected by the individual differences among infants and ultrasound machines. Hip sonographic images of 493 infants from various ultrasound machines were collected in the Department of Orthopedics in Yangzhou Maternal and Child Health Care Service Centre. Herein, we propose a semi-supervised learning method based on a feature pyramid network (FPN) and a contrastive learning scheme based on a Siamese architecture. A large amount of unlabeled data of ultrasound images was used via the Siamese network in the pre-training step, and then a small amount of annotated data for anatomical structures was adopted to train the model for landmark identification and standard plane recognition. The method was evaluated on our collected dataset. The method achieved a mean Dice similarity coefficient (DSC) of 0.7873 and a mean Hausdorff distance (HD) of 5.0102 in landmark identification, compared to the model without contrastive learning, which had a mean DSC of 0.7734 and a mean HD of 6.1586. The accuracy, precision, and recall of standard plane recognition were 95.4%, 91.64%, and 94.86%, respectively. The corresponding area under the curve (AUC) was 0.982. This study proposes a semi-supervised deep learning method following Graf's principle, which can better utilize a large volume of ultrasound images from various devices and infants. This method can identify the landmarks of infant hips more accurately than manual operators, thereby improving the efficiency of diagnosis of infant hip dysplasia.

A major objective of this project was to apply GE’s commercial machine learning and data analytics toolsets to large-scale, real-world, anonymized Phasor Measurement Unit (PMU) datasets in order to extract signatures, correlated and/or causal factors, and precursor patterns associated with significant power system phenomena. The project had a particular emphasis on extraction of insights relevant to asset health monitoring, real-time load modeling and cybersecurity monitoring. Additionally, the team was directed to undertake a comprehensive data quality analysis for the provided datasets and encouraged to estimate the ‘machine-learning readiness’ of the datasets by documenting any major obstacles to the application of commercial machine learning algorithms. To accomplish the aforementioned objectives, the project team’s work centered around the identification of key event signatures and application of the identified event signatures for event detection and event classification. The industry-validated, semi-supervised machine learning strategy employed for event signature identification involved several major tasks, including data-preprocessing, generation of an overabundance of features, normal data identification, normality modeling, and event signature identification through a methodical, quantitative ranking of features in order of relevance to each studied event type. Throughout the project, data quality issues and mitigation techniques were investigated. In this report, insights are provided regarding the readiness of the provided synchrophasor datasets for application of machine learning and data analytics. The methodologies employed for this technical strategy are summarized in this report. With regards to data preprocessing and feature generation, the provided Training and Test Datasets were ingested into GE’s big data environment. Subsequently, the team applied bad data cleansing and data imputation scripts, event detection scripts, and application programming interfaces (APIs) to the datasets for convenient data access. The project team completed development and validation of dozens of physics-based, statistics-based and transformation-based feature functions used for the extraction of over 60 synchrophasor features. Using a new parallel feature generation technology developed on this project, over 60 features have been rapidly generated for the full two years’ worth of Training and Test Dataset data associated with both the Eastern and Western interconnects. Even accommodating for temporal down-sampling inherent to the feature extraction procedure, this parallel feature generation activity resulted in a massive feature set with a storage requirement approximately equal to that of the raw training dataset itself. With regards to normal data identification and normality modeling, a normality model was built using the feature data extracted from the Training Dataset and iteratively refined subsequent to incremental adjustments and expansions of the Training Dataset feature data. With respect to event characterization and signature identification, an event signature identification pipeline was developed and used in conjunction with the normality model to identify over 15 event signatures for key event categories within the Training Dataset. The identified event signatures were used to characterize hundreds of key events in terms of relative severity, duration, and location of the event. An investigation was undertaken to identify correlated and causal factors involved in transformer events. A separate investigation into temporal trends in ring-down analysis results was undertaken to determine possible associations between system dynamics and various other factors such as loading, season or year. To validate the identified event signatures, additional work was undertaken to develop signature-based anomaly detection and classification tools suitable for convenient application to the synchrophasor datasets. The anomaly detection and classification tools, suitable for online application, were then applied to the entirety of the Eastern Interconnect Training and Test Datasets. Performance of the event detection and classification tools was evaluated upon receipt of the Test Dataset event logs (i.e., the labels for events contained in the Test Dataset), and promising results were obtained despite several challenges (documented herein) associated with application of supervised or semi-supervised machine learning methods to large-scale, anonymized datasets. Finally, the detection and classification tools were used to detect, classify, and characterize thousands of new events not included in the original event logs provided by the DOE within both the Training and Test Datasets.

Semi-supervised learning has made significant progress in medical image segmentation. However, existing methods primarily utilize information from a single dimensionality, resulting in sub-optimal performance on challenging magnetic resonance imaging (MRI) data with multiple segmentation objects and anisotropic resolution. To address this issue, we present a Hybrid Dual Mean-Teacher (HD-Teacher) model with hybrid, semi-supervised, and multi-task learning to achieve effective semi-supervised segmentation. HD-Teacher employs a 2D and a 3D mean-teacher network to produce segmentation labels and signed distance fields from the hybrid information captured in both dimensionalities. This hybrid mechanism allows HD-Teacher to utilize features from 2D, 3D, or both dimensions as needed. Outputs from 2D and 3D teacher models are dynamically combined based on confidence scores, forming a single hybrid prediction with estimated uncertainty. We propose a hybrid regularization module to encourage both student models to produce results close to the uncertainty-weighted hybrid prediction to further improve their feature extraction capability. Extensive experiments of binary and multi-class segmentation conducted on three MRI datasets demonstrated that the proposed framework could (1) significantly outperform state-of-the-art semi-supervised methods (2) surpass a fully-supervised VNet trained on substantially more annotated data, and (3) perform on par with human raters on muscle and bone segmentation task. Code will be available at https://github.com/ThisGame42/Hybrid-Teacher.

Data scarcity and quality pose significant challenges to supervised learning. The process of generating informative annotations can be time-consuming and often requires high domain expertise. Active and semi-supervised learning methods can reduce labeling effort by either automatically expanding the training set or by selecting the most informative examples to request domain expert annotation. As most selection methods are heuristic, the performance varies widely across datasets and tasks. Bootstrapping approaches such as self-training can result in negative effects due to the addition of incorrectly pseudo-labeled instances. In this work, we take a holistic approach to label acquisition and consider the expansion of clean and pseudo-labeled subsets jointly. To address the challenge of producing high-quality pseudo-labels, we introduce a collaborative teacher-student framework, where the teacher, termed AdaReNet, learns a data-driven curriculum. Experimental results on several natural language processing (NLP) tasks demonstrate that the proposed framework outperforms baselines.

The integrity and efficiency of modern energy grids are increasingly reliant on accurate anomaly detection within energy consumption data. However, class imbalance poses significant challenges, where normal consumption patterns vastly outnumber critical anomalies, leading to biased detection models. This paper presents a novel heterogeneous deep ensemble model specifically designed to handle class imbalance in energy consumption anomaly detection. The architecture strategically integrates Bidirectional Long ShortTerm Memory (BiLSTM) networks for capturing temporal dependencies and Convolutional Neural Networks (CNNs) for feature extraction. Cost-sensitive learning was incorporated to address class imbalance, with rigorous hyperparameter tuning using Bayesian optimization. The model was evaluated using the State Grid Corporation of China (SGCC) dataset containing 42,372 customers' electricity consumption data.The deep ensemble model achieved impressive performance metrics: accuracy of 97.5%, precision of 97%, recall of 99%, F1-score of 98%, and AUC-ROC score of 99%. Statistical analysis confirmed significant improvements over baseline methods (BiLSTM and CNN) and existing ensemble models, with p-values consistently below 0.05.The heterogeneous ensemble architecture demonstrates superior performance compared to individual models and existing approaches. Cost-sensitive learning effectively addresses class imbalance while maintaining high accuracy. The findings establish new performance benchmarks for anomaly detection in energy systems with significant implications for energy efficiency, grid stability, and infrastructure security.

Modal frequencies are widely used as damage-sensitive features in vibration-based unsupervised anomaly detection for full-scale civil structures. However, environmental variations introduce nonlinear and non-Gaussian behaviors in modal frequencies, characterized by multicentricity and substantial variance differences. These behaviors inevitably mask damage-induced frequency shifts, leading to increased false-detection rates and degraded real-time performance. To address this, this paper presents a novel unsupervised framework for early-stage anomaly detection that integrates exponential slow feature analysis (SFA) with just-in-time learning-aided standard K -nearest neighbor metrics. First, nonlinear modal features with minimum temporal variation are extracted from modal frequencies via exponential SFA for dimensionality reduction. Afterward, under the standard measurement criteria, the Euclidean squared distances between each sample in the entire training set and its nearest neighbors are computed. The standard distance for each neighbor is then modeled, with the number of neighbors automatically selected through Bayesian optimization and cross-validation. The primary purpose of this step is to ensure that the nearest neighbor distances between samples under different environmental conditions and damage conditions including abrupt anomalies or gradual degradations, are measured on a consistent scale, thereby minimizing environmental variability. After that, a robust detection indicator is derived from the mean of squared nearest neighbor distances and combined with kernel density estimation to determine an online detection threshold for outliers. The results indicate the proposed method effectively mitigates strong environmental variability without prior environmental measurements and accurately identifies anomalies in bridge modal frequencies with negligible false-positive and false-negative errors.

A new graph-based semi-supervised method for surface defect classification

The development and distribution of interlayers in sandstones can lead to increased heterogeneity in the formation within the reservoir, which further affects the movement of fluids in the formation. Therefore, it is essential to accurately evaluate the interlayers in sandstones, as the precise evaluation of interlayers in sandstones is of great importance for identifying the distribution of underground fluid systems. The logging data of interlayers are inadequate for traditional Machine Learning training due to their low measurement proportion compared to the conventional layers. In logging data, the amount of data in interlayers is significantly smaller than that of conventional reservoirs. Traditional machine learning models are mostly based on samples with balanced distribution. By contrast, semi-supervised learning requires a small number of labeled samples for learning, and then combines a large number of unlabeled samples for modeling. To verify the feasibility of semi-supervised learning in the identification of interlayers, the Donghe sandstone section of the H oilfield was taken as an example. First, the core analysis results were used to label the logging data; then, to uncover more response information that can characterize the interlayers on the logging curve, multiple features were extracted to construct cross features. Finally, an improved model based on autoencoders—probabilistic autoencoder (PAE)—is proposed to address the issue of interlayer recognition for imbalanced samples. The PAE model can calculate the probability of belonging to a different class for unlabeled samples, and classify new samples according to the maximum probability. Experimental results show that, compared with traditional machine learning methods and ensemble learning methods, PAE achieves higher recognition accuracy and better generalization performance by updating the algorithm, and can be used as a simple and fast method for interlayer recognition. The results of the algorithm demonstrate that the semi-supervised method is of great significance for exploring and developing complex heterogeneous oil reservoirs. The research results are of great importance for exploring and developing complex heterogeneous oil reservoirs.

In machine learning, whether one can build a more accurate classifier by using unlabeled data (semi-supervised learning) is an important issue. Although a number of semi-supervised methods have been proposed, their effectiveness on NLP tasks is not always clear. This paper presents a novel semi-supervised method that employs a learning paradigm which we call structural learning. The idea is to find what good classifiers are like by learning from thousands of automatically generated auxiliary classification problems on unlabeled data. By doing so, the common predictive structure shared by the multiple classification problems can be discovered, which can then be used to improve performance on the target problem. The method produces performance higher than the previous best results on CoNLL'00 syntactic chunking and CoNLL'03 named entity chunking (English and German).