Open Set Recognition for Random Forest

Open set recognition requires models to recognize samples of known classes learned in the training set while reject unknowns not learned. Compared with the structural risk minimization theory for closed-set problems, structural risk in open set tasks remains rarely explored. In this paper, we point out that balancing between structural risk and open space risk is crucial for open set recognition, and re-formalize it as open set structural risk. This brings a new view towards the general relationship between closed set recognition and open set recognition against the common intuition, which argues that a good closed set classifier always benefits for open set recognition. Specifically, we theoretically and experimentally show that recent mix-based data augmentation methods are aggressive closed set regularization methods, which reduce structural risk at cost of sacrificing open space risk. Besides, we show that existing negative data augmentation designed for open space risk reduction also ignore the trade-off problem between structural risk and open space risk, which limits their performance. We propose an efficient negative data augmentation strategy named self-mix and a corresponding method named OpenMix. OpenMix generates high-quality negative samples by mixing samples themselves, which can take care of both risks simultaneously. When combining OpenMix with conservative closed set regularization methods to form OpenMix+, models can achieve lower open set structural risk. Extensive experiments validate the superiority of OpenMix and OpenMix+ in terms of both effectiveness and universality.

Image classification methods are usually trained to perform predictions taking into account a predefined group of known classes. Real-world problems, however, may not allow for a full knowledge of the input and label spaces, making failures in recognition a hazard to deep visual learning. Open set recognition methods are characterized by the ability to correctly identify inputs of known and unknown classes. In this context, we propose GeMOS: simple and plug-and-play open set recognition modules that can be attached to pretrained Deep Neural Networks for visual recognition. The GeMOS framework pairs pre-trained Convolutional Neural Networks with generative models for open set recognition to extract open set scores for each sample, allowing for failure recognition in object recognition tasks. We conduct a thorough evaluation of the proposed method in comparison with state-of-the-art open set algorithms, finding that GeMOS either outperforms or is statistically indistinguishable from more complex and costly models.

Spatial Location Constraint Prototype Loss for Open Set Recognition

Spatial Location Constraint Prototype Loss for Open Set Recognition

Interest in emerging diseases is increasing due to recent global outbreaks like COVID-19. Unlike general image classification tasks, medical imaging is a multi-label classification that can have multiple diseases simultaneously. Then, suppose the results of all classes do not exceed the thresholds. In that case, it is classified as normal rather than unknown. This is why the existing open-set recognition (OSR) methods cannot be applied to open-set medical diagnosis. In this paper, we propose a novel open-set medical diagnosis method to solve the fundamental problem of OSR in multi-label classification. To solve this problem, we employ Copycat and entropy-based thresholds. To our knowledge, open-set multi-label medical diagnosis has not yet been addressed in research. We conduct experiments to confirm that our proposed method performs well in both multi-label classification and recognizing normal and unknown.

It is important to predict critical alarms in the manufacturing process that can reduce the utilization rate of the production facility. In recent years, deep neural networks have been widely used for prediction of alarm types in the manufacturing processes. However, the existing deep neural network classifiers follow a closed-set assumption that all predictable categories should be learned in the training stage. For this reason, when an unknown alarm type comes into the classifier, it should be classified as one of the predefined alarm types. In the actual manufacturing processes, it is extremely difficult to collect all possible types of alarm data. Therefore, a model with open set recognition is required to identify unknown alarm types. In addition, because the alarm type data collected from production facilities occurs simultaneously, a multi-label classification model is necessary. In this study, we propose a multi-label open set recognition model combined with background data that can improve the ability to identify unknown alarm types. We demonstrated the usefulness and applicability of the proposed method by comparing it with existing open set classification methods using real process data obtained from an automobile industry.

Evaluations of large-scale recognition methods typically focus on overall performance. While this approach is common, it often fails to provide insights into performance across individual classes, which can lead to fairness issues and misrepresentation. Addressing these gaps is crucial for accurately assessing how well methods handle novel or unseen classes and ensuring a fair evaluation. To address fairness in Open-Set Recognition (OSR), we demonstrate that per-class performance can vary dramatically. We introduce Gaussian Hypothesis Open Set Technique (GHOST), a novel hyperparameter-free algorithm that models deep features using class-wise multivariate Gaussian distributions with diagonal covariance matrices. We apply Z-score normalization to logits to mitigate the impact of feature magnitudes that deviate from the model’s expectations, thereby reducing the likelihood of the network assigning a high score to an unknown sample. We evaluate GHOST across multiple ImageNet-1K pre-trained deep networks and test it with four different unknown datasets. Using standard metrics such as AUOSCR, AUROC and FPR95, we achieve statistically significant improvements, advancing the state-of-the-art in large-scale OSR. Source code is provided online.

Current deep learning methods are regarded as favorable if they empirically perform well on dedicated test sets. This mentality is seamlessly reflected in the resurfacing area of continual learning, where consecutively arriving data is investigated. The core challenge is framed as protecting previously acquired representations from being catastrophically forgotten. However, comparison of individual methods is nevertheless performed in isolation from the real world by monitoring accumulated benchmark test set performance. The closed world assumption remains predominant, i.e. models are evaluated on data that is guaranteed to originate from the same distribution as used for training. This poses a massive challenge as neural networks are well known to provide overconfident false predictions on unknown and corrupted instances. In this work we critically survey the literature and argue that notable lessons from open set recognition, identifying unknown examples outside of the observed set, and the adjacent field of active learning, querying data to maximize the expected performance gain, are frequently overlooked in the deep learning era. Hence, we propose a consolidated view to bridge continual learning, active learning and open set recognition in deep neural networks. Finally, the established synergies are supported empirically, showing joint improvement in alleviating catastrophic forgetting, querying data, selecting task orders, while exhibiting robust open world application.

We propose a generalized Sparse Representation-based Classification (SRC) algorithm for open set recognition where not all classes presented during testing are known during training. The SRC algorithm uses class reconstruction errors for classification. As most of the discriminative information for open set recognition is hidden in the tail part of the matched and sum of non-matched reconstruction error distributions, we model the tail of those two error distributions using the statistical Extreme Value Theory (EVT). Then we simplify the open set recognition problem into a set of hypothesis testing problems. The confidence scores corresponding to the tail distributions of a novel test sample are then fused to determine its identity. The effectiveness of the proposed method is demonstrated using four publicly available image and object classification datasets and it is shown that this method can perform significantly better than many competitive open set recognition algorithms.

We propose an unsupervised open set recognition approach that uses an Adversarial Autoencoder (AAE) for clustering and combines it with ii-loss to learn a representation that facilitates open set recognition. We evaluate our approach on two malware datasets. To show the applicability of the proposed method outside the malware domain we also evaluate it with an image dataset. Our proposed approach gives a statistically significant improvement over other evaluated approaches.

An open set recognition methodology utilising discrepancy analysis for gear diagnostics under varying operating conditions

A significant challenge in the domain of anti‐drone warfare is the identification of enemies or own aircraft through the analysis of data broadcast by drones (e.g. ADS‐B). This issue can be conceptualized as an open set recognition (OSR) problem. This paper proposes a DV‐OSR‐QSED framework for the purpose of data visualization‐based OSR (DV‐OSR). Phase‐based 2D high‐importance features are extracted, the DV‐OSR framework is designed and mapped to 2D, and the 5th and 95th quantile selection‐Euclidean distance (QSED) strategy is proposed. Experiments show that by using the proposed framework, the correct classification rate for known and unknown samples is 96.04% and 95.79%, the recall rate and F1 value are 89.00% and 92.27%, and the AUC is 0.9630.

In Synthetic Aperture Radar (SAR) image classification tasks, the performance depends on both the classifier and the dataset itself. However, in comparison with plenty of SAR classification methods, there is little work aimed at analyzing the distinguishability of the dataset. In the classification dataset, some classes are semantically different but their distinguishability is low, the classes are hard to be classified especially in some more practical cases that there are unknown classes without supervision exist. Referring to open set recognition (OSR), in this paper, we proposed the SAR Distinguishability Analysor (SAR-DA) to evaluate the distinguishability of the OpenSARUrban dataset. By modeling each class as a multivariate Gaussian distribution in latent space, SAR-DA can not only classify the classes having been seen in training phase, but also can recognize unknown samples if a test sample is out of each known distribution. Each class in OpenSARUr-ban is set unknown in turn, then we apply the SAR-DA on the split dataset in OSR and supervised setting. The distinguishability can be reflected by the unknown recognition recall rate. The experimental results show that the unknown recognition recall rate in OSR setting significantly decreased compared with those in supervised setting, indicating that even though the classes in OpenSARUrban are semantically different from each other, the latent distributions of some classes are quite similar and hard to be classified, thus these classes are of low distinguishability.

Most of the progresses achieved in radar high range resolution profile (HRRP) recognition rely on the closed set condition, where the test sample is from a known class. In realistic scenario, however, the test sample may be drawn from unknown classes, which is regarded as an open set recognition task. In such cases, conventional recognition algorithms will inevitably make a wrong prediction. In this paper, the open set problem is addressed by incorporating the extreme value theory (EVT) into the random forest (RF) classifier. The outputs of RF are analyzed to determine whether the test sample should be rejected as an unknown class. At the training phase, a Weibull-based extreme-value meta-recognition is introduced to describe the statistical characteristics of the known classes. At the testing phase, a probability estimation method is introduced to compute the probabilities of the test sample belonging to known and unknown classes based on trained Weibull distributions. The test sample is assigned to the class of highest probability. Experimental results demonstrate that the proposed method outperforms the state-of-art NN, 1-vs-set machine and W-SVM in rejecting unknown classes.

There are available metrics for predicting fault prone classes, which may help software organizations for planning and performing testing activities. This may be possible due to proper allocation of resources on fault prone parts of the design and code of the software. Hence, importance and usefulness of such metrics is understandable, but empirical validation of these metrics is always a great challenge. Random Forest (RF) algorithm has been successfully applied for solving regression and classification problems in many applications. In this work, the authors predict faulty classes/modules using object oriented metrics and static code metrics. This chapter evaluates the capability of RF algorithm and compares its performance with nine statistical and machine learning methods in predicting fault prone software classes. The authors applied RF on six case studies based on open source, commercial software and NASA data sets. The results indicate that the prediction performance of RF is generally better than statistical and machine learning models. Further, the classification of faulty classes/modules using the RF method is better than the other methods in most of the data sets.