Comparing Complexities of Decision Boundaries for Robust Training: A Universal Approach

Compressed signal processing (CSP) is a branch of compressive sensing (CS), which gives a direction to solve a class of signal processing problems directly from the compressive measurements of a signal. CSP utilizes the information preserved in the compressive measurements of a signal to solve certain inference problems like: classification, detection, and estimation, without reconstructing the original signal. It further simplifies the signal processing compared to conventional CS by omitting their complex reconstruction stage. This, in turn, reduces the implementation complexity of signal processing systems. This paper investigates the performance of CSP for classification application. After extracting the features from compressive measurements, these features or the data instances are used for classification purpose. Through experimental analysis, it has been found that as the CS undersampling factor is increased, the overlapping among the data instances predominates. This results in a complex decision boundary, which in turn degrades the classification accuracy at higher undersampling factors. To overcome the above issue, this paper proposes the use of a machine learning method known as overlap aware learning along with CSP. This generates a smoother decision boundary and hence improves the classification accuracy at higher undersampling factors. The simulation results show the trend of improved classification accuracy using the proposed method. An analysis of the proposed method has been done on different datasets and based on run-time complexity and complexity vs gain analysis to verify the effectiveness of proposed method.

It has been widely accepted that the classification accuracy can be improved by combining outputs of multiple classifiers. However, how to combine multiple classifiers with various (potentially conflicting) decisions is still an open problem. A rich collection of classifier combination procedures -- many of which are heuristic in nature -- have been developed for this goal. In this brief, we describe a dynamic approach to combine classifiers that have expertise in different regions of the input space. To this end, we use local classifier accuracy estimates to weight classifier outputs. Specifically, we estimate local recognition accuracies of classifiers near a query sample by utilizing its nearest neighbors, and then use these estimates to find the best weights of classifiers to label the query. The problem is formulated as a convex quadratic optimization problem, which returns optimal nonnegative classifier weights with respect to the chosen objective function, and the weights ensure that locally most accurate classifiers are weighted more heavily for labeling the query sample. Experimental results on several data sets indicate that the proposed weighting scheme outperforms other popular classifier combination schemes, particularly on problems with complex decision boundaries. Hence, the results indicate that local classification-accuracy-based combination techniques are well suited for decision making when the classifiers are trained by focusing on different regions of the input space.

BoAu: Malicious traffic detection with noise labels based on boundary augmentation

Contrastive learning of graphs under label noise

The neural tree classifier: Complex decision boundaries with single-layer nets

We investigate star-galaxy classification for astronomical surveys in the context of four methods enabling the interpretation of black-box machine learning systems. The first is outputting and exploring the decision boundaries as given by decision tree based methods, which enables the visualization of the classification categories. Secondly, we investigate how the Mutual Information based Transductive Feature Selection (MINT) algorithm can be used to perform feature pre-selection. If one would like to provide only a small number of input features to a machine learning classification algorithm, feature pre-selection provides a method to determine which of the many possible input properties should be selected. Third is the use of the tree-interpreter package to enable popular decision tree based ensemble methods to be opened, visualized, and understood. This is done by additional analysis of the tree based model, determining not only which features are important to the model, but how important a feature is for a particular classification given its value. Lastly, we use decision boundaries from the model to revise an already existing method of classification, essentially asking the tree based method where decision boundaries are best placed and defining a new classification method. We showcase these techniques by applying them to the problem of star-galaxy separation using data from the Sloan Digital Sky Survey (hereafter SDSS). We use the output of MINT and the ensemble methods to demonstrate how more complex decision boundaries improve star-galaxy classification accuracy over the standard SDSS frames approach (reducing misclassifications by up to $\approx33\%$). We then show how tree-interpreter can be used to explore how relevant each photometric feature is when making a classification on an object by object basis.

In machine learning applications, the reliability of predictions is significant for assisted decision and risk control. As an effective framework to quantify the prediction reliability, conformal prediction (CP) was commonly developed with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> -nearest neighbor, known as CPKNN. However, the conventional CPKNN suffers from high variance and bias, and long computational time as the feature dimensionality increases. To address these limitations, a new CP framework-CP with shrunken centroids (CPSC) is proposed. It regularizes the class centroids to attenuate the irrelevant features and shrink the sample space for predictions and reliability quantification. We have also compared the CPSC with CP based on SVM, LightGBM, RF on the herbal medicine dataset, with the electronic nose as a case and assessed them in two tasks: 1) off-line prediction: the training set was fixed, and the accuracy on the testing set was evaluated and 2) online prediction with data augmentation: CPKNN and CPSC filtered unlabeled data to augment the training data based on the prediction reliability, and the final accuracy of the testing set was compared. The result shows that CPSC significantly outperformed CPKNN in both two tasks: 1) CPSC reached a significantly higher accuracy with lower computation cost, and with the same credibility output, CPSC generally achieved higher accuracy and 2) the data augmentation process with CPSC robustly manifested a statistically significant improvement in prediction accuracy with different reliability thresholds, and the augmented data were more balanced in classes. This novel CPSC provides higher prediction accuracy and better reliability quantification, which can assist in reliable decision support.

Deep learning models rely significantly on large volumes of labeled data, the gathering of which is both costly and time-intensive. To tackle this challenge, active learning, which seeks to attain comparable results by selecting and labeling a small portion of data, has attracted more interests. Traditionally, most studies assessing sample importance have focused solely on its individual features. Nonetheless, the relationship between samples is also crucial. For instance, a sample is more likely located around the decision boundary if its several closest neighbors are from different classes. To incorporate this aspect, we propose to explore the diversity of the predicted labels of nearby samples. Specifically, we calculate the variance of the class distribution among the K nearest neighbors, treating it as an indicator of a sample’s significance. Those samples with higher scores are selected for querying the oracle to obtain annotations. Moreover, we employ a nearest neighbor classifier, eventually enhancing generalization performance on unlabeled samples. To evaluate this approach, we conduct experiments using the CIFAR-10 and CIFAR-100 datasets, well-known for image classification based on active learning. The results demonstrate the effectiveness of the proposed approach.

Collecting large-scale data with clean labels for supervised training of neural networks is practically challenging. Although noisy labels are usually cheap to acquire, existing methods suffer a lot from label noise. This paper targets at the challenge of robust training at high label noise regimes. The key insight to achieve this goal is to wisely leverage a small trusted set to estimate exemplar weights and pseudo labels for noisy data in order to reuse them for supervised training. We present a holistic framework to train deep neural networks in a way that is highly invulnerable to label noise. Our method sets the new state of the art on various types of label noise and achieves excellent performance on large-scale datasets with real-world label noise. For instance, on CIFAR100 with a $40\%$ uniform noise ratio and only 10 trusted labeled data per class, our method achieves $80.2{\pm}0.3\%$ classification accuracy, where the error rate is only $1.4\%$ higher than a neural network trained without label noise. Moreover, increasing the noise ratio to $80\%$, our method still maintains a high accuracy of $75.5{\pm}0.2\%$, compared to the previous best accuracy $48.2\%$. Source code available: https://github.com/google-research/google-research/tree/master/ieg

Artificial neural networks (ANNs) are widely used in applications with complex decision boundaries. A large number of activation functions have been proposed in the literature to achieve better representations of the observed data. However, only a few works employ Tsallis statistics, which has successfully been applied to various other fields. This paper presents a random neural network (RNN) with q -Gaussian activation functions [ q -generalized RNN (QRNN)] based on Tsallis statistics. The proposed method employs an additional parameter q (called the entropic index) which reflects the degree of nonextensivity. This approach has the flexibility to model complex decision boundaries of different shapes by varying the entropic index. We conduct numerical experiments to analyze the efficiency of QRNN compared with RNNs and several other classical methods. Statistical tests (Wilcoxon and Friedman) are used to validate our results and show that the QRNN performs significantly better than RNNs with different activation functions. In addition, we find that QRNN outperforms many of the compared classical methods, with the exception of support vector machines, in which case it still exhibits a substantial advantage in terms of implementation simplicity and speed.

Algorithms based on Nested Generalized Exemplar (NGE) theory (Salzberg, 1991) classify new data points by computing their distance to the nearest “generalized exemplar” (i.e., either a point or an axis-parallel rectangle). They combine the distance-based character of nearest neighbor (NN) classifiers with the axis-parallel rectangle representation employed in many rule-learning systems. An implementation of NGE was compared to the k-nearest neighbor (kNN) algorithm in 11 domains and found to be significantly inferior to kNN in 9 of them. Several modifications of NGE were studied to understand the cause of its poor performance. These show that its performance can be substantially improved by preventing NGE from creating overlapping rectangles, while still allowing complete nesting of rectangles. Performance can be further improved by modifying the distance metric to allow weights on each of the features (Salzberg, 1991). Best results were obtained in this study when the weights were computed using mutual information between the features and the output class. The best version of NGE developed is a batch algorithm (BNGE FWMI) that has no user-tunable parameters. BNGE FWMI's performance is comparable to the first-nearest neighbor algorithm (also incorporating feature weights). However, the k-nearest neighbor algorithm is still significantly superior to BNGE FWMI in 7 of the 11 domains, and inferior to it in only 2. We conclude that, even with our improvements, the NGE approach is very sensitive to the shape of the decision boundaries in classification problems. In domains where the decision boundaries are axis-parallel, the NGE approach can produce excellent generalization with interpretable hypotheses. In all domains tested, NGE algorithms require much less memory to store generalized exemplars than is required by NN algorithms.

Few-shot learning remains a challenging problem, with unsatisfactory 1-shot accuracies for most real-world data. Here, we present a new perspective for data distributions in the feature space of a deep network and show how to exploit this perspective for few-shot learning. First, we observe that nearest neighbors in the feature space are with high probability members of the same class while generally two random points from one class are not much closer to each other than two points between classes. This observation suggests that classes in feature space form sparse, loosely connected graphs instead of dense clusters. To exploit this property, we propose using label propagation to the nearest unlabeled data and then using a kernel PCA reconstruction error as decision boundary in feature-space for the data distribution of each class. Using this method, which we call "K-Prop," we demonstrate largely improved few-shot learning performances (e.g., 83% accuracy for 1-shot 5-way classification on the RESISC45 satellite-images dataset) for datasets for which a backbone network can be trained to produce high within-class nearest-neighbor probabilities. We demonstrate this relationship using six different datasets.

Noise-robust re-identification with triple-consistency perception

The class imbalance problem is the main problem in classification. This issue arises because real-world datasets frequently exhibit an imbalance as a result of a class with more instances than other classes. In handling class imbalance, a Hybrid Approach that blends data-level and algorithm-level approaches produce good results. However, apart from the class imbalance, which reduces classification accuracy, the complexity of the data also has an effect. The complexity of this data causes a minority noise sample which lies between the minority and the majority. In order to determine how close minority samples are to their homogeneous and heterogeneous nearest neighbors, it is necessary to calculate the relative density. The greater the proximity to the homogeneous nearest neighbors, the greater the relative density, which causes the minority samples to be in a safe state but otherwise be categorized as noisy samples. This research will combine the application of the Hybrid Approach with A self-adaptive Robust SMOTE (RSMOTE), which is an adaptive method from SMOTE that applies the concept of relative density in the over-sampling process on minority samples. The research contribution is to implement the Hybrid Approach-RSMOTE in handling class imbalance with noise by using relative density in over-sampling and also to improve classification performance. The results showed that the Hybrid Approach-RSMOTE and Hybrid Approach-SMOTE had given good results in handling class imbalance. However, the Hybrid Approach-RSMOTE gave better results in the Precision, Recall, F1-Measure, and G-Mean and showed significant differences. Based on the results of the study, it can be stated that the performance of the Hybrid Approach in handling class imbalance is influenced by the selection of the over-sampling method. The results show that RSMOTE can be considered an over-sampling method in the Hybrid Approach.

Many recent works on knowledge distillation have provided ways to transfer the knowledge of a trained network for improving the learning process of a new one, but finding a good technique for knowledge distillation is still an open problem. In this paper, we provide a new perspective based on a decision boundary, which is one of the most important component of a classifier. The generalization performance of a classifier is closely related to the adequacy of its decision boundary, so a good classifier bears a good decision boundary. Therefore, transferring information closely related to the decision boundary can be a good attempt for knowledge distillation. To realize this goal, we utilize an adversarial attack to discover samples supporting a decision boundary. Based on this idea, to transfer more accurate information about the decision boundary, the proposed algorithm trains a student classifier based on the adversarial samples supporting the decision boundary. Experiments show that the proposed method indeed improves knowledge distillation and achieves the state-of-the-arts performance.