Diversity and separable metrics in over-sampling technique for imbalanced data classification

As machine learning techniques mature and are used to tackle complex scientific problems, challenges arise such as the imbalanced class distribution problem, where one of the target class labels is under-represented in comparison with other classes. Existing oversampling approaches for addressing this problem typically do not consider the probability distribution of the minority class while synthetically generating new samples. As a result, the minority class is not well represented which leads to high misclassification error. We introduce two Gibbs sampling-based oversampling approaches, namely RACOG and wRACOG, to synthetically generating and strategically selecting new minority class samples. The Gibbs sampler uses the joint probability distribution of attributes of the data to generate new minority class samples in the form of Markov chain. While RACOG selects samples from the Markov chain based on a predefined lag, wRACOG selects those samples that have the highest probability of being misclassified by the existing learning model. We validate our approach using five UCI datasets that were carefully modified to exhibit class imbalance and one new application domain dataset with inherent extreme class imbalance. In addition, we compare the classification performance of the proposed methods with three other existing resampling techniques.

An overlapping minimization-based over-sampling algorithm for binary imbalanced classification

The relation between the structure of a market and the diversity of its product offering has been extensively explored by theorists. We develop 2 measures of diversity and explore the content of local news for 60 stations and 20 designated market areas (DMAs) in the United States. Using a relative station-level diversity metric, ordinary least squares (OLS) estimates imply that relative diversity of local news content decreases as market concentration increases. This result is not, however, robust to an instrumental variables specification. Using a total market diversity metric, the Herfindahl-Hirschman Index (Hirshman, 1964) is significant in OLS and robust to instrumental variable estimation. Because the total market diversity metric is arguablysuperior to the incremental metric as a measure of overall diversity, this result is useful-it suggests that the total diversity of local news content within a DMA is sensitive to the level of concentration.

Imbalanced image datasets consist of image datasets where there is a significant disparity in the number of samples across different classes. With imbalanced image datasets, learning algorithms often tend to be biased toward the majority class samples. This leads to poor classification of minority class samples as their training is not properly conducted. It becomes more complicated when the number of samples in the minority class is very low. In this paper, a novel hierarchical algorithm is proposed for generating new data using Capsule Generative Adversarial Networks (CapsGAN) to address the class imbalance problem in imbalanced image datasets. Unlike common GAN models, the proposed method incorporates an auxiliary CapsNet to identify high‐value images in both minority and majority classes. This identification is based on the ability to detect complex relationships between low‐level and high‐level features present in capsule networks. Furthermore, the proposed CapsGAN model is conditioned to generate minority class samples based on feature vectors of last capsule layer to achieve a more balanced image dataset. For evaluating the performance of the proposed model, an image dataset called CICS was collected and introduced. Extensive experiments were also conducted using various online image datasets from different domains, with varying numbers of classes and data sizes. The experimental results demonstrated that the proposed model can generate high‐quality samples in cases where the image dataset or the number of minority class samples is relatively small. Furthermore, the proposed model has maintained an accuracy of over 80% in an imbalanced ratio of 1:60.

Skewed class proportions in real-world datasets present a challenge for machine learning algorithms, as they have a tendency to correctly categorize the majority class while incorrectly classifying the minority class. Such classification disparities hold significant implications, particularly in predictive scenarios involving minority groups, where misclassifying minority instances could lead to adverse outcomes. To tackle this, class imbalance learning has gained attention, with the Synthetic Minority Oversampling Technique (SMOTE) being a notable approach that addresses class imbalance by generating synthetic instances for the minority class based on their feature space neighbors. Despite its effectiveness and simplicity, SMOTE is known to suffer from a noise propagation issue where noisy and uninformative samples are introduced. While various SMOTE variants, including hybrids with undersampling, have been developed to tackle this problem, identifying noisy samples in complex real-world datasets remains a challenge. To address this, our study introduces a new SMOTE-based hybrid approach called SMOTE-centroid displacement-based k-NN (SMOTE-CDNN). SMOTE-CDNN employs centroid displacement for class prediction, which is more robust against noisy data. After SMOTE is applied, noise instances are detected and removed for clearer decision boundaries if their labels predicted by our centroid displacement-based k-NN algorithm are different from the real ones. While our experiments on 24 imbalance datasets demonstrate the resilience and efficiency of our proposed algorithm, which outperforms state-of-art resampling algorithms with various classification models, we acknowledge the need for further investigation into specific dataset characteristics and classification scenarios to determine the generalizability of our approach.

A Review of Concentration, Diversity or Entropy Metrics in Economics, Finance, Ecology and Communication Science

When classifiers face imbalanced class distributions, they often misclassify minority class samples, consequently diminishing the predictive performance of machine learning models. Existing oversampling techniques predominantly rely on the selection of neighboring data via interpolation, with less emphasis on uncovering the intrinsic patterns and relationships within the data. In this research, we present the usefulness of an algorithm named RuLer to deal with the problem of classification with imbalanced data. RuLer is a learning algorithm initially designed to recognize new sound patterns within the context of the performative artistic practice known as live coding. This paper demonstrates that this algorithm, once adapted (Ad-RuLer), has great potential to address the problem of oversampling imbalanced data. An extensive comparison with other mainstream oversampling algorithms (SMOTE, ADASYN, Tomek-links, Borderline-SMOTE, and KmeansSMOTE), using different classifiers (logistic regression, random forest, and XGBoost) is performed on several real-world datasets with different degrees of data imbalance. The experiment results indicate that Ad-RuLer serves as an effective oversampling technique with extensive applicability.

Preterm birth is a leading cause of global neonate mortality. Hospitalization costs associated with preterm deliveries present a huge economic burden. Existing physical/biochemical markers for predicting preterm birth risk are mostly suited for application at mid/late pregnancy stages, thereby leaving very short time (between diagnosis and delivery) for adopting appropriate intervention strategies. Recent studies indicating correlations between pre/full-term delivery and the composition of vaginal microbiota in pregnant women have opened new diagnostic possibilities. In this study, we performed a thorough meta-analysis of vaginal microbiome datasets to evaluate the utility of popular diversity and inequality measures for predicting, at an early stage, the risk of preterm delivery. Results indicate significant differences (in diversity measures) between ‘first-trimester’ vaginal microbiomes obtained from women with term and preterm outcomes, indicating the potential diagnostic utility of these measures. In this context, we introduce a novel diversity metric that has significantly better diagnostic ability as compared to established diversity measures. The metric enables ‘early’ and highly accurate prediction of preterm delivery outcomes, and can potentially be deployed in clinical settings for preterm birth risk-assessment. Our findings have potentially far reaching implications in the fight against neonatal deaths due to preterm birth.

Bond rating is regarded as an important event for measuring financial risk of companies and for determining the investment returns of investors. As a result, it has been a popular research topic for researchers to predict companies' credit ratings by applying statistical and machine learning techniques. The statistical techniques, including multiple regression, multiple discriminant analysis (MDA), logistic models (LOGIT), and probit analysis, have been traditionally used in bond rating. However, one major drawback is that it should be based on strict assumptions. Such strict assumptions include linearity, normality, independence among predictor variables and pre-existing functional forms relating the criterion variablesand the predictor variables. Those strict assumptions of traditional statistics have limited their application to the real world. Machine learning techniques also used in bond rating prediction models include decision trees (DT), neural networks (NN), and Support Vector Machine (SVM). Especially, SVM is recognized as a new and promising classification and regression analysis method. SVM learns a separating hyperplane that can maximize the margin between two categories. SVM is simple enough to be analyzed mathematical, and leads to high performance in practical applications. SVM implements the structuralrisk minimization principle and searches to minimize an upper bound of the generalization error. In addition, the solution of SVM may be a global optimum and thus, overfitting is unlikely to occur with SVM. In addition, SVM does not require too many data sample for training since it builds prediction models by only using some representative sample near the boundaries called support vectors. A number of experimental researches have indicated that SVM has been successfully applied in a variety of pattern recognition fields. However, there are three major drawbacks that can be potential causes for degrading SVM's performance. First, SVM is originally proposed for solving binary-class classification problems. Methods for combining SVMs for multi-class classification such as One-Against-One, One-Against-All have been proposed, but they do not improve the performance in multi-class classification problem as much as SVM for binary-class classification. Second, approximation algorithms (e.g. decomposition methods, sequential minimal optimization algorithm) could be used for effective multi-class computation to reduce computation time, but it could deteriorate classification performance. Third, the difficulty in multi-class prediction problems is in data imbalance problem that can occur when the number of instances in one class greatly outnumbers the number of instances in the other class. Such data sets often cause a default classifier to be built due to skewed boundary and thus the reduction in the classification accuracy of such a classifier. SVM ensemble learning is one of machine learning methods to cope with the above drawbacks. Ensemble learning is a method for improving the performance of classification and prediction algorithms. AdaBoost is one of the widely used ensemble learning techniques. It constructs a composite classifier by sequentially training classifiers while increasing weight on the misclassified observations through iterations. The observations that are incorrectly predicted by previous classifiers are chosen more often than examples that are correctly predicted. Thus Boosting attempts to produce new classifiers that are better able to predict examples for which the current ensemble's performance is poor. In this way, it can reinforce the training of the misclassified observations of the minority class. This paper proposes a multiclass Geometric Mean-based Boosting (MGM-Boost) to resolve multiclass prediction problem. Since MGM-Boost introduces the notion of geometric mean into AdaBoost, it can perform learning process considering the geometric mean-based accuracy and errors of multiclass. This study applies MGM-Boost to the real-world bond rating case for Korean companies to examine the feasibility of MGM-Boost. 10-fold cross validations for threetimes with different random seeds are performed in order to ensure that the comparison among three different classifiers does not happen by chance. For each of 10-fold cross validation, the entire data set is first partitioned into tenequal-sized sets, and then each set is in turn used as the test set while the classifier trains on the other nine sets. That is, cross-validated folds have been tested independently of each algorithm. Through these steps, we have obtained the results for classifiers on each of the 30 experiments. In the comparison of arithmetic mean-based prediction accuracy between individual classifiers, MGM-Boost (52.95%) shows higher prediction accuracy than both AdaBoost (51.69%) and SVM (49.47%). MGM-Boost (28.12%) also shows the higher prediction accuracy than AdaBoost (24.65%) and SVM (15.42%)in terms of geometric mean-based prediction accuracy. T-test is used to examine whether the performance of each classifiers for 30 folds is significantly different. The results indicate that performance of MGM-Boost is significantly different from AdaBoost and SVM classifiers at 1% level. These results mean that MGM-Boost can provide robust and stable solutions to multi-classproblems such as bond rating.

ABSTRACTIn many remote-sensing projects, one is usually interested in a small number of land-cover classes present in a study area and not in all the land-cover classes that make-up the landscape. Previous studies in supervised classification of satellite images have tackled specific class mapping problem by isolating the classes of interest and combining all other classes into one large class, usually called others, and by developing a binary classifier to discriminate the class of interest from the others. Here, this approach is called focused approach. The strength of the focused approach is to decompose the original multi-class supervised classification problem into a binary classification problem, focusing the process on the discrimination of the class of interest. Previous studies have shown that this method is able to discriminate more accurately the classes of interest when compared with the standard multi-class supervised approach. However, it may be susceptible to data imbalance problems present in the training data set, since the classes of interest are often a small part of the training set. A result the classification may be biased towards the largest classes and, thus, be sub-optimal for the discrimination of the classes of interest. This study presents a way to minimize the effects of data imbalance problems in specific class mapping using cost-sensitive learning. In this approach errors committed in the minority class are treated as being costlier than errors committed in the majority class. Cost-sensitive approaches are typically implemented by weighting training data points accordingly to their importance to the analysis. By changing the weight of individual data points, it is possible to shift the weight from the larger classes to the smaller ones, balancing the data set. To illustrate the use of the cost-sensitive approach to map specific classes of interest, a series of experiments with weighted support vector machines classifier and Landsat Thematic Mapper data were conducted to discriminate two types of mangrove forest (high-mangrove and low-mangrove) in Saloum estuary, Senegal, a United Nations Educational, Scientific and Cultural Organisation World Heritage site. Results suggest an increase in overall classification accuracy with the use of cost-sensitive method (97.3%) over the standard multi-class (94.3%) and the focused approach (91.0%). In particular, cost-sensitive method yielded higher sensitivity and specificity values on the discrimination of the classes of interest when compared with the standard multi-class and focused approaches.

In the field of data mining applications, the most challenging research work is to classify imbalanced data in supervised learning. From the beginning of the machine learning era, a huge number of researches has done by inventing a lot of successful data balancing methods, however, these are not yet fully compatible to learn from imbalanced data. The key reason is data is producing in every moment and randomly by various sources, like humans, machines, sensors, robots, and so on. To date, the existing machine learning algorithms mostly biased to the majority class instances and neglecting the minority class instances, therefore they have a crucial effect on the prediction capacity and exactness on the final results. To deal with imbalanced data the most popular classifiers are under-sampling and over-sampling, cost-sensitive learning and ensemble learning. In this paper, we have presented a novel data balancing method by employing clustering technique with SVM especially decision boundary instances for creating balanced data. The proposed approach selects the most informative majority class and minority class support vectors nearest to the decision boundary or hyperplane, and finally, combines them to establishes a novel method. We have experimented with the performance of our proposed technique with some popular single classifiers such as C4.5 Decision Tree (DT) and Naive Bayes (NB) classifier and ensemble classifiers such as Random Forest and AdaBoost on thirteen standard real-life imbalanced datasets. The experimental result shows apparently, that the proposed idea generates improved accuracy when classifying both the minority and majority class instances compared to other existing methods.

Network intrusion datasets often face class imbalance issues in intrusion detection tasks, where the number of majority class samples is much higher than minority class samples. Current solutions face notable limitations: traditional normalization weakens the multimodal distribution of continuous features, while mainstream generative models focus excessively on minority class mining while neglecting majority class information. To address these issues, we propose M2M-VAEGAN, which innovatively incorporates a Variational Gaussian Mixture (VGM) model to preserve multimodal characteristics of continuous features. We design a transfer learning framework, pre-training on majority classes to capture general attack patterns, followed by fine-tuning with balanced batches of majority and minority samples to prevent catastrophic forgetting. Additionally, we enhance the VAEGAN architecture with an auxiliary classifier to strengthen conditional information learning. On the NSL-KDD and CIC-IDS2017 datasets, M2M-VAEGAN outperforms methods such as SMOTE, CTGAN, and CTABGAN, achieving a 1.25% to 6.42% improvement in minority class recall. These results demonstrate the effectiveness of the proposed approach.

The class imbalance problem is a well-known classification challenge in machine learning that has vexed researchers for over a decade. Under-representation of one or more of the target classes (minority class(es)) as compared to others (majority class(es)) can restrict the application of conventional classifiers directly on the data. In addition, emerging challenges such as overlapping classes, make class imbalance even harder to solve. Class overlap is caused due to ambiguous regions in the data where the prior probability of two or more classes are approximately equal. We are motivated to address the challenge of class overlap in the presence of imbalanced classes by a problem in pervasive computing. Specifically, we are designing smart environments that perform health monitoring and assistance. Our solution, ClusBUS, is a clustering-based under sampling technique that identifies data regions where minority class samples are embedded deep inside majority class. By removing majority class samples from these regions, ClusBUS preprocesses the data in order to give more importance to the minority class during classification. Experiments show that ClusBUS achieves improved performance over an existing method for handling class imbalance.

Class imbalance occurs when the distribution of classes between the majority and the minority classes is not the same. The data on imbalanced classes may vary from mild to severe. The effect of high-class imbalance may affect the overall classification accuracy since the model is most likely to predict most of the data that fall within the majority class. Such a model will give biased results, and the performance predictions for the minority class often have no impact on the model. The use of the oversampling technique is one way to deal with high-class imbalance, but only a few are used to solve data imbalance. This study aims for an in-depth performance analysis of the oversampling techniques to address the high-class imbalance problem. The addition of the oversampling technique will balance each class’s data to provide unbiased evaluation results in modeling. We compared the performance of Random Oversampling (ROS), ADASYN, SMOTE, and Borderline-SMOTE techniques. All oversampling techniques will be combined with machine learning methods such as Random Forest, Logistic Regression, and k-Nearest Neighbor (KNN). The test results show that Random Forest with Borderline-SMOTE gives the best value with an accuracy value of 0.9997, 0.9474 precision, 0.8571 recall, 0.9000 F1-score, 0.9388 ROC-AUC, and 0.8581 PRAUC of the overall oversampling technique.

Although imbalanced data have been studied for many years, the problem of data imbalance is still a major problem in the development of machine learning and artificial intelligence. The development of deep learning and artificial intelligence has further expanded the impact of imbalanced data, so studying imbalanced data classification is of practical significance. We propose an image oversampling algorithm based on the influence function and sample weights. Our scheme not only synthesizes high-quality minority class samples but also preserves the original features and information of minority class images. To address the lack of visually reasonable features in SMOTE when synthesizing images, we improve the pre-training model by removing the pooling layer and the fully connected layer in the model, extracting the important features of the image by convolving the image, executing SMOTE interpolation operation on the extracted important features to derive the synthesized image features, and inputting the features into a DCGAN network generator, which maps these features into the high-dimensional image space to generate a realistic image. To verify that our scheme can synthesize high-quality images and thus improve classification accuracy, we conduct experiments on the processed CIFAR10, CIFAR100, and ImageNet-LT datasets.