Boosting for Learning From Imbalanced, Multiclass Data Sets

Multi-class imbalanced big data classification on Spark

Genetic programming (GP) is an artificial intelligence technique that benefits from evolutionary computations allowing computers to solve problems automatically. In this paper, we present an optimized genetic-programming-based classifier that directly solves the multi-class classification problems in data mining and image analysis. A new fitness function is proposed for multiclass classification and brain tumor detection, which is validated by 10-fold cross validation. Instead of defining static thresholds as boundaries to differentiate between multiple labels, our work presents a method of classification where a GP system learns the relationships among experiential data and models them mathematically during the evolutionary process. We propose an optimized GP classifier based on a combination of pruning subtrees and a new fitness function. An orthogonal least squares algorithm is also applied in the training phase to create a robust GP classifier. Our approach has been assessed on six multiclass datasets and on a magnetic resonance imaging (MRI) brain image for tumor detection. The results of data classification for Iris, Wine, Glass, Pima, BUPA Liver and Balance Scale datasets are compared with existing algorithms. The high accuracy of brain tumor classification provided by our GP classifier confirms the strong ability of the developed technique for complicated classification problems. We compared our approach in terms of speed with previous GP algorithms as well. The analyzed results illustrate that the developed classifier produces a productive and rapid method for classification tasks that outperforms the previous methods for more challenging multiclass classification problems.

This paper proposes a convolutional neural network for diagnosing various lung illnesses from chest CT images based on a customized Medical Image Analysis and Detection network (MIDNet18). With simplified model building, minimal complexity, easy technique, and high-performance accuracy, the MIDNet-18 CNN architecture classifies binary and multiclass medical images. Fourteen convolutional layers, 7 pooling layers, 4 dense layers, and 1 classification layer comprise the MIDNet-18 architecture. The medical image classification process involves training, validating, and testing the MIDNet-18 model. In the Lung CT image binary class dataset, 2214 images as training set, 1800 images as validation set, and 831 as test set are considered for classifying COVID images and normal lung images. In the multiclass dataset, 6720 images as training sets belonging to 3 classes, 3360 images as validation sets and 601 images as test sets are considered for classifying COVID, cancer images and normal images. Independent sample size calculated for binary classification is 26 samples for each group. Similarly, 10 sample sizes are calculated for multiclass dataset classification keeping GPower at 80%. To validate the performance of the MIDNet18 CNN architecture, the medical images of two different datasets are compared with existing models like LeNet-5, VGG-16, VGG-19, ResNet-50. In multiclass classification, the MIDNet-18 architecture gives better training accuracy and test accuracy, while the LeNet5 model obtained 92.6% and 95.9%, respectively. Similarly, VGG-16 is 89.3% and 77.2% respectively; VGG-19 is 85.8% and 85.4%, respectively; ResNet50 is 90.6% and 99%, respectively. For binary classification, the MIDNet18 architecture gives better training accuracy and test accuracy, while the LeNet-5 model has obtained 52.3% and 54.3%, respectively. Similarly, VGG 16 is 50.5% and 45.6%, respectively; VGG-19 is 50.6% and 45.6%, respectively; ResNet-50 is 96.1% and 98.4%, respectively. The classified images are further predicted using detectron-2 model and the results identify abnormalities (cancer, COVID-19) with 99% accuracy. The MIDNET18 is significantly more accurate than LeNet5, VGG19, VGG16 algorithms and is marginally better than the RESNET50 algorithm for the given lung binary dataset (Bonferroni — one-way Anova and pairwise comparison of MIDNET, LeNet5, VGG19, VGG16, and RESNET 50 ([Formula: see text])). The proposed MIDNet18 model is significantly more accurate than LeNet5, VGG19, VGG16, ResNet50 algorithms in classifying the diseases for the given multiclass lung dataset (Bonferroni — one-way Anova and pairwise comparison of MIDNET18, LeNet5, VGG19, VGG16, ResNet50 ([Formula: see text])).

Learning from imbalanced data sets, where the number of examples of one (majority) class is much higher than the others, presents an important challenge to the machine learning community. Traditional machine learning algorithms may be biased towards the majority class, thus producing poor predictive accuracy over the minority class. In this paper, we describe a new approach that combines boosting, an ensemble-based learning algorithm, with data generation to improve the predictive power of classifiers against imbalanced data sets consisting of two classes. In the DataBoost-IM method, hard examples from both the majority and minority classes are identified during execution of the boosting algorithm. Subsequently, the hard examples are used to separately generate synthetic examples for the majority and minority classes. The synthetic data are then added to the original training set, and the class distribution and the total weights of the different classes in the new training set are rebalanced. The DataBoost-IM method was evaluated, in terms of the F-measures, G-mean and overall accuracy , against seventeen highly and moderately imbalanced data sets using decision trees as base classifiers. Our results are promising and show that the DataBoost-IM method compares well in comparison with a base classifier, a standard benchmarking boosting algorithm and three advanced boosting-based algorithms for imbalanced data set. Results indicate that our approach does not sacrifice one class in favor of the other, but produces high predictions against both minority and majority classes.

Convolutional Neural Networks (CNNs) have proven to be very effective in image classification due to their status as a powerful feature learning algorithm. Traditional approaches have considered the problem of multiclass classification, where the goal is to classify a set of objects at once. However, co-occurrence can make the discriminative features of the target less salient and may lead to overfitting of the model, resulting in lower performance. To address this, we propose a multi-label classification ensemble model including a Vision Transformer (ViT) and CNN for directly detecting one or multiple objects in an image. First, we improve the MobileNetV2 and DenseNet201 models using extra convolutional layers to strengthen image classification. In detail, three convolution layers are applied in parallel at the end of both models. ViT can learn dependencies among distant positions and local detail, making it an effective tool for multi-label classification. Finally, an ensemble learning algorithm is used to combine the classification predictions of the ViT, the modified MobileNetV2, and DenseNet201 bands for increased image classification accuracy using a voting system. The performance of the proposed model is examined on four benchmark datasets, achieving accuracies of 98.24%, 98.89%, 99.91%, and 96.69% on ASCAL VOC 2007, PASCAL VOC 2012, MS-COCO, and NUS-WIDE 318, respectively, showing that our framework can enhance current state-of-the-art methods.

Bond rating is regarded as an important event for measuring financial risk of companies and for determining the investment returns of investors. As a result, it has been a popular research topic for researchers to predict companies' credit ratings by applying statistical and machine learning techniques. The statistical techniques, including multiple regression, multiple discriminant analysis (MDA), logistic models (LOGIT), and probit analysis, have been traditionally used in bond rating. However, one major drawback is that it should be based on strict assumptions. Such strict assumptions include linearity, normality, independence among predictor variables and pre-existing functional forms relating the criterion variablesand the predictor variables. Those strict assumptions of traditional statistics have limited their application to the real world. Machine learning techniques also used in bond rating prediction models include decision trees (DT), neural networks (NN), and Support Vector Machine (SVM). Especially, SVM is recognized as a new and promising classification and regression analysis method. SVM learns a separating hyperplane that can maximize the margin between two categories. SVM is simple enough to be analyzed mathematical, and leads to high performance in practical applications. SVM implements the structuralrisk minimization principle and searches to minimize an upper bound of the generalization error. In addition, the solution of SVM may be a global optimum and thus, overfitting is unlikely to occur with SVM. In addition, SVM does not require too many data sample for training since it builds prediction models by only using some representative sample near the boundaries called support vectors. A number of experimental researches have indicated that SVM has been successfully applied in a variety of pattern recognition fields. However, there are three major drawbacks that can be potential causes for degrading SVM's performance. First, SVM is originally proposed for solving binary-class classification problems. Methods for combining SVMs for multi-class classification such as One-Against-One, One-Against-All have been proposed, but they do not improve the performance in multi-class classification problem as much as SVM for binary-class classification. Second, approximation algorithms (e.g. decomposition methods, sequential minimal optimization algorithm) could be used for effective multi-class computation to reduce computation time, but it could deteriorate classification performance. Third, the difficulty in multi-class prediction problems is in data imbalance problem that can occur when the number of instances in one class greatly outnumbers the number of instances in the other class. Such data sets often cause a default classifier to be built due to skewed boundary and thus the reduction in the classification accuracy of such a classifier. SVM ensemble learning is one of machine learning methods to cope with the above drawbacks. Ensemble learning is a method for improving the performance of classification and prediction algorithms. AdaBoost is one of the widely used ensemble learning techniques. It constructs a composite classifier by sequentially training classifiers while increasing weight on the misclassified observations through iterations. The observations that are incorrectly predicted by previous classifiers are chosen more often than examples that are correctly predicted. Thus Boosting attempts to produce new classifiers that are better able to predict examples for which the current ensemble's performance is poor. In this way, it can reinforce the training of the misclassified observations of the minority class. This paper proposes a multiclass Geometric Mean-based Boosting (MGM-Boost) to resolve multiclass prediction problem. Since MGM-Boost introduces the notion of geometric mean into AdaBoost, it can perform learning process considering the geometric mean-based accuracy and errors of multiclass. This study applies MGM-Boost to the real-world bond rating case for Korean companies to examine the feasibility of MGM-Boost. 10-fold cross validations for threetimes with different random seeds are performed in order to ensure that the comparison among three different classifiers does not happen by chance. For each of 10-fold cross validation, the entire data set is first partitioned into tenequal-sized sets, and then each set is in turn used as the test set while the classifier trains on the other nine sets. That is, cross-validated folds have been tested independently of each algorithm. Through these steps, we have obtained the results for classifiers on each of the 30 experiments. In the comparison of arithmetic mean-based prediction accuracy between individual classifiers, MGM-Boost (52.95%) shows higher prediction accuracy than both AdaBoost (51.69%) and SVM (49.47%). MGM-Boost (28.12%) also shows the higher prediction accuracy than AdaBoost (24.65%) and SVM (15.42%)in terms of geometric mean-based prediction accuracy. T-test is used to examine whether the performance of each classifiers for 30 folds is significantly different. The results indicate that performance of MGM-Boost is significantly different from AdaBoost and SVM classifiers at 1% level. These results mean that MGM-Boost can provide robust and stable solutions to multi-classproblems such as bond rating.

Effective Deep Learning Data Augmentation Techniques for Diabetic Retinopathy Classification

Credit score models are essential tools for evaluating creditworthiness and mitigating financial risks. However, the imbalanced nature of multi-class credit score datasets poses significant challenges for traditional classification algorithms, leading to poor performance in minority classes. This study explores the effectiveness of Generative Adversarial Network (GAN)-based oversampling methods, including CTGAN, CopulaGAN, WGAN-GP, and DraGAN, in addressing this issue. By synthesizing realistic data for minority classes and integrating it with majority class data, the study benchmarks these GAN-based methods across classical (KNN, Decision Tree, Logistic Regression) and ensemble machine learning models (XGBoost, Random Forest, LightGBM). Evaluation metrics such as accuracy and F1-score reveal that WGAN-GP consistently achieves superior performance, especially when combined with Random Forest, outperforming other methods in balancing dataset representation and enhancing classification accuracy. The results showed that WGAN-GP + RF achieved 0.873 in accuracy, 0.936 F1-score in the “good” class, 0.806 F1-score in the “poor” class, and 0.816 F1-score in the “standard” class. The findings underscore the potential of GAN-based oversampling in improving multi-class credit score classification and highlight future directions, including hybrid sampling and cost-sensitive learning, to address remaining challenges.

FAMCDM: A fusion approach of MCDM methods to rank multiclass classification algorithms

A combinatorial optimization approach for multi-label associative classification

Overlooked pitfalls in multi-class machine learning classification in radiation oncology and how to avoid them

If their assumptions are not met, classifiers may fail. In this paper, the possibility of combining classifiers in multi-class problems is investigated. Multi-class classification problems are split into two class problems. For each of the latter problems an optimal classifier is determined. The results of applying the optimal classifiers on the two class problems can be combined using the Pairwise Couplingalgorithm by Hastie and Tibshirani (1998). In this paper exemplary situations are investigated where the respective assumptions of Naive Bayes or the classical Linear Discriminant Analysis (LDA, Fisher, 1936) fail. It is investigated at which degree of violations of the assumptions it may be advantageous to use single methods or a classifier combination by Pairwise Coupling.

Error Correcting Output Codes reveal an efficient strategy in dealing with multi-class classification problems. According to this technique, a multi-class problem is decomposed into several binary ones. On these created sub-problems we apply binary classifiers and then, by combining the acquired solutions, we are able to solve the initial multi-class problem. In this paper we consider the optimization of the Linear Discriminant Error Correcting Output Codes framework using Particle Swarm Optimization. In particular, we apply the Particle Swarm Optimization algorithm in order to optimally select the free parameters that control the split of the initial problem’s classes into sub-classes. Moreover, by using the Support Vector Machine as classifier we can additionally apply the Particle Swarm Optimization algorithm to tune its free parameters. Our experimental results show that by applying Particle Swarm Optimization on the Sub-class Linear Discriminant Error Correcting Output Codes framework we get a significant improvement in the classification performance.

In multiclass classification problems we face the challenge of having many binary classifiers. Consulting this large number of classifiers might be confusing and time consuming. In this paper, we propose a new framework for training and prediction in multiclass problems. In this framework, we perform traditional training. Next we map training examples to prediction models. Finally we produce the Example Classifier (EC). In prediction a new example is passed through the EC to determine the appropriate classifier which in turn makes the last prediction decision. We conduct experiments comparing our framework with one-VS-one and Directed Acyclic Graph (DAG) using Support Vector Machines. Additionally, we compare our model with well-known ensemble models, namely, AdaBoost and Bagging, Our results indicate that prediction accuracy is comparable to other methodologies with the advantage of consuming less prediction time.

Improved multiclass feature selection via list combination