COMPARATIVE ANALYSIS OF CLUSTER CONCENTRIC CIRCLE BASED UNDER SAMPLING OVER LOW VERSUS HIGH DIMENSIONAL IMBALANCED DATASETS

DBFS: An effective Density Based Feature Selection scheme for small sample size and high dimensional imbalanced data sets

One of the more prevalent problems when working with bioinformatics datasets is class imbalance, when there are more instances in one class compared to the other class (es). This problem is made worse because frequently, the class of interest is also the minority class. A possible solution is data sampling, a powerful tool for combating class imbalance by adding or removing instances to make the dataset more balanced. In addition to the choice of including data sampling, one of the most important decisions when applying data is what the final class ratio should be. Commonly, the final class ratio when data is applied is 50:50, however it is an open question whether other ratios are more appropriate for certain imbalanced datasets (all datasets in this paper have 25.16% minority instances or less) where a 50:50 ratio requires extreme modification to the dataset. In this work we compare six different data approaches (feature selection with the pair wise combinations of three data techniques and two final class ratios) with feature selection without data with the goal of determining if the inclusion of data is beneficial and if so, what should be the final class ratio. In order to test the six data approaches and feature selection alone thoroughly, we utilize seven imbalanced and high-dimensional datasets, three feature selection techniques, and six classifiers. Our results show that for a majority of scenarios, random under along with either 35:65 or 50:50 is the best data approach. Statistical analysis shows that there is significant difference between the data approaches. However, despite this, we still recommend using random under along with 35:65 as the final class ratio. This is because of the frequency of both random under and 35:65 being the most frequent top performing data technique and class ratio respectively. Additionally, 35:65 will have fewer negative impacts than 50:50 (less data loss or over fitting, which makes it a better choice if all other factors are equal) and random under is more computationally efficient than any other form of sampling, including no sampling (both by not requiring any internal calculations and by producing a reduced, easier-to-work-with dataset). To our knowledge, this is the most comprehensive work which focuses on the choice of the inclusion and implementation of data with different final class ratios on bioinformatics datasets which exhibit such large levels of class imbalance.

In the domain of bioinformatics, two common problems encountered when analyzing real-world datasets are class imbalance and high dimensionality. Boosting is a technique that can be used to improve classification performance, even in the presence of class imbalance. In addition, data sampling and feature selection are two important preprocessing techniques used to counter the adverse effects of both challenges collectively. In this study, we examine whether the inclusion of boosting along with joint deployment of feature selection and data sampling techniques affect the classification performance of inductive models. To this end, we used two approaches: filter-based feature selection followed by either data sampling (denoted as FS-DS) or a hybrid data sampling and boosting technique entitled RUSBoost (denoted as FRB) which integrates random under sampling within the boosting process. We conducted an extensive experimental study using six high dimensional and imbalanced bioinformatics datasets along with three learners and four feature subset sizes. Our results show that the improvement of classification performance due to boosting depends on the choice of learner used to build the model. We recommend FRB because it outperforms FS-DS for nearly all scenarios. Additionally, our ANOVA analysis shows that the FRB is statistically distinguishable from the FS-DS when using the LR learner. To our knowledge, this is the first study to investigate the effects of boosting along with combined feature selection and data sampling on classification performance of inductive models in the domain of bioinformatics.

High-dimensional imbalanced datasets pose significant challenges in pattern recognition, often leading to overfitting and classifier bias toward majority classes. While numerous feature selection algorithms exist, most struggle to effectively address both high dimensionality and class imbalance simultaneously. This paper introduces Harmony Search Kullback–Leibler (HKL), a novel feature selection algorithm that integrates Kullback–Leibler divergence with the Harmony Search metaheuristic to specifically address these dual challenges. HKL establishes an information-theoretic foundation by employing KL divergence as a statistical framework to evaluate feature subsets based on their ability to separate minority and majority classes. Unlike existing Harmony Search variants that operate as class-blind optimizers treating feature selection as a generic optimization problem, HKL fundamentally shifts the paradigm by incorporating direct class distribution awareness into the optimization process. The algorithm implements a dual optimization approach that simultaneously balances classification performance metrics with class distribution divergence measures. This design specifically enhances minority class discrimination by prioritizing features that maximize the divergence between class distributions, ensuring that selected features provide discriminative power for underrepresented classes rather than simply favoring the majority class. Experimental validation across multiple high-dimensional biomedical datasets demonstrates that HKL consistently outperforms existing state-of-the-art methods in terms of AUC and G-mean metrics, with particular improvements for minority class classification. The algorithm achieves optimal performance while using substantially reduced feature subsets, often requiring only a quarter to half of the original features to maintain or exceed baseline classification accuracy. Statistical significance testing confirms that these performance improvements represent genuine algorithmic advantages rather than random variation. The proposed approach offers an effective solution to both dimensionality reduction and class imbalance challenges, providing a valuable tool for complex classification tasks across various domains.

Network intrusion detection systems are an important defense technology to guarantee information security and protect a network from attacks. In recent years, the broad learning system has attracted much attention and has been introduced into intrusion detection systems with some success. However, since the traditional broad learning system is a simple linear structure, when dealing with imbalanced datasets, it often ignores the feature learning of minority class samples, leading to a poorer recognition rate of minority class samples. Secondly, the high dimensionality and redundant features in intrusion detection datasets also seriously affect the training time and detection performance of the traditional broad learning system. To address the above problems, we propose a deep belief network broad equalization learning system. The model fully learns the large-scale high-dimensional dataset via a deep belief network and represents it as an optimal low-dimensional dataset, and then introduces the equalization loss v2 reweighing idea into the broad learning system and learns to classify the low-dimensional dataset via a broad equalization learning system. The model was experimentally tested using the CICIDS2017 dataset and fully validated using the CICIDS2018 dataset. Compared with other algorithms in the same field, the model shortens the training time and has a high detection rate and a low false alarm rate.

Learning from imbalanced data sets, where the number of examples of one (majority) class is much higher than the others, presents an important challenge to the machine learning community. Traditional machine learning algorithms may be biased towards the majority class, thus producing poor predictive accuracy over the minority class. In this paper, we describe a new approach that combines boosting, an ensemble-based learning algorithm, with data generation to improve the predictive power of classifiers against imbalanced data sets consisting of two classes. In the DataBoost-IM method, hard examples from both the majority and minority classes are identified during execution of the boosting algorithm. Subsequently, the hard examples are used to separately generate synthetic examples for the majority and minority classes. The synthetic data are then added to the original training set, and the class distribution and the total weights of the different classes in the new training set are rebalanced. The DataBoost-IM method was evaluated, in terms of the F-measures, G-mean and overall accuracy , against seventeen highly and moderately imbalanced data sets using decision trees as base classifiers. Our results are promising and show that the DataBoost-IM method compares well in comparison with a base classifier, a standard benchmarking boosting algorithm and three advanced boosting-based algorithms for imbalanced data set. Results indicate that our approach does not sacrifice one class in favor of the other, but produces high predictions against both minority and majority classes.

DNA microarray data analysis is notorious because it contains a large number of characteristics, asymmetrical class distribution, and it is restricted with less number of samples. We emphasize high-dimensional multi-class unbalanced issues in this work. The high dimensional, multi-class unbalanced problem has found it challenging for traditional classifiers to execute classification assignments effectively on both minority and majority classes. Numerous approaches have been made to handle either high dimensionality datasets or difficulties with class imbalance. Despite this, because of their complex relationships, few techniques have been presented to address the intersection of multi-class unbalanced and high-dimensional issues at the same time. Using multiple filter-based rankers (MF) and a hybrid political optimizer (PO), this study proposes unique hybrid algorithms for feature selection with the high-dimensional multi-class imbalanced issue. To evaluate the performance of the model, we have used four different classifiers named Support vector machine (SVM), Naive Bayes(NB), Decision Tree(DT), and K Nearest Neighbour(KNN). The experimental results show that the proposed methods are more effective than other well-known methods in the case of classification performance improvisation. Performance metrics accuracy indicates that the proposed methods are capable of searching the feature space and identifying very robust and discriminative features that best predict the minority class.

CIRA: Class imbalance resilient adaptive Gaussian process classifier

Deep learning has been popular for a few years, and it shows great capability on unsupervised leaning of representation. Deep belief network consists of multi layers of restricted Boltzmann machine(RBM) and a deep auto-encoder, which uses a stack architecture learning feature layer by layer. The learning rule is that one deeper layer learns more complex representations, which are the high level features of the input data, from the representations learnt by the layer before. Fuzzy C-Means(FCM) is one of the most popular clustering algorithms, which allows one piece of data belong to several clusters. In this paper the authors propose a novel clustering model, and introduce a novel clustering technique(DBNOC) which combines deep belief network and fuzzy c-means. The main idea is that: first, it clusters with the high level representations learnt by stacked RBM to produce the initial cluster center, then it uses the fine-tune step including one center holding clustering algorithm and deep auto-encoder to optimize the cluster center and membership between input data and every cluster by cross iteration. The authors use FCM clustering algorithm to fulfill the model and do experiment on both low dimensional datasets and high dimensional datasets. The experiment results suggest that the proposed deep belief network oriented clustering method is better than the standard K-Means and FCM algorithm on the test datasets. Even on high dimensional datasets, the DBNOC clustering method show more generalization. What's more, the proposed model is suitable both in theoretical and practical.

Tracking recurrence of correlation structure in neuronal recordings

Regarding the magnificent ability of the fuzzy logic in explanation and analyzing the real phenomena, using fuzzy data is growing rapidly. But the tools for doing data mining in a fuzzy data set; especially in high dimension, are not sufficient. Considering the remarkable ability of the LoOP (Local Outlier Probabilities) method in outlier detecting in high dimensional crisp data set, like supplying for each data object, determining outlier probability as scar which is easily explainable competitor on various synthetic and real world data sets, the aim of this article is to present an extended version of this method for using in high dimensional fuzzy database. In order to provide a completely fuzzy strategy, first we propose a new approach to assigning distance between fuzzy high dimension data based on Hausdorff and Vertex distance. After proving that this distance measuring is a metric, since LoOP is a distance based method, we redefined the LoOP algorithm step by step based on the new meter. Then we allocate an outlier score in the range of [0,1] to each data based on LoOP idea, to identify the outliers. The presented numerical tests show the efficiency of the new method, for random cases of knowing and unknowing the outlier and a real case. The results shows also the stability and robustness of the presented method more that its accuracy.

Medical diagnosis is a crucial step for patient treatment. However, diagnosis is prone to bias due to imbalanced datasets. To overcome the imbalanced dataset problem, simple minority oversampling technique (SMOTE) was proposed that can generate new synthetic samples at data level to create the balance between minority and majority classes. However, the synthetic samples are generated on a random basis which causes class mixture problem; thus, resulting in deteriorating the classification performance and biased diagnosis. In order to overcome the SMOTE shortcomings, some modified methods were proposed that try to generate synthetic samples along the line segment of selected minority samples. Most of these methods adopt one of the two policies for selecting minority samples to generate synthetic samples: borderline region sampling or safe region sampling. However, they both suffer from over-generalisation problem. We propose a modified SMOTE-based resampling method called RSMOTE to alleviate the medical imbalanced dataset problem. We provide an in-depth analysis and verify the performance of RSMOTE over imbalanced medical datasets. In this paper, the proposed RSMOTE divides the minority sample domain into four regions (normal, semi-normal, semi-critical, and critical) based on the minority sample density analysis. RSMOTE discovers the minority sample region globally and applies the resampling near a specific group of samples. Our analysis and experiments verify that if synthetic samples are generated in the regions with high minority sample density, classification performance will be improved due to low risk of class mixture. Unlike some safe region methods, RSMOTE decides the region of minority samples on a global basis, thus removing the over-generalisation problem. Classic and additional evaluation metrics are considered to measure the effectiveness of the modified method: Recall, FP Rate, Precision, F-Measure, ROC area, and Average Aggregated Metric. We carried out experiments over various imbalanced medical datasets. Based on the minority sample density analysis, we propose RSMOTE method that divides the minority sample domain into four regions. The proposed RSMOTE includes four re-sampling methods that each of them carries out resampling on a specific region. According to the experimental results, resampling on the regions with high minority sample density obtained better results while those with lower minority sample density got the inferior results. Thus, we conclude that the RSMOTE is a more flexible resampling method for the imbalanced medical datasets that is capable of generating samples with various minority sample densities.

Background: Class imbalance is often discussed as a strenuous task in the realm of sentiment analysis. In an imbalanced classification, few minority class instances are unable to provide sufficient information, therefore direct learning from an unbalanced dataset can produce unsatisfactory results. This work aims to address the problem of class imbalance. Methods: At primary level this study uses a novel Synthetic Minority Oversampling Technique (SMOTE) for balancing the dataset and then proposes an ensemble model, named Ensemble Bagging Support Vector Machine (EBSVM) for opinion mining. To measure the performance of the particular approach, numerous analyses are conducted on both imbalanced and balanced datasets. Then the work compares the effectiveness of the suggested model with three base classifiers (Nave Bayes (NB), Decision Tree (DT), and Support Vector Machine (SVM)). The customer reviews for restaurants is chose as the dataset for this work. Accuracy, precision, recall and F-measure are used as metrics for evaluation. Findings: According to the results, the suggested EBSVM model excels all other individual classifiers with the imbalanced and SMOTE balanced dataset. The balanced EBSVM classifier improves the imbalanced EBSVM Classifier in terms of accuracy. Precision, recall and F-measure of the minority class in the imbalanced classifiers have been improved in balanced Classifiers. Novelty: The performance of opinion mining classifiers for imbalanced and balanced datasets is evaluated in this paper. The work examines not only general opinions, but also specific aspects such as food, service, ambiance, quality, and price. Comparing the suggested model with existing classification algorithms in the literature, it has found that it outperformed the other models. Keywords: Bagging; Accuracy; Ensemble; Precision; Recall; Fmeasure

<abstract><p>In the realm of machine learning, where data-driven insights guide decision-making, addressing the challenges posed by class imbalance in datasets has emerged as a crucial concern. The effectiveness of classification algorithms hinges not only on their intrinsic capabilities but also on their adaptability to uneven class distributions, a common issue encountered across diverse domains. This study delves into the intricate interplay between varying class imbalance levels and the performance of ten distinct classification models, unravelling the critical impact of this imbalance on the landscape of predictive analytics. Results showed that random forest (RF) and decision tree (DT) models outperformed others, exhibiting robustness to class imbalance. Logistic regression (LR), stochastic gradient descent classifier (SGDC) and naïve Bayes (NB) models struggled with imbalanced datasets. Adaptive boosting (ADA), gradient boosting (GB), extreme gradient boosting (XGB), light gradient boosting machine (LGBM), and k-nearest neighbour (kNN) models improved with balanced data. Adaptive synthetic sampling (ADASYN) yielded more reliable predictions than the under-sampling (UNDER) technique. This study provides insights for practitioners and researchers dealing with imbalanced datasets, guiding model selection and data balancing techniques. RF and DT models demonstrate superior performance, while LR, SGDC and NB models have limitations. By leveraging the strengths of RF and DT models and addressing class imbalance, classification performance in imbalanced datasets can be enhanced. This study enriches credit risk modelling literature by revealing how class imbalance impacts default probability estimation. The research deepens our understanding of class imbalance's critical role in predictive analytics. Serving as a roadmap for practitioners and researchers dealing with imbalanced data, the findings guide model selection and data balancing strategies, enhancing classification performance despite class imbalance.</p></abstract>

Classification of imbalanced data set is a challenging problem as it is very difficult to achieve good classification accuracy for each class in case of imbalanced data sets. This problem arises in many real world applications like medical diagnosis of rare medical disease, fraud detection in financial domain, and faulty area detection in network troubleshooting etc. The imbalanced data set consists of small number of instances of minority classes and large number of instances of majority classes. Overall classification accuracy is computed by taking the ratio of correctly classified instances to total number of instances in a data set. For imbalanced data sets, correct classification of minority class instances contribute minimum in improvement of overall classification accuracy as compared to classification of majority class instances. Conventional classification techniques like Artificial Neural Network (ANN), Support Vector Machine (SVM), and Naïve Bayes (NB) consider overall classification accuracy of the classifier only and thus evolve biased classifiers in case of imbalanced data set. However, instances of minority classes may contain rare but important information in many real world data sets. Thus, a classification technique that provides good classification accuracy on both minority and majority classes is needed. This paper proposes a combination of Multi Objective Genetic Programming (MOGP) and probability based Gaussian classifier for classification of imbalanced data set. MOGP considers classification accuracy of each class as separate objective and not the overall accuracy as single objective. Gaussian classifier is generative classifier in which distribution of one class never affect the classification of instances of other classes. The proposed methodology is applied on classification of imbalanced data sets from medical, life science, automobile, and space science domain. The results suggest that MOGP classifier outperformed other conventional classifiers (ANN, SVM, and NB) on tested imbalanced data sets.