Multi-label imbalanced data enrichment process in neural net classifier training

This research has three parts addressing classification, especially the imbalanced data problem, which is one of the most popular and essential issues in the domain of classification. The first part is to study the Adaptive Boosting (AdaBoost) algorithm. AdaBoost is an effective solution for classification, but it still needs improvement in the imbalanced data problem. This part proposes a method to improve the AdaBoost algorithm using the new weighted vote parameters for the weak classifiers. Our proposed weighted vote parameters are determined not only by the global error rate but also by the classification accuracy rate of the positive class, which is our primary interest. The imbalanced index of the data is also a factor in constructing our algorithms. The numeric studies show that our proposed algorithms outperform the traditional ones, especially regarding the evaluation criterion of the F--1 Measure. Theoretic proofs of the advantages of our proposed algorithms are presented. The second part treats the Relevance Vector Machine (RVM), which is a supervised learning algorithm extended from the Support Vector Machine (SVM) based on the Bayesian sparsity model. Compared with the regression problem, RVM classification is challenging to conduct because there is no closed-form solution for the weight parameter posterior. The original RVM classification algorithm uses Newton's method in optimization to obtain the mode of weight parameter posterior, then approximates it by a Gaussian distribution in Laplace's method. This original model would work, but it just applies the frequency methods in a Bayesian framework. This part first proposes a Generic Bayesian RVM classification, which is a pure Bayesian model. We conjecture that our algorithm achieves convergent estimates of the quantities of interest compared with the nonconvergent estimates of the original RVM classification algorithm. Furthermore, a fully Bayesian approach with the hierarchical hyperprior structure for RVM classification is proposed, which improves the classification performance, especially in the imbalanced data problem. The third part is an extended work of the second one. The original RVM classification model uses the logistic link function to build the likelihood, which makes the model hard to conduct since the posterior of the weight parameter has no closed-form solution. This part proposes the probit link function approach instead of the logistic one for the likelihood function in RVM classification, namely PRVM (RVM with the Probit link function). We show that the posterior of the weight parameter in our model follows the multivariate normal distribution and achieves a closed-form solution. A latent variable is needed in our algorithm to simplify the Bayesian computation greatly, and its conditional posterior follows a truncated normal distribution. Compared with the original RVM classification model, our proposed one is another pure Bayesian approach and it has a more efficient computation process. For the prior structure, we first consider the Normal-Gamma independent prior to propose a Generic Bayesian PRVM algorithm. Furthermore, the Fully Bayesian PRVM algorithm with a hierarchical hyperprior structure is proposed, which improves the classification performance, especially in the imbalanced data problem.

Bone age assessment is an important indicator of child maturity. Generally, a person evaluated for bone age is mostly in the puberty stage. Compared to toddlers and post-puberty stages, the data of bone age at the puberty stage are hence much easier to obtain. As a result, the amounts of bone age data collected at the toddlers and post-puberty stages are often less than that of bone age data at the pulpy stage. This causes a data imbalance problem that affects the prediction accuracy. To deal with this problem, a data imbalance immunity bone age assessment (DIIBAA) system is proposed which integrates a CNN-based autoencoder and CNN-based scoring network. The DIIBAA system builds three autoencoders for the bone age data of toddlers, puberty, and post-puberty stages. Since the three types of autoencoders do not interfere with each other, there is no data imbalance problem in the DIIBAA system. After that, the outputs of the three autoencoders are input to the scoring network, and the auto-encoder produced the image with the highest score is regarded as the final prediction result. In the experiments, imbalanced training data with a positive and negative sample ratio of 1:2 were used, which has been alleviated compared to the original very imbalanced data. Meanwhile, the DenseNet169-based autoencoders are employed in the experiments, and the obtained evaluation accuracies are improved.

The integration of industrialization and informatization has exposed industrial control systems (ICSs) to increasingly serious security challenges. Currently, the mainstream method to protect the security of ICSs is intrusion detection system (IDS) based on deep-learning. However, these methods depend on a massive amount of high-quality data. Owing to the characteristics and protocol limitations, ICSs data usually experience low-quality and data imbalance problems, which significantly affects the accuracy of IDS.In this study, an IDS for ICS that combines data expansion algorithm and CNN was proposed. A novel normalized neighborhood weighted convex combined random sample (NNW-CCRS) oversampling algorithm was designed, which automatically attenuates the effects of noise and expanding imbalanced data to produce balanced ICS datasets. By reducing the impact of imbalanced ICS data on IDSs, our system effectively protects the security of ICS. Secure Water Treatment dataset (SWaT) was used for experimental validation. The experimental results confirmed that the accuracy of the proposed system improved by approximately 20%, compared to the ICS without data expansion.

Although imbalanced data have been studied for many years, the problem of data imbalance is still a major problem in the development of machine learning and artificial intelligence. The development of deep learning and artificial intelligence has further expanded the impact of imbalanced data, so studying imbalanced data classification is of practical significance. We propose an image oversampling algorithm based on the influence function and sample weights. Our scheme not only synthesizes high-quality minority class samples but also preserves the original features and information of minority class images. To address the lack of visually reasonable features in SMOTE when synthesizing images, we improve the pre-training model by removing the pooling layer and the fully connected layer in the model, extracting the important features of the image by convolving the image, executing SMOTE interpolation operation on the extracted important features to derive the synthesized image features, and inputting the features into a DCGAN network generator, which maps these features into the high-dimensional image space to generate a realistic image. To verify that our scheme can synthesize high-quality images and thus improve classification accuracy, we conduct experiments on the processed CIFAR10, CIFAR100, and ImageNet-LT datasets.

R-GDORUS technology: Effectively solving the Raman spectral data imbalance in medical diagnosis

In recent years, credit evaluation has become an issue of increasing concern for financial institutions. However, since most research focuses on the risk classification process, the problem of data imbalance is ignored. In real data sets, there are often far more users with good credit than users with bad credit, and the imbalance of data often easily leads to a decline in the classification performance of the model. Therefore, previous research is very limited in practical application scenarios. In this paper, we establish a new integration method for credit evaluation, which is classified into three steps: First, data preprocessing. Before inputting samples into the model, we take a series of preprocessing steps, such as missing data processing, data dimensionality reduction. Secondly, in view of the imbalance problem, the data is divided into multiple clusters using an unsupervised clustering algorithm, and the SMOTE method is used to generate minority samples in the clusters whose ratio exceeds the threshold. Finally, the GBDT2NN and Factorization Machine methods are integrated to classify the samples. In order to verify the effectiveness of this method, we use the Kaggle competition data set for verification. The results show that this method is better than other algorithms in the field of credit evaluation in terms of recall rate and AUC value.

In machine learning, classification and regression are two of the most noteworthy key topics since they occur extensively in numerous real-world applications. However, real life data is hardly ever found balanced, rather skewed data is the common occurrence. This poses some serious challenges to the standard techniques of classification and regression. The performance and effectiveness of these techniques are substantially affected by overfitting, creating a bias towards the majority class. In recent years, quite a few number of methods have been introduced for classification of imbalanced data. But most of them are designed for binary classes and it is difficult or inefficient to extend them for multiple classes. Moreover, data imbalance problem occurs frequently in regression analysis too, with only a handful of algorithms robust enough to tackle to this problem. In this paper, we propose an effective recursive method for multi-class classification with imbalanced data. Our proposed algorithm partitions and balances the data, and is applied recursively coupled with ensemble techniques. Furthermore, we also extend our proposed method to solve the data imbalance problem in regression analysis. Experimental results demonstrate that the proposed recursive technique is effective and improves the performance when compared to existing methods for classification and regression with imbalanced distribution.

To address the imbalanced data problem in molecular docking-based virtual screening methods, this paper proposes a virtual screening method for drug proteins based on imbalanced data mining, which introduces machine learning technology into the virtual screening technology for drug proteins to deal with the imbalanced data problem in the virtual screening process and improve the accuracy of the virtual screening. First, to address the data imbalance problem caused by the large difference between the number of active compounds and the number of inactive compounds in the docking conformation generated by the actual virtual screening process, this paper proposes a way to improve the data imbalance problem using SMOTE combined with genetic algorithm to synthesize new active compounds artificially by upsampling active compounds. Then, in order to improve the accuracy in the virtual screening process of drug proteins, the idea of integrated learning is introduced, and the random forest (RF) extended from Bagging integrated learning technique is combined with the support vector machine (SVM) technique, and the virtual screening of molecular docking conformations using RF-SVM technique is proposed to improve the prediction accuracy of active compounds in docking conformations. To verify the effectiveness of the proposed technique, first, HIV-1 protease and SRC kinase were used as test data for the experiments, and then, CA II was used to validate the model of the test data. The virtual screening of drug proteins using the proposed method in this paper showed an improvement in both enrichment factor (EF) and AUC compared with the use of the traditional virtual screening, for the test dataset. Therefore, it can be shown that the proposed method can effectively improve the accuracy of drug virtual screening.

An unsolved problem in the digital mapping of categorical soil variables and soil types is the imbalanced number of observations, which leads to reduced accuracy and the loss of the minority class (the class with a significantly lower number of observations compared to other classes) in the final map. So far, synthetic over‐ and under‐sampling techniques have been explored in soil science; however, more efficient approaches that do not have the drawbacks of these techniques and guarantee retention of the minority classes in the produced map are essentially required. Such approaches suggested in the present study for digital mapping of soil classes include machine learning models of ensemble gradient boosting, cost‐sensitive learning and one‐class classification (OCC) of the minority class combined with multi‐class classification. In this regard, extreme gradient boosting (XGB) as an ensemble gradient learner, a cost‐sensitive decision tree (CSDT) within the C5.0 algorithm, and a one‐class support vector machine combined with multi‐class classification (OCCM) were investigated to map eight soil great groups with a naturally imbalanced frequency of observations in northwest Iran. A total of 453 profile data points were used for mapping the soil great groups of the study area. A data split was done manually for each class separately, which resulted in an overall 70% of the data for calibration and 30% for validation. The bootstrapping approach of calibration (with 10 runs) was performed to produce multiple maps for each model. The 10 bootstraps were evaluated against the hold‐out validation dataset. The average values of accuracy measures, including Kappa (K), overall accuracy (OA), producer's accuracy (PA) and user's accuracy (UA), were explored. In addition, the results of this study were compared with a previous study in the same area, in which resampling techniques were used to deal with imbalanced data for digital soil class mapping. The findings show that all three suggested methods can deal well with the imbalanced classification problem, with OCCM showing the highest K (= 0.76) and OA (= 82) in the validation stage. Also, this model can guarantee the retention of the minority classes in the final map. Comparing the present approaches with the previous study approach demonstrates that the three newly suggested methods can remarkably increase both overall and individual class accuracy for mapping.

The intelligent diagnosis of rotating machinery with big data has been widely studied. However, due to the variability of working conditions and difficulty in marking fault samples, it is difficult to obtain enough high-quality fault marking data for training bearing fault diagnosis models in practical industrial application scenarios. Aiming at the problem of training data imbalance caused by lack of fault samples, a novel metalearning fault diagnosis method (MOFD) is proposed to get the bearing fault diagnosis solution under data imbalance. Firstly, in order to enhance the variety of fault samples, a Feature Space Density Adaptive Synthetic Minority Oversampling Technique (FSDA-SMOTE) is proposed in this paper, which takes the density difference of minority samples in the spatial domain within the class as the constraint of local neighbor similarity to generate new fault samples for data augmentation. In addition, in order to strengthen the model’s learning ability and diagnosis performance under limited fault samples, a residual-attention convolutional neural network (RA-CNN) was constructed to identify the deep features of fault signals, and a metalearning strategy based on parameter gradient optimization was applied to RA-CNN for refining the learning process of the diagnosis model. Finally, the reliability of the proposed method is verified through experimental analysis of public bearing dataset.

Fine-grained attribute analysis and data imbalance have always been research hotspots in the field of computer vision. Due to the complexity and diversity of fine-grained attribute images, traditional image classification methods have shortcomings in paying attention to fine-grained attributes of images and perform poorly when dealing with imbalanced data sets. To overcome these problems, this study proposes a fine-grained image threshold classification algorithm based on deep metric learning. By introducing a metric learning method, the focus on fine-grained attributes of images is enhanced. At the same time, by applying pairwise loss and proxy loss, the classification accuracy of the model is improved and the model convergence speed is accelerated. In order to deal with the problem of data imbalance, a classifier based on threshold analysis is designed. The classifier uses threshold analysis technology to achieve multi-level classification of fine-grained images, thereby improving the problem of low classification accuracy of a few categories in imbalanced data sets. Experimental results show that the proposed fine-grained image threshold classification algorithm based on deep metric learning is significantly superior to other methods in terms of classification accuracy.

The parameters of support vector machines (SVMs) such as kernel parameters and the penalty parameter have a great influence on the accuracy and complexity of the classification models. In the past, different evolutionary optimization algorithms were employed for optimizing SVMs; in this paper, we propose a social ski-driver (SSD) optimization algorithm which is inspired from different evolutionary optimization algorithms for optimizing the parameters of SVMs, with the aim of improving the classification performance. To cope with the problem of imbalanced data which is one of the challenging problems for building robust classification models, the proposed algorithm (SSD-SVM) was enhanced to deal with imbalanced data. In this study, eight standard imbalanced datasets were used for testing our proposed algorithm. For verification, the results of the SSD-SVM algorithm are compared with grid search, which is a conventional method of searching parameter values, and particle swarm optimization (PSO). The experimental results show that the SSD-SVM algorithm is capable of finding near-optimal values of SVMs parameters. The results also demonstrated high classification performance compared to the PSO algorithm.

Fully Bayesian analysis of the relevance vector machine classification for imbalanced data problem

Ubiquitous wearable and mobile devices provide access to a diverse set of data. However, the mobility demand for our devices naturally imposes constraints on their computational and communication capabilities. A solution is to locally learn knowledge from data captured by ubiquitous devices, rather than to store and transmit the data in its original form. In this paper, we develop a federated learning framework, called Centaur, to incorporate on-device data selection at the edge, which allows partition-based training of a deep neural nets through collaboration between constrained and resourceful devices within the multidevice ecosystem of the same user. We benchmark on five neural net architecture and six datasets that include image data and wearable sensor time series. On average, Centaur achieves ~19% higher classification accuracy and ~58% lower federated training latency, compared to the baseline. We also evaluate Centaur when dealing with imbalanced non-iid data, client participation heterogeneity, and different mobility patterns. To encourage further research in this area, we release our code at github.com/nokia-bell-labs/data-centric-federated-learning.

Imbalanced data is a common and serious problem in many biomedical classification tasks. It causes a bias on the training of classifiers and results in lower accuracy of minority classes prediction. This problem has attracted a lot of research interests in the past decade. Unfortunately, most research efforts only concentrate on 2-class problems. In this paper, we study a new method of formulating a multiclass Support Vector Machine (SVM) problem for imbalanced biomedical data to improve the classification performance. The proposed method applies cost-sensitive approach and ramp loss function to the Crammer and Singer multiclass SVM formulation. Experimental results on multiple biomedical datasets show that the proposed solution can effectively cure the problem when the datasets are noisy and highly imbalanced.