Adaptive Large Margin Training for Multilabel Classification

A novel random fast multi-label deep forest classification algorithm

Prediction of the stage of cancer plays an important role in planning the course of treatment and has been largely reliant on imaging tools which do not capture molecular events that cause cancer progression. Gene-expression data-based analyses are able to identify these events, allowing RNA-sequence and microarray cancer data to be used for cancer analyses. Breast cancer is the most common cancer worldwide, and is classified into four stages - stages 1, 2, 3, and 4 [2]. While machine learning models have previously been explored to perform stage classification with limited success, multi-class stage classification has not had significant progress. There is a need for improved multi-class classification models, such as by investigating deep learning models. Gene-expression-based cancer data is characterised by the small size of available datasets, class imbalance, and high dimensionality. Class balancing methods must be applied to the dataset. Since all the genes are not necessary for stage prediction, retaining only the necessary genes can improve classification accuracy. The breast cancer samples are to be classified into 4 classes of stages 1 to 4. Invasive ductal carcinoma breast cancer samples are obtained from The Cancer Genome Atlas (TCGA) and Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) datasets and combined. Two class balancing techniques are explored, synthetic minority oversampling technique (SMOTE) and SMOTE followed by random undersampling. A hybrid feature selection pipeline is proposed, with three pipelines explored involving combinations of filter and embedded feature selection methods: Pipeline 1 - minimum-redundancy maximum-relevancy (mRMR) and correlation feature selection (CFS), Pipeline 2 - mRMR, mutual information (MI) and CFS, and Pipeline 3 - mRMR and support vector machine-recursive feature elimination (SVM-RFE). The classification is done using deep learning models, namely deep neural network, convolutional neural network, recurrent neural network, a modified deep neural network, and an AutoKeras generated model. Classification performance post class-balancing and various feature selection techniques show marked improvement over classification prior to feature selection. The best multiclass classification was found to be by a deep neural network post SMOTE and random undersampling, and feature selection using mRMR and recursive feature elimination, with a Cohen-Kappa score of 0.303 and a classification accuracy of 53.1%. For binary classification into early and late-stage cancer, the best performance is obtained by a modified deep neural network (DNN) post SMOTE and random undersampling, and feature selection using mRMR and recursive feature elimination, with an accuracy of 81.0% and a Cohen-Kappa score (CKS) of 0.280. This pipeline also showed improved multiclass classification performance on neuroblastoma cancer data, with a best area under the receiver operating characteristic (auROC) curve score of 0.872, as compared to 0.71 obtained in previous work, an improvement of 22.81%. The results and analysis reveal that feature selection techniques play a vital role in gene-expression data-based classification, and the proposed hybrid feature selection pipeline improves classification performance. Multi-class classification is possible using deep learning models, though further improvement particularly in late-stage classification is necessary and should be explored further.

目的 基于计算机的胸腔X线影像疾病检测和分类目前存在误诊率高，准确率低的问题。本文在视觉Transformer（vision Transformer，ViT）预训练模型的基础上，通过迁移学习方法，实现胸腔X线影像辅助诊断，提高诊断准确率和效率。方法 选用带有卷积神经网络（convolutional neural network，CNN）的ViT模型，其在超大规模自然图像数据集中进行了预训练；通过微调模型结构，使用预训练的ViT模型参数初始化主干网络，并迁移至胸腔X线影像数据集中再次训练，实现疾病多标签分类。结果 在IU X-Ray数据集中对ViT迁移学习前、后模型平均AUC（area under ROC curve）得分进行对比分析实验。结果表明，预训练ViT模型平均AUC得分为0.774，与不使用迁移学习相比提升了0.208。并针对模型结构和数据预处理进行了消融实验，对ViT中的注意力机制进行可视化，进一步验证了模型有效性。最后使用Chest X-Ray14和CheXpert数据集训练微调后的ViT模型，平均AUC得分为0.839和0.806，与对比方法相比分别有0.014～0.031的提升。结论 与其他方法相比，ViT模型胸腔X线影像的多标签分类精确度更高，且迁移学习可以在降低训练成本的同时提升ViT模型的分类性能和泛化性。消融实验与模型可视化表明，包含CNN结构的ViT模型能重点关注有意义的区域，高效获取胸腔X线影像的视觉特征。;Objective The chest X-ray-relevant screening and diagnostic method is essential for radiology nowadays. Most of chest X-ray images interpretation is still restricted by clinical experience and challenged for misdiagnose and missed diagnoses. To detect and identify one or more potential diseases in images automatically，it is beneficial for improving diagnostic efficiency and accuracy using computer-based technique. Compared to natural images，multiple lesions are challenged to be detected and distinguished accurately in a single image because abnormal areas have a small proportion and complex representations in chest X-ray images. Current convolutional neural network（CNN）based deep learning models have been widely used in the context of medical imaging. The structure of the CNN convolution kernel has sensitive to local detail information，and it is possible to extract richer image features. However，the convolution kernel cannot be used to get global information，and the features-extracted are restricted of redundant information like its relevance of background， muscles，and bones. The model’s performance in multi-label classification tasks are affected to a certain extent. At present，the vision Transformer（ViT）model has achieved its priorities in computer vision-related tasks. The ViT can be used to capture information simultaneously and effectively for multiple regions of the entire image. However，it is required to use large-scale dataset training to achieve good performance. Due to some factors like patient privacy and manual annotate costs，the size of the chest X-ray image data set has been limited. To reduce the model's dependence on data scale and improve the performance of multi-label classification，we develop the CNN-based ViT pre-training model in terms of the transfer learning method for diagnosis-assisted of chest X-ray image and multi-label classification. Method The CNN-based ViT model is pre-trained on a huge scale ground truth dataset，and it is used to obtain the initial parameters of the model. The model structure is fine-tuned according to the features of chest X-ray dataset. A 1×1 convolution layer is used to convert the chest X-ray images channels between 1 to 3. The number of output nodes of the linear layer in the classifier is balanced from 1 000 to the number of chest X-ray classification labels，and the Sigmoid is used as an activation function. The parameters of the backbone network are initialized in terms of the pre-trained ViT model parameters，and it is trained in the chest X-ray dataset after that to complete multi-label classification. The experiment is configured of Python3. 7 and PyTorch1. 8 to construct the model and RTX3090 GPU for training. Stochastic gradient descent（SGD）optimizer，binary cross-entropy（BCE）loss function，an initial learning rate of 1E-3，the cosine annealing learning rate decay are used. For training，each image is scaled to a size of 512×512 pixels，and a 224×224 pixels area and it is then cropped in random as the model input，and data augmentation is performed randomly by some of the flipping，perspective transformation， shearing，translation，zooming，and changing brightness. For testing，the chest X-ray image is scaled to 256×256 pixels and center crop a 224×224 area to input the trained model. Result The experiment is performed on the IU X-Ray，which is a small-scale chest X-ray dataset. This model is evaluated in quantitative using the average of area under ROC curve （AUC）scores across all classification labels. The results show that the average AUC score of the pre-trained ViT model is 0. 774. The accuracy and training efficiency of the non-pre-trained ViT model is dropped significantly. The average AUC score is reached to 0. 566 only，which is 0. 208 lower. In addition，the attention mechanism heat map is generated based on the ViT model，which can strengthen the interpretability of the model. A series of ablation experiments are carried out for data augmentation，model structure，and batch size design. The fine-tuned ViT model is trained on the Chest-Ray14 and CheXpert dataset as well. The average AUC score is reached to 0. 839 and 0. 806，which is optimized by 0. 014 and 0. 031. Conclusion A pre-trained ViT model is used for the multi-label classification of chest X-ray images via transfer learning. The experimental results illustrate that the ViT has its stronger multi-label classification performance in chest Xray images，and its attention mechanism is beneficial for lesions precision-focused like the interior of the chest cavity and the heart. Transfer learning is potential to improve the classification performance and model generalization of the ViT in small-scale datasets，and the training cost is reduced greatly. Ablation experiments demonstrate that the incorporated model of CNN and Transformer has its priority beyond single-structure model. Data enhancement and the batch size cutting can improve the performance of the model，but smaller scale of batch is still interlinked to longer training span. To improve the model's ability，we predict that future research direction can be focused on the extraction for complex disease and highlevel semantic information，such as their small lesions，disease location，and severity.

Multi-label image classification aims to classify all categories in images simultaneously. When current multi-label classification methods meet fine-grained objects in a single image, the extreme inter-class similarity and over-prediction problems are two major challenges that hinder model performance. To solve the above two problems, we propose Voronoi density based Locally Unique Network (VoLUNet). First, due to high correlation between predictions of different classes, following the Kolmogorov-Arnold Network (KAN), we design the Weak Inter-class Correlation Classifier (WIC-Classifier) to replace linear weights setting in MLP architecture, promoting the potential of fine-grained discrimination. Second, we propose a Local Non-Maximum Suppression (Local-NMS) loss to multi-label classification model, predicting only one unique class with high prediction value for each local region. Third, different classes may have different pixel proportions and Local-NMS loss will be imbalanced for diverse fine-grained classes, we design the Voronoi Density based Superpixel Module (VDSM) to balance the quantities of local feature vectors with different classes. Finally, comprehensive experiments are conducted on four datasets, TreeSatAI, GeoLifeCLEF, FothemNet ShipRSImageNet, and our VoLUNet can significantly improve the classification performance compared to current state-of-the-art models. Codes of this paper are public available at https://github.com/cv516Buaa/BinghaoLiu/tree/main/VoLUNet.

Multi label classification is concerned with learning from a set of instances that are associated with a set of labels, that is, an instance could be associated with multiple labels at the same time. This task occurs frequently in application areas like text categorization, multimedia classification, bioinformatics, protein function classification and semantic scene classification. Current multi-label classification methods could be divided into two categories. The first is called problem transformation methods, which transform multi-label classification problem into single label classification problem, and then apply any single label classifier to solve the problem. The second category is called algorithm adaptation methods, which adapt an existing single label classification algorithm to handle multi-label data. In this paper, we propose a multi-label classification approach based on correlations among labels that use both problem transformation methods and algorithm adaptation methods. The approach begins with transforming multi-label dataset into a single label dataset using least frequent label criteria, and then applies the PART algorithm on the transformed dataset. The output of the approach is multi-labels rules. The approach also tries to get benefit from positive correlations among labels using predictive Apriori algorithm. The proposed approach has been evaluated using two multi-label datasets named (Emotions and Yeast) and three evaluation measures (Accuracy, Hamming Loss, and Harmonic Mean). The experiments showed that the proposed approach has a fair accuracy in comparison to other related methods.

Determining environmental chemical carcinogenicity is urgently needed as humans are increasingly exposed to these chemicals. In this study, we developed a hybrid neural network (HNN) method called HNN-Cancer to predict potential carcinogens of real-life chemicals. The HNN-Cancer included a new SMILES feature representation method by modifying our previous 3D array representation of 1D SMILES simulated by the convolutional neural network (CNN). We developed binary classification, multiclass classification, and regression models based on diverse non-congeneric chemicals. Along with the HNN-Cancer model, we developed models based on the random forest (RF), bootstrap aggregating (Bagging), and adaptive boosting (AdaBoost) methods for binary and multiclass classification. We developed regression models using HNN-Cancer, RF, support vector regressor (SVR), gradient boosting (GB), kernel ridge (KR), decision tree with AdaBoost (DT), KNeighbors (KN), and a consensus method. The performance of the models for all classifications was assessed using various statistical metrics. The accuracy of the HNN-Cancer, RF, and Bagging models were 74%, and their AUC was ~0.81 for binary classification models developed with 7994 chemicals. The sensitivity was 79.5% and the specificity was 67.3% for the HNN-Cancer, which outperforms the other methods. In the case of multiclass classification models with 1618 chemicals, we obtained the optimal accuracy of 70% with an AUC 0.7 for HNN-Cancer, RF, Bagging, and AdaBoost, respectively. In the case of regression models, the correlation coefficient (R) was around 0.62 for HNN-Cancer and RF higher than the SVM, GB, KR, DTBoost, and NN machine learning methods. Overall, the HNN-Cancer performed better for the majority of the known carcinogen experimental datasets. Further, the predictive performance of HNN-Cancer on diverse chemicals is comparable to the literature-reported models that included similar and less diverse molecules. Our HNN-Cancer could be used in identifying potentially carcinogenic chemicals for a wide variety of chemical classes.

The long non-coding RNAs (lncRNAs) are subject of intensive recent studies due to its association with various human diseases. It is desirable to build the artificial intelligence-based models for prediction of diseases or tissues based on the lncRNAs data, which will be useful in disease diagnosis and therapy. The accuracy and robustness of existing models based on the machine learning techniques are subject to further improvement. In this study, we propose a deep learning model, called Multi-Label Classifications with Deep Forest, termed MLCDForest, to address multi-label classification on tissue prediction for a given lncRNA, which can be regarded as an implementation of the deep forest model in multi-label classification. The MLCDForest is a sequential multi-label-grained scanning method, which distinguishes from the standard deep forest model. It is proposed to train in sequential of multi-labels with label correlation considered. A systematic comparison using the lncRNA-disease association datasets demonstrates that our method consistently shows superior performance over the state-of-the-art methods in disease prediction. Considering label correlation in the sequential multi-label-grained scanning, our model provides a powerful tool to make multi-label classification and tissue prediction based on given lncRNAs.

The theoretical analysis of multiclass classification has proved that the existing multiclass classification methods can train a classifier with high classification accuracy on the test set, when the instances are precise in the training and test sets with same distribution and enough instances can be collected in the training set. However, one limitation with multiclass classification has not been solved: how to improve the classification accuracy of multiclass classification problems when only imprecise observations are available. Hence, in this article, we propose a novel framework to address a new realistic problem called multiclass classification with imprecise observations (MCIMO), where we need to train a classifier with fuzzy-feature observations. First, we give the theoretical analysis of the MCIMO problem based on fuzzy Rademacher complexity. Then, two practical algorithms based on support vector machine and neural networks are constructed to solve the proposed new problem. The experiments on both synthetic and real-world datasets verify the rationality of our theoretical analysis and the efficacy of the proposed algorithms.

In the last decades, the use of ensembles of classifiers in different domains of application has received significant attention of the Machine Learning community. In a typical architecture of ensemble, a new input pattern is presented to all K components. The individual classifiers provide their output and send them to a combination method, which is responsible for providing the final output. Ensemble of classifiers can be applied to any classification problem, single-label or multi- label problems. Multi-label (ML) classification has been received much attention from various research domains, such as text categorization, bioinformatics, computer vision, among others. In this paper, we investigate the use of ensemble of classifiers when applied to multi-label problems, focusing on the use of diversity measures in these systems when applied to multi-label problems. In order to do this, the present work proposes an adapted approach for two well-known diversity measures, good and bad, for ensembles of multi-label classifiers. In order to assess the feasibility of the proposed diversity measures in ensemble of multi-label classifiers, an empirical analysis is conducted, with eight multi-label classification problems. Generally, our finds indicate that there is a strong correlation between performance of the ML ensembles with the proposed diversity measures, showing that these diversity measures can be used as an important tool in the design of efficient ensembles of multi-label classifiers.

Correlation analysis of performance measures for multi-label classification

Multi-label classification has many applications in the text categorization, biology and medical diagnosis, in which multiple class labels can be assigned to each training instance simultaneously. As it is often the case that there are relationships between the labels, extracting the existing relationships between the labels and taking advantage of them during the training or prediction phases can bring about significant improvements. In this paper, we have introduced positive, negative and hybrid relationships between the class labels for the first time and we have proposed a method to extract these relations for a multi-label classification task and consequently, to use them in order to improve the predictions made by a multi-label classifier. We have conducted extensive experiments to assess the effectiveness of the proposed method. The obtained results advocate the merits of the proposed method in improving the multi-label classification results.

News has become a basic human need along with technological and internet developments. This causes the process of disseminating information on the news that switched from print media to the digital era. Another problem that appears when classifying news is multi-label. Multi-label classification is different from single label classification. A single label classification will classify documents into one label only. While multi-label classification can group documents into more than one label. For example, news articles that discuss in detail the early detection of ovarian cancer with a bioinformatics approach may have more than one label such as health, bioinformatics, and women. In this paper, a classification model is developed that can identify classes in each multi-label news article using K-Nearest Neighbor. The advantages of K-Nearest Neighbor are algorithms that are very suitable for multi-label cases; even KNN can be superior to other classifiers. From the system created, the results of the value of system performance as measured by the size of the closeness are the comparison between Manhattan Distance, Euclidean Distance and Supremum Distance using the K = 11 parameters, resulting in a Hamming Loss value of 11.16.%.

This article describes the code that was used in a multi-topic labeling system, the code starts with loading the train, validation, and test data set then installing the pyarabic and Simple Transformers libraries." pyarabic" allows the system to manipulate Arabic letters. "Simple Transformers" is a Natural Language Processing (NLP) library designed to simplify the usage of Transformer models without having to compromise on utility. The model was used from the Simple Transformer library is "Multi Label Classification Model" with the model type "bert" and the model's name "asafaya/bert-base-arabic". In multi-label text classification, the target for a single article (row) from the training dataset is a list of 10 distinct binary labels. A transformer-based multi-label text classification model typically consists of a transformer model with a classification layer on top of it. The results were impressive compared to the short training time for two epochs, which is five minutes and 27 seconds. The accuracy results were as follows: F1 macro: 0.866, F1 micro: 0.869, competition website on Codalab: 0.8468.

Multi-label text classification is an increasingly important field as large amounts of text data are available and extracting relevant information is important in many application contexts. Probabilistic generative models are the basis of a number of popular text mining methods such as Naive Bayes or Latent Dirichlet Allocation. However, Bayesian models for multi-label text classification often are overly complicated to account for label dependencies and skewed label frequencies while at the same time preventing overfitting. To solve this problem we employ the same technique that contributed to the success of deep learning in recent years: greedy layer-wise training. Applying this technique in the supervised setting prevents overfitting and leads to better classification accuracy. The intuition behind this approach is to learn the labels first and subsequently add a more abstract layer to represent dependencies among the labels. This allows using a relatively simple hierarchical topic model which can easily be adapted to the online setting. We show that our method successfully models dependencies online for large-scale multi-label datasets with many labels and improves over the baseline method not modeling dependencies. The same strategy, layer-wise greedy training, also makes the batch variant competitive with existing more complex multi-label topic models.

The goal of human resource management is to select the right people to the right positions, no matter by recruitment, assessment or promotion. To achieve this goal, competency analysis is an effective way. We can obtain the employee’s competency and the position’s requirements by the analysis. The competency analysis also provide a strong intellectual support in the downstream works, such as assessing or promoting employees, or establishing employee files. The multi-label text classification model, which is proposed in this paper based on deep neural network, can successfully complete the competency analysis, and its performance is much better than the current text multi-label classification method. We also construct a multi-label classification dataset in human resource field, which is the first one focused on competency analysis, as far as we know.