Discriminating proteins using a novel ensemble algorithm

An Intelligent Prediction System for Educational Data Mining Based on Ensemble and Filtering approaches

This paper presents a computational bench-marking assessment of Ensemble Learning algorithms in the prediction of heart disease, combining different Machine Learning algorithms, such as hard voting, soft voting, and stacking, in a single framework. The evaluation was conducted using publicly available cardiovascular dataset obtained from the Kaggle repository (https://www.kaggle.com/datasets/sid321axn/heart-statlog-cleveland-hungary-final) comprising 1,190 instances and 11 clinical features. The process involves data preprocessing, which includes handling missing values, removing outliers, scaling variables and class balancing to ensure uniform input feature selection, based on Random Forest (RF), is used to eliminate unnecessary features. Among the evaluated models, the stacking ensemble classifier achieved the highest overall accuracy of 91.88% on the test dataset. Although additional metrics such as precision, recall and F1-score were computed for comparative analysis, the emphasis of this study remains on methodological benchmarking rather than clinical validation. Various base classifiers, including Decision Tree, Random Forest, AdaBoost, and XGBoost, are applied and tested independently. These models are then combined using ensemble techniques with hard voting, soft voting, and stacking. In stacking, Logistic Regression is used as the meta-model, which is trained on cross-validated predictions of the out-of-fold samples to avoid overfitting. Evaluations are carried out using accuracy as the primary criterion for comparison, so that individual classification systems and their combination strategies can be compared uniformly in the same preprocessing and validation environment. Though performance metrics are provided for comparative indications, the emphasis of the approach lies in the development and evaluation of strategies and not in their clinical assessment. This protocol makes it easy to compare ensemble machine learning algorithms on publicly available cardiovascular datasets and helps to make a systematic comparison of data preprocessing and ensemble configuration approaches.

This paper proposes a reduct construction method based on discernibility matrix simplification. The method works with genetic algorithm. To identify potential problems and prevent complete failure of bearings, a new method based on rule-based classifier ensemble is presented. Genetic algorithm is used for feature reduction. The generated rules of the reducts are used to build the candidate base classifiers. Then, several base classifiers are selected according to their diversity and the scale of them. Weights of the selected base classifiers are calculated based on a measure of support rate. The classifier ensemble is constructed by the base classifiers. The accuracy reached 98.44% which is 4.5% higher than that of the three base classifiers.

Network intrusion behaviour data is the imbalanced data. It includes a large amount of normal behavior data and a small amount of intrusion behavior data. The traditional selective ensemble learning algorithm will lead to high false negative rate. This paper proposes a selective ensemble learning algorithm for imbalanced data based on under sampling (SELAUS). First of all, the algorithm uses Bootstrap method to extract samples equal to the number of samples of a few classes from majority class samples to construct multiple balanced training subsets. Then, in order to ensure that the obtained base classifiers have great differences, several features are randomly selected on the training subset and a decision tree is constructed as the base classifier using CART algorithm. This method can also make some base classifiers have poor performance, so it can select and integrate base classifiers instead of all base classifiers. In order to accurately evaluate the generalization error of the classifier for imbalanced dataset, this paper defines the performance evaluation method in the imbalanced dataset and the difference evaluation method between the base classifiers. Then the generalization error of each base classifier is calculated, and the base classifier is selected according to the generalization error. In the integration of weighted voting, the weight of each base classifier is calculated by the weight calculation method for imbalanced data. Finally, the validity of the algorithm is verified by UCI dataset and applied to network intrusion detection. The simulation results show that the algorithm can improve the detection rate of minority class samples, that is to say, reduce the false negative rate.

One of the major developments in machine learning in the past decade is the ensemble method, which finds highly accurate classifier by combining many moderately accurate component classifiers. In this research work, new ensemble classification methods are proposed with homogeneous ensemble classifier using bagging and heterogeneous ensemble classifier using arcing and their performances are analyzed in terms of accuracy. A Classifier ensemble is designed using Radial Basis Function RBF and Support Vector Machine SVM as base classifiers. The feasibility and the benefits of the proposed approaches are demonstrated by the means of standard datasets of direct marketing. The main originality of the proposed approach is based on three main parts: preprocessing phase, classification phase and combining phase. A wide range of comparative experiments are conducted for standard datasets of direct marketing. The performance of the proposed homogeneous and heterogeneous ensemble classifiers are compared to the performance of other standard homogeneous and heterogeneous ensemble methods. The standard homogeneous ensemble methods include Error correcting output codes, dagging and heterogeneous ensemble methods include majority voting, stacking. The proposed ensemble methods provide significant improvement of accuracy compared to individual classifiers and the proposed bagged RBF and SVM performs significantly better than ECOC and Dagging and the proposed hybrid RBF-SVM performs significantly better than voting and stacking. Also heterogeneous models exhibit better results than homogeneous models for standard datasets of direct marketing.

Data mining, and specifically supervised data classification, is a key application area for Learning Classifier Systems (LCS). Scaling to larger classification problems, especially to higher dimensional problems, is a key challenge. Ensemble based approaches can be applied to LCS to address scalability issues. To this end a rough set based ensemble of LCS is proposed, which relies on a pre-processed feature partitioning step to train multiple LCS on feature subspaces. Each base classifier in the ensemble is a Michigan style supervised LCS. The traditional genetic algorithm based rule evolution is replaced by a differential evolution based rule discovery, to improve generalisation capabilities of LCS. A voting mechanism is then used to generate output for test instances. This paper describes the proposed ensemble algorithm in detail, and compares its performance with different versions of base LCS on a number of benchmark classification tasks. Analysis of computational time and model accuracy show the relative merits of the ensemble algorithm and base classifiers on the tested data sets. The rough set based ensemble learning approach and differential evolution based rule searching out-perform the base LCS on classification accuracy over the data sets considered. Results also show that small ensemble size is sufficient to obtain good performance.

Stress is an almost basic human instinct, especially in the internet age. People’s health is being endangered by psychological stress. It is critical to diagnose stress early on to provide proactive therapy. In the age of the Internet, social media has an enormous impact on human thinking. It can cause mental health issues such as stress but can also play an important part in detecting it. Advancements in machine learning and natural language processing have enabled information extraction from a huge amount of raw textual data. In this study machine learning approach is used to detect stress in a social media text. Data used in this study is from Reddit. Classical and Ensemble machine learning approaches were used for stress detection. Classical Techniques include Decision Tree, Logistic Regression, support vector machine, Random Forest, and Na ̈ıve Bayes. Ensemble approaches used are boosting, bagging, and voting. The ensemble approach was able to provide better results than all machine learning baselines on the dataset. Also, it was able to outperform all non-transformed-based neural network architectures discussed in the baseline. The best-performing model in this study is Logistic regression with a 76.6% F1 Score.

Diversity is a key characteristic of a classifier ensemble. A classifier ensemble must be composed of base classifiers with different performance in different areas of the problem space. Several works studied different diversity measures by performing extensive numerical experiments. However, up to our knowledge, no method has been proposed to visualize the diversity of a classifier ensemble. In this contribution, we propose a novel approach to visualize and analyze classifier ensembles diversity. Our novel proposal (called DivGrams), allows the visualization of the diversity between base classifiers in the classifier ensemble by using a network representation. The nodes represent base classifiers and its edges represent diversity relations between the base classifiers. We apply DivGrams to fuzzy rule-based classifier ensembles, as in our previous work these machine learning models proved to deal well with complex classification problems. A preliminary experimental study is performed using two UCI datasets in order to show the usefulness of DivGrams. Furthermore, we show a practical application of DivGrams to develop manual classifier selection on eight UCI datasets. The results obtained show the interesting characteristics of our proposal and support this novel approach in the fuzzy classifier ensemble field.

Many theoretical or experimental studies has shown that ensemble learning is an effective technique to achieve better classification accuracy and stability than individual classifiers. In this paper, we propose a novel weighted classifier ensemble method through classifiers pruning with two stages. In the first stage, we use canonical correlation analysis (CCA) to model maximum correlation relationships between training data points and base classifiers. Based on such globally multi-linear projections, a sparse regression method is proposed to prune base classifiers so that each test data point will dynamically select a subset of classifiers to form a unique classifier ensemble, to decrease effects of noisy input data and incorrect classifiers in such a global view. In the second stage, the pruned classifiers are weighted locally by a fusion method, which utilizes the generalization ability of pruned classifiers among nearest neighbors of testing data points. By this way, each test data point can build a unique locally weighted classifier ensemble. Analysis of experimental results on several UCI data sets shows that the classification results of our method are better than other ensemble methods such as Random Forests, Majority Voting, AdaBoost and DREP.

To predict potential problems and avoid an unexpected breakdown of rotating machinery, a rule-based classifier ensemble approach is presented. Feature reduction was first implemented on a fault decision table using discernibility matrices and the genetic algorithm. The generated rules of the reducts were used to build the candidate base classifiers. Then, several base classifiers were selected according to their diversity and scale. The weights of the selected base classifiers were also calculated based on the support rate measurements. A classifier ensemble was constructed through an integration of the base classifiers using an improved weighted voting technique. Finally, the proposed classifier ensemble was verified based on the vibration data of bearing types SKF6203 and NU205. The accuracy for the SKF6203 bearing type reached 88.75 %, which is at least 5 % higher than that of the three base classifiers for this type of bearing. In addition, the recognition rate for the latter bearing type was 90 %. The reasoning process was much easier to comprehend owing to the semantic descriptions of the rules. The results show that this is a promising and transparent approach for diagnosing typical faults of rotating machinery.

Traditionally, the approach that has been used in the design of pattern classification was to experimentally assess the performance of several classifiers so that the best one would be chosen. Ensemble classifiers is developed to design a such reliable system with improved accuracy. Recognising that each classifier may make different or complementary errors, the idea is to pool the results from all classifiers to find a composite system that would out-perform any individual (base) classifier. In this way, a single complex classifier may be replaced by a set of relatively simple classifiers. Similarly, the idea of ensemble learning is to employ multiple learners and combine their predictions. Most ensemble methods use a single base learning algorithm to produce homogeneous base learners to produce homogeneous ensembles. Some methods use heterogeneous learners to create heterogeneous ensembles. For ensemble methods to be more accurate than any of its members, the base learners must be as accurate and as diverse as possible. Motivated by this ensemble approach, a heterogeneous ensemble model with different algorithms for members' training is proposed. More specifically, three types of classifiers namely the ELM, SVM and RF have been adopted for classification of Malay syllables. In the process of improving ensemble accuracy and stability, different combination methods are explored. These methods vary in their approach to treat the training data, the type of algorithms used, and the combination methods followed. Diversity among the performance of each single classifier in the ensemble classifiers is also analysed and compared with SD, SI, FI, FD, MI and MD configurations. Based on the obtained results, heterogeneous ensemble method has been successfully proven in improving recognition accuracy on the selected Malay syllables.

Educational data mining has illustrated an increasing demand for extracting and maneuvering data from academic backdrop, to generate prolific information which is indispensible for decision making. Therefore in this paper, an attempt has been made to deploy various data mining techniques including base and meta learning classifiers across our pedagogical dataset to foretell the performance of students. Among several contemporary ensemble approaches, researchers have practiced widespread learning classifiers viz. boosting to predict the performance of students. As exploitation of ensemble methods is considered to be significant phenomenon in classification and prediction mechanisms, therefore analogous method (boosting) has been applied across our pedagogical dataset. The entire results have been evaluated with 10-fold cross validation, once pedagogical dataset has been subjected to base classifiers including j48, random tree, naive bayes and knn. In addition, techniques such as oversampling (SMOTE) and undersampling (Spread subsampling) have been employed to further draw a comparison among ensemble classifiers and base classifiers. These methods were exploited with the key objective to observe any improvement in prediction accuracy of students.

SubFeat: Feature subspacing ensemble classifier for function prediction of DNA, RNA and protein sequences

The continuous development of network technologies plays a major role in increasing the utilization of these technologies in many aspects of our lives, including e-commerce, electronic banking, social media, e-health, and e-learning. In recent times, phishing websites have emerged as a major cybersecurity threat. Phishing websites are fake web pages that are created by hackers to mimic the web pages of real websites to deceive people and steal their private information, such as account usernames and passwords. Accurate detection of phishing websites is a challenging problem because it depends on several dynamic factors. Ensemble methods are considered the state-of-the-art solution for many classification tasks. Ensemble learning combines the predictions of several separate classifiers to obtain a higher performance than a single classifier. This paper proposes an intelligent ensemble learning approach for phishing website detection based on weighted soft voting to enhance the detection of phishing websites. First, a base classifier consisting of four heterogeneous machine-learning algorithms was utilized to classify the websites as phishing or legitimate websites. Second, a novel weighted soft voting method based on Kappa statistics was employed to assign greater weights of influence to stronger base learners and lower weights of influence to weaker base learners, and then integrate the results of each classifier based on the soft weighted voting to differentiate between phishing websites and legitimate websites. The experiments were conducted using the publicly available phishing website dataset from the UCI Machine Learning Repository, which consists of 4898 phishing websites and 6157 legitimate websites. The experimental results showed that the suggested intelligent approach for phishing website detection outperformed the base classifiers and soft voting method and achieved the highest accuracy of 95% and an Area Under the Curve (AUC) of 98.8%.

Ensemble algorithms have been a very popular research topic because of their high performances. In this work, performance based ensemble pruning and decision weighting methods are investigated on 3 ensemble algorithms (Bagging, Random Subspaces, Random Forest) over 26 classification datasets. According to our experiments; the algorithm including most diversity among its base learners is Random Subspaces. The best performed ensemble algorithm is Random Subspaces with decision weighting.