Heterogeneous classifier ensembles for EEG-based motor imaginary detection

The object of the study is the process of identifying anomalies in the operation of a computer system (CS). The subject of the study is ensemble methods for identifying the state of the CS. The goal of the study is to improve the performance of ensemble classifiers based on heterogeneous models. Methods used: machine learning methods, homogeneous and heterogeneous ensemble classifiers, Pasting and Bootstrapping technologies. Results obtained: a comparative analysis of the use of homogeneous and heterogeneous bagging ensembles in data classification problems was carried out. The effectiveness of various approaches to the selection of base ensemble classifiers has been studied. A method for identifying the state of a computer system, based on the heterogeneous bagging ensemble was proposed. Experimental studies made it possible to confirm the main theoretical assumptions, as well as evaluate the efficiency of the constructed heterogeneous ensembles. Conclusions. Based on the results of the study, the method for constructing a heterogeneous bagging ensemble classifier, which differs from known methods in the procedure for selecting base models was proposed. It made possible to increase the classification accuracy. Further development of this research could include the creating and integration of dissimilarity metrics as well as other quantitative metrics for a more accurate and balanced base model selection procedure, which would further improve the performance of the computer system state classifier.

One of the major developments in machine learning in the past decade is the ensemble method, which finds highly accurate classifier by combining many moderately accurate component classifiers. In this research work, new ensemble classification methods are proposed with homogeneous ensemble classifier using bagging and heterogeneous ensemble classifier using arcing and their performances are analyzed in terms of accuracy. A Classifier ensemble is designed using Radial Basis Function RBF and Support Vector Machine SVM as base classifiers. The feasibility and the benefits of the proposed approaches are demonstrated by the means of standard datasets of direct marketing. The main originality of the proposed approach is based on three main parts: preprocessing phase, classification phase and combining phase. A wide range of comparative experiments are conducted for standard datasets of direct marketing. The performance of the proposed homogeneous and heterogeneous ensemble classifiers are compared to the performance of other standard homogeneous and heterogeneous ensemble methods. The standard homogeneous ensemble methods include Error correcting output codes, dagging and heterogeneous ensemble methods include majority voting, stacking. The proposed ensemble methods provide significant improvement of accuracy compared to individual classifiers and the proposed bagged RBF and SVM performs significantly better than ECOC and Dagging and the proposed hybrid RBF-SVM performs significantly better than voting and stacking. Also heterogeneous models exhibit better results than homogeneous models for standard datasets of direct marketing.

Background/Objectives: Recent studies emphasized on using ensemble models over single ones to solve credit scoring problems. The objective of this study is to build a heterogeneous ensemble classifier model with an improved classification accuracy. Methods: This study focuses on developing a heterogeneous ensemble classifier using Logistic Regression, K-nearest neighbor, Decision tree, Random Forest, Naïve Base and Support vector machine as base classifiers and Random Forest, Logistic Regression and Support vector machine as meta-classifiers. The proposed model is built using these six base classifiers for ensemble aggregation. A feature selection algorithm based on the random forest technique is used for selecting the best features. A stacking and voting method are used for building ensemble model. Findings: The ensemble classifier gives superior predictive performance than single classifiers SVM, DT, RF, NB, KNN and LR with an accuracy of 91.56% for Australian dataset and 84.35% for German dataset. Novelty: The proposed model uses stacking and majority voting method for ensemble classification. Initially, stacking is applied to the base classifiers. This is done in two levels. First the training dataset is split into 10 folds for cross validation. The output of each classifier is taken, and the dataset is updated with the meta-features. In the second level, three meta-classifiers (MC), namely LR, SVM and RF are used. Majority voting is applied to the output of these meta-classifiers for the prediction. Keywords: Credit scoring; ensemble model; SVM; DT; RF; NB; KNN; LR

In recent years, ensemble classification methods have been widely investigated in both industry and literature in the field of machine learning and artificial intelligence. The main advantage of this approach is to benefit from a set of classifiers instead of using a single classifier with the aim of improving the prediction performance, such as accuracy. Selecting the base classifiers and the method for combining them are the most challenging issues in the ensemble classifiers. In this paper, we propose a heterogeneous dynamic ensemble classifier (HDEC) which uses multiple classification algorithms. The main advantage of using heterogeneous algorithms is increasing the diversity among the base classifiers as it is a key point for an ensemble system to be successful. In this method, we first train many classifiers with the original data. Then, they are separated based on their strength in recognizing either positive or negative instances. For doing this, we consider the true positive rate and true negative rate, respectively. In the next step, the classifiers are categorized into two groups according to their efficiency in the mentioned measures. Finally, the outputs of the two groups are compared with each other to generate the final prediction. For evaluating the proposed approach, it has been applied to 12 datasets from the UCI and LIBSVM repositories and calculated two popular prediction performance metrics, including accuracy and geometric mean. The experimental results show the superiority of the proposed approach in comparison to other state-of-the-art methods.

Using electroencephalogram (EEG) signals to control intelligent wheelchairs is one of the new control methods for controlling intelligent wheelchairs. The most serious problem in the control process is recognition and classification of EEG signals, especially a variety of EEG signals. To improve the accuracy of EEG classification, improved-common spatial patterns and a deep-belief network algorithm are proposed and used for recognition and classification of EEG signals. A variety of different EEG signals were collected by an Emotiv EPOC+ EEG collector, and the characteristic signals were extracted by improved-common spatial patterns and identified and classified by a deep-belief network algorithm. Simulation results show that the classification accuracy of improved-common spatial patterns and the deep-belief network algorithm is higher than that of the traditional classification method, and a research perspective is provided for the classification of various EEG signals in the future.

Predictive models, which leverage the relationship between environmental variables and river health, serve as a valuable tool for predicting the river health at unmonitored sites. Such models should be generalizable to unseen data. However, predictions derived from machine learning (ML) models can exhibit large variability even with minor changes in the training dataset. The potentially unstable behaviors of a ML model decrease the model’s generalizability to unseen data, likely limiting its applicability as an assistant tool for decision making. Heterogeneous ensemble models are recognized to achieve greater generalizability compared to single models owing to their structural diversity. In this study, various machine learning (ML) models are employed to understand the relationship between environmental factors and benthic macroinvertebrate health. To obtain a model with better generalizability, the present study compares the generalizability of heterogeneous ensembles with those of homogeneous ensembles and single models by using the bias–variance decomposition. The models classified five grades (very good to very poor) of benthic macroinvertebrate index (BMI). The models incorporated diverse environmental factors, including water quality, hydrology, meteorological conditions, land cover, and stream properties, as input variables. The data were monitored at 2,915 sites in the four major river watersheds in South Korea during the 2016–2021 period. The results indicated better generalizability of the heterogeneous and homogeneous ensembles than single models. Moreover, heterogeneous ensembles tended to show higher generalizability than homogeneous ensembles, although the differences were marginal. Weighted soft voting was the most generalizable of the heterogeneous ensembles, with loss of 0.49. Weighted soft voting also delivered acceptable classification performance on the test set, with accuracy of 0.52. The identified contributions of the environmental factors to BMI predictions and the directions of their effects agreed with established knowledge, confirming the reliability of the predictions. These results demonstrate the usefulness of the heterogeneous ensemble models for increasing the generalizability of ML model predictions. Furthermore, despite the slightly lower generalizability than voting-based ensembles, homogeneous ensembles demonstrated comparable levels of generalizability to heterogeneous ensembles.

Predicting stock returns by classifier ensembles

EEG signal classification is an important task to build an accurate Brain Computer Interface (BCI) system. Many machine learning and deep learning approaches have been used to classify EEG signals. Besides, many studies have involved the time and frequency domain features to classify EEG signals. On the other hand, a very limited number of studies combine the spatial and temporal dimensions of the EEG signal. Brain dynamics are very complex across different mental tasks, thus it is difficult to design efficient algorithms with features based on prior knowledge. Therefore, in this study, we utilized the 2D AlexNet Convolutional Neural Network (CNN) to learn EEG features across different mental tasks without prior knowledge. First, this study adds spatial and temporal dimensions of EEG signals to a 2D EEG topographic map. Second, topographic maps at different time indices were cascaded to populate a 2D image for a given time window. Finally, the topographic maps enabled the AlexNet to learn features from the spatial and temporal dimensions of the brain signals. The classification performance was obtained by the proposed method on a multiclass dataset from BCI Competition IV dataset 2a. The proposed system obtained an average classification accuracy of 81.09%, outperforming the previous state-of-the-art methods by a margin of 4% for the same dataset. The results showed that converting the EEG classification problem from a (1D) time series to a (2D) image classification problem improves the classification accuracy for BCI systems. Also, our EEG topographic maps enabled CNN to learn subtle features from spatial and temporal dimensions, which better represent mental tasks than individual time or frequency domain features.

Feature extraction and classification is a subject of broad and current interest in the brain-computer interfaces (BCIs) community, and remains a challenging task when working with Electroencephalogram (EEG) data, with no agreed upon optimum features set and classifier algorithm. In this 75-participant lab study, we compare different feature extraction methods and classifiers as we investigate the relationship between users' perceptions of the memorability of a number of passwords and the users' EEG data collected using BCIs when presented with these passwords. We asked the participants to rank the comparative memorability of 15 blocks of 5 passwords each, while recording their EEG data. Features from the EEG signals are extracted in three domains, power spectrum from the frequency domain, statistics from the time domain, and wavelet coefficients from the time-frequency domain. The feature subsets are submitted for classification with two classes, most memorable and least memorable, based on the user's perceived memorability of the passwords. Classification performance of Neural Networks and Support Vector Machine algorithms are compared. Results show distinctive features of EEG signals in the two different classes, achieving a classification accuracy of 89%. Our results indicate that features extracted in the time-frequency domain using wavelet transform, and classified using neural networks, resulted in the highest classification performance.

Electroencephalogram (EEG) is the main diagnostic tool for the monitoring, diagnosis and treatment of epilepsy which is a neurological disorder. EEG signals can disrupt easily by involuntary movements that are called artifact contaminants such as blinking, coughing. In this study, the artifacts in the extrac- and intracranial EEG signals have been cancelled out from the EEG with the use of Kaiser window based Finite Impulse Response (FIR) filter. The most important features in the EEG signals have been selected by the Principle Component Analysis (PCA) method. The selected features have been classified by applying ensemble learning methods that are Boosting, Bagging and Random Subspace. The aim of this study is to increase the extrac- and intracranial EEG signal classification by calculating window spectral parameters. The algorithms' classification performances have been compared in terms of accuracy rates, sensitivities, specificities, prediction rates and training times according to the 5 × 5 cross validation. Subspace KNN algorithm, as revealed by results, is higher than the other algorithms' classification performances.

Impact of the learners diversity and combination method on the generation of heterogeneous classifier ensembles

Non-parametric classification procedures based on a certainty measure and nearest neighbour rule for motor unit potential classification (MUP) during electromyographic (EMG) signal decomposition were explored. A diversity-based classifier fusion approach is developed and evaluated to achieve improved classification performance. The developed system allows the construction of a set of non-parametric base classifiers and then automatically chooses, from the pool of base classifiers, subsets of classifiers to form candidate classifier ensembles. The system selects the classifier ensemble members by exploiting a diversity measure for selecting classifier teams. The kappa statistic is used as the diversity measure to estimate the level of agreement between base classifier outputs, i.e., to measure the degree of decision similarity between base classifiers. The pool of base classifiers consists of two kinds of classifiers: adaptive certainty-based classifiers (ACCs) and adaptive fuzzy k-NN classifiers (AFNNCs) and both utilize different types of features. Once the patterns are assigned to their classes, by the classifier fusion system, firing pattern consistency statistics for each class are calculated to detect classification errors in an adaptive fashion. Performance of the developed system was evaluated using real and simulated EMG signals and was compared with the performance of the constituent base classifiers and the performance of the fixed ensemble containing the full set of base classifiers. Across the EMG signal data sets used, the diversity-based classifier fusion approach had better average classification performance overall, especially in terms of reducing classification errors.

Traditionally, the approach that has been used in the design of pattern classification was to experimentally assess the performance of several classifiers so that the best one would be chosen. Ensemble classifiers is developed to design a such reliable system with improved accuracy. Recognising that each classifier may make different or complementary errors, the idea is to pool the results from all classifiers to find a composite system that would out-perform any individual (base) classifier. In this way, a single complex classifier may be replaced by a set of relatively simple classifiers. Similarly, the idea of ensemble learning is to employ multiple learners and combine their predictions. Most ensemble methods use a single base learning algorithm to produce homogeneous base learners to produce homogeneous ensembles. Some methods use heterogeneous learners to create heterogeneous ensembles. For ensemble methods to be more accurate than any of its members, the base learners must be as accurate and as diverse as possible. Motivated by this ensemble approach, a heterogeneous ensemble model with different algorithms for members' training is proposed. More specifically, three types of classifiers namely the ELM, SVM and RF have been adopted for classification of Malay syllables. In the process of improving ensemble accuracy and stability, different combination methods are explored. These methods vary in their approach to treat the training data, the type of algorithms used, and the combination methods followed. Diversity among the performance of each single classifier in the ensemble classifiers is also analysed and compared with SD, SI, FI, FD, MI and MD configurations. Based on the obtained results, heterogeneous ensemble method has been successfully proven in improving recognition accuracy on the selected Malay syllables.

In recent years, Deep Neural Networks (DNNs) have achieved excellent performance on many tasks, but it is very difficult to train good models from imbalanced datasets. Creating balanced batches either by majority data down-sampling or by minority data up-sampling can solve the problem in certain cases. However, it may lead to learning process instability and overfitting. In this paper, we propose the Batch Balance Wrapper (BBW), a novel framework which can adapt a general DNN to be well trained from extremely imbalanced datasets with few minority samples. In BBW, two extra network layers are added to the start of a DNN. The layers prevent overfitting of minority samples and improve the expressiveness of the sample distribution of minority samples. Furthermore, Batch Balance (BB), a class-based sampling algorithm, is proposed to make sure the samples in each batch are always balanced during the learning process. We test BBW on three well-known extremely imbalanced datasets with few minority samples. The maximum imbalance ratio reaches 1167:1 with only 16 positive samples. Compared with existing approaches, BBW achieves better classification performance. In addition, BBW-wrapped DNNs are 16.39 times faster, relative to unwrapped DNNs. Moreover, BBW does not require data preprocessing or additional hyper-parameter tuning, operations that may require additional processing time. The experiments prove that BBW can be applied to common applications of extremely imbalanced data with few minority samples, such as the classification of EEG signals, medical images and so on.

Alzheimer's disease (AD), a psychiatric problem, availing between those who are 65 and older. Additionally, the disease's steady development of a variety of visible and invisible symptoms, such as irritability and aggression, has a substantial negative impact on a patient's overall quality of life. Although many treatments have been developed to help reduce its symptoms, AD has no known cure. As a result, the field of AD management is growing, and a comprehensive framework for the early detection of AD must be created. In this study, we created three classification models for predicting AD using machine learning and five models for predicting AD using ensemble learning. SVM, DTs, and RF are the three basis classifiers employed in the current work. Five ensemble classifiers, XGBoost, Voting Classifier, Extra Trees (ETs) Classifier, Gradient Boost, and AdaBoost, are then thoroughly compared. After thoroughly inspecting the dataset for outliers or other noise, a feature selection method known as PCA and several preprocessing techniques are used to lessen the issue of overfitting and performance enhancement. Additionally, this study utilised the longitudinal information from the OASIS website, which included 150 patients overall, 72 of whom were not demented and 78 of whom were. The RF model, which had an accuracy of 83.92% compared to the other two base classifiers, provided the best classification performance, while the ETs Classifier, an ensemble classifier, performed the best when compared to base and ensemble classifiers, with an accuracy of 86.60%.