- Conference Article
- 10.1109/ijcnn55064.2022.9891892
Localized Feature Ranking approach for Multi-Target Regression
- Jul 18, 2022
- Hugo Bertrand + 4 more +4
International audience
The prediction of multiple numeric outputs at the same time is called multi-target regression (MTR), and it has gained attention during the last decades. This task is a challenging research topic in supervised learning because it poses additional difficulties to traditional single-target regression (STR), and many real-world problems involve the prediction of multiple targets at once. One of the most successful approaches to deal with MTR, although not the only one, consists in transforming the problem in several STR problems, whose outputs will be combined building up the MTR output. In this paper, the Rotation Forest ensemble method, previously proposed for single-label classification and single-target regression, is adapted to MTR tasks and tested with several regressors and data sets. Our proposal rotates the input space in an efficient and novel fashion, avoiding extra rotations forced by MTR problem decomposition. Four approaches for MTR are used: single-target (ST), stacked-single target (SST), Ensembles of Regressor Chains (ERC), and Multi-target Regression via Quantization (MRQ). For assessing the benefits of the proposal, a thorough experimentation with 28 MTR data sets and statistical tests are used, concluding that Rotation Forest, adapted by means of these approaches, outperforms other popular ensembles, such as Bagging and Random Forest.
Localized Feature Ranking approach for Multi-Target Regression
International audience
Towards Meta-Learning for Multi-Target Regression Problems
Several multi-target regression methods were developed in the last years aiming at improving predictive performance by exploring inter-target correlation within the problem. However, none of these methods outperforms the others for all problems. This motivates the development of automatic approaches to recommend the most suitable multi-target regression method. In this paper, we propose a meta-learning system to recommend the best predictive method for a given multi-target regression problem. We performed experiments with a meta-dataset generated by a total of 648 synthetic datasets. These datasets were created to explore distinct inter-targets characteristics toward recommending the most promising method. In experiments, we evaluated four different algorithms with different biases as meta-learners. Our meta-dataset is composed of 58 meta-features, based on: statistical information, correlation characteristics, linear landmarking, from the distribution and smoothness of the data, and has four different meta-labels. Results showed that induced meta-models were able to recommend the best method for different base level datasets with a balanced accuracy superior to 70% using a Random Forest meta-model, which statistically outperformed the meta-learning baselines.
Read moreA data-driven intelligent decision support system that combines predictive and prescriptive analytics for the design of new textile fabrics
In this paper, we propose an Intelligent Decision Support System (IDSS) for the design of new textile fabrics. The IDSS uses predictive analytics to estimate fabric properties (e.g., elasticity) and composition values (% cotton) and then prescriptive techniques to optimize the fabric design inputs that feed the predictive models (e.g., types of yarns used). Using thousands of data records from a Portuguese textile company, we compared two distinct Machine Learning (ML) predictive approaches: Single-Target Regression (STR), via an Automated ML (AutoML) tool, and Multi-target Regression, via a deep learning Artificial Neural Network. For the prescriptive analytics, we compared two Evolutionary Multi-objective Optimization (EMO) methods (NSGA-II and R-NSGA-II) when optimizing 100 new fabrics, aiming to simultaneously minimize the physical property predictive error and the distance of the optimized values when compared with the learned input space. The two EMO methods were applied to design of 100 new fabrics. Overall, the STR approach provided the best results for both prediction tasks, with Normalized Mean Absolute Error values that range from 4% (weft elasticity) to 11% (pilling) in terms of the fabric properties and a textile composition classification accuracy of 87% when adopting a small tolerance of 0.01 for predicting the percentages of six types of fibers (e.g., cotton). As for the prescriptive results, they favored the R-NSGA-II EMO method, which tends to select Pareto curves that are associated with an average 11% predictive error and 16% distance.
Read morePrivileged Multi-Target Support Vector Regression
Multi-target regression is the problem where each instance is associated with multiple continuous target outputs simultaneously. Its major challenges arise with jointly exploring the complex input-output relationships and inter-target correlations. One representative approach is to build many independent single-target Support Vector Regression (SVR) models for each output target which can capture complex input-output relationships via the kernel trick. However, it does not involve inter-target correlations to improve the performance. Meanwhile, there are also many regularization-based methods which mainly explore the linear inter-target correlations, e.g., a low-rank constraint on the parameter matrix. However, in practice, it might be restrictive to assume the targets to be linearly related, and allowing for nonlinear relationships is a challenge. Motivated by Learning Using Privileged Information (LUPI), we propose a novel privileged multi-target support vector regression (MT-PSVR) model which can jointly explore the complex input-output relationships and nonlinear inter-target correlations. It explicitly explores inter-target correlations by viewing other targets as privileged information when training each target model. Besides, it can naturally use the kernel trick to explore both the complex input-output relationships and nonlinear inter-target correlations. Experimental results on many benchmark datasets validate the effectiveness of our approach.
Read moreReliability Estimation of Individual Multi-target Regression Predictions
To estimate the quality of the induced predictive model we generally use measures of averaged prediction accuracy, such as the relative mean squared error on test data. Such evaluation fails to provide local information about reliability of individual predictions, which can be important in risk-sensitive fields (medicine, finance, industry etc.). Related work presented several ways for computing individual prediction reliability estimates for single-target regression models, but has not considered their use with multi-target regression models that predict a vector of independent target variables. In this paper we adapt the existing single-target reliability estimates to multi-target models. In this way we try to design reliability estimates, which can estimate the prediction errors without knowing true prediction errors, for multi-target regression algorithms, as well. We approach this in two ways: by aggregating reliability estimates for individual target components, and by generalizing the existing reliability estimates to higher number of dimensions. The results revealed favorable performance of the reliability estimates that are based on bagging variance and local cross-validation approaches. The results are consistent with the related work in single-target reliability estimates and provide a support for multi-target decision making.
Read moreMulti-target regression from high-speed data streams with adaptive model rules
Many real life prediction problems involve predicting a structured output. Multi-target regression is an instance of structured output prediction whose task is to predict for multiple target variables. Structured output algorithms are usually computationally and memory demanding, hence are not suited for dealing with massive amounts of data. Most of these algorithms can be categorized as local or global methods. Local methods produce individual models for each output component and combine them to produce the structured prediction. Global methods adapt traditional learning algorithms to predict the output structure as a whole. We propose the first rule-based algorithm for solving multi-target regression problems from data streams. The algorithm builds on the adaptive model rules framework. In contrast to the majority of the structured output predictors, this particular algorithm does not fall into the local and global categories. Instead, each rule specializes on related subsets of the output attributes. To evaluate the performance of the proposed algorithm, two other rule-based algorithms were developed, one using the local strategy and the other using the global strategy. These methods were compared considering their prediction error, memory usage, computational time, and model complexity. Experimental results on synthetic and real data show that the local-strategy algorithm usually obtains the lowest error. However, the proposed and the global-strategy algorithms use much less memory and run significantly much faster at the cost of a slightly increase in the error, which make them very attractive when computation resources are an important factor. Also, the models produced by the latter approaches are much easier to understand since considerably less rules are produced.
Read moreAn empirical comparison on multi-target regression learning
Multi-target regression is concerned with the simultaneous prediction of multiple continuous target variables based on the same set of input variables. It has received relatively small attention from the Machine Learning community. However, multi-target regression exists in many real-world applications. In this paper we conduct extensive experiments to investigate the performance of three representative multi-target regression learning algorithms (i.e., multi-target stacking (MTS), random linear target combination (RLTC), and multi-objective random forest (MORF)), comparing the baseline single-target learning. Our experimental results show that all three multi-target regression learning algorithms do improve the performance of the single-target learning. Among them, MTS performs the best, followed by RLTC, followed by MORF. However, the single-target learning sometimes still performs very well, even the best. This analysis sheds the light on multi-target regression learning and indicates that the single-target learning is a competitive baseline for multi-target regression learning on multi-target domains.
Read moreSemi-supervised trees for multi-target regression
Semi-supervised trees for multi-target regression
Detection of congestive heart failure from short-term heart rate variability segments using hybrid feature selection approach
Detection of congestive heart failure from short-term heart rate variability segments using hybrid feature selection approach
Read moreSpectral–Spatial Classification for Hyperspectral Data Using Rotation Forests With Local Feature Extraction and Markov Random Fields
In this paper, we propose a new spectral-spatial classification strategy to enhance the classification performances obtained on hyperspectral images by integrating rotation forests and Markov random fields (MRFs). First, rotation forests are performed to obtain the class probabilities based on spectral information. Rotation forests create diverse base learners using feature extraction and subset features. The feature set is randomly divided into several disjoint subsets; then, feature extraction is performed separately on each subset, and a new set of linear extracted features is obtained. The base learner is trained with this set. An ensemble of classifiers is constructed by repeating these steps several times. The weak classifier of hyperspectral data, classification and regression tree (CART), is selected as the base classifier because it is unstable, fast, and sensitive to rotations of the axes. In this case, small changes in the training data of CART lead to a large change in the results, generating high diversity within the ensemble. Four feature extraction methods, including principal component analysis (PCA), neighborhood preserving embedding (NPE), linear local tangent space alignment (LLTSA), and linearity preserving projection (LPP), are used in rotation forests. Second, spatial contextual information, which is modeled by MRF prior, is used to refine the classification results obtained from the rotation forests by solving a maximum a posteriori problem using the α-expansion graph cuts optimization method. Experimental results, conducted on three hyperspectral data with different resolutions and different contexts, reveal that rotation forest ensembles are competitive with other strong supervised classification methods, such as support vector machines. Rotation forests with local feature extraction methods, including NPE, LLTSA, and LPP, can lead to higher classification accuracies than that achieved by PCA. With the help of MRF, the proposed algorithms can improve the classification accuracies significantly, confirming the importance of spatial contextual information in hyperspectral spectral-spatial classification.
Read moreRotation Forest Ensemble Classifier to Improve the Cardiovascular Disease Risk Prediction Accuracy
Heart disease risk prediction is very important as it is one of the primary causes of sudden death in the world. Early-stage prediction can save the lives by undergoing appropriate diagnosis steps or making necessary changes in their lifestyles. Recent studies have focused on the use of data mining and machine learning in the detection of diseases based on specific features of a person. The Rotation Forest, a tree-based ensemble classifier that uses Principal Component Analysis for feature extraction, is proposed to improve the prediction accuracy of heart disease risk. The Statlog heart dataset has been selected from the publicly available UCI machine learning repository in this research work. The dataset was trained with a Rotation Forest ensemble classifier with default base classifier J48, and then, Random Forest on full features and selected features obtained from One Rule and Support Vector Machines attribute evaluators. The performance of the Rotation Forest was compared with the standard machine learning classifiers, Naïve Bayes, Logistic Regression, Support Vector Machines, K-Nearest Neighbors, AdaBoostM1, and Bagging. The Rotation Forest algorithm with Random Forest provided the highest accuracy of 94.44% and area under the ROC curve 0.980 on selected features of the Statlog dataset from the One Rule method.
Read moreRotation Double Random Forest Algorithm to Predict The Food Insecurity Status of Households
The ensemble tree method has been proven to handle classification problems well. The strength of the ensemble tree technique lies in the diversity and independence between each tree. Increasing the diversity of mutually independent decision trees improves the performance of the model. Various studies propose the development of ensemble tree-based models by forming algorithms that create decision trees that are formed independently of each other and have various inputs. These include random forest (RF), rotation forest (RoF), double random forest (DRF), and the latest is rotation double random forest (RoDRF). RoDRF rotates or transforms data with the intent of producing better diversity among the learning base. RoDRF applies the concept of variable rotation to trees based on the DRF algorithm. Random rotations or transformations on different feature subspaces produce different projections, leading to better generalization or prediction performance. This research aims to compare the performance of RoDRF with the RF, RoF, and DRF models on unbalanced data in cases of food insecurity. Class imbalance will be handled with two methods, namely EasyEnsemble and SMOTE-NC. The research results show that the DRF's model with EasyEnsemble techniques produces a model with the best performance among several algorithms tested. Although the resulting precision is 0.62274 and the AUC value is 0.68501, the model can predict each class equally. All algorithms with EasyEnsemble treatment have average AUC values significantly different from each other based on statistical test results. This research also used SHAP to explain variables that significantly contribute to the household's food insecurity status model.
Read moreVMFS: A VIKOR-based multi-target feature selection
VMFS: A VIKOR-based multi-target feature selection
Feature ranking for semi-supervised learning
The data used for analysis are becoming increasingly complex along several directions: high dimensionality, number of examples and availability of labels for the examples. This poses a variety of challenges for the existing machine learning methods, related to analyzing datasets with a large number of examples that are described in a high-dimensional space, where not all examples have labels provided. For example, when investigating the toxicity of chemical compounds, there are many compounds available that can be described with information-rich high-dimensional representations, but not all of the compounds have information on their toxicity. To address these challenges, we propose methods for semi-supervised learning (SSL) of feature rankings. The feature rankings are learned in the context of classification and regression, as well as in the context of structured output prediction (multi-label classification, MLC, hierarchical multi-label classification, HMLC and multi-target regression, MTR) tasks. This is the first work that treats the task of feature ranking uniformly across various tasks of semi-supervised structured output prediction. To the best of our knowledge, it is also the first work on SSL of feature rankings for the tasks of HMLC and MTR. More specifically, we propose two approaches—based on predictive clustering tree ensembles and the Relief family of algorithms—and evaluate their performance across 38 benchmark datasets. The extensive evaluation reveals that rankings based on Random Forest ensembles perform the best for classification tasks (incl. MLC and HMLC tasks) and are the fastest for all tasks, while ensembles based on extremely randomized trees work best for the regression tasks. Semi-supervised feature rankings outperform their supervised counterparts across the majority of datasets for all of the different tasks, showing the benefit of using unlabeled in addition to labeled data.
Read moreImbalanced Multi-Class Classification of Hyperspectral Image Based on Smote and Deep Rotation Forest
In this paper, a novel Synthetic Minority Oversampling Technique based Deep Rotation Forest(SMOTE-DRoF) algorithm is proposed for the classification of imbalanced hyperspectral image data. It builds a multi -level forests cascade model by training a balanced dataset generated by SMOTE. In this model, each level of the random forest produces misclassification information of the data which are used as guidance information to adjust the sample weight adaptively for the next level. Experiment results on the hyperspectral image Indian Pines AVRIS and University of Pavia ROSIS demonstrate that the proposed method can get better performance than support vector machine, random forest, rotation forest, SMOTE combined random forest, and SMOTE combined rotation forest in imbalance learning.
Read more