Two-stage rule extraction method based on tree ensemble model for interpretable loan evaluation

DCLF: A divide-and-conquer learning framework for the predictions of steel hardness using multiple alloy datasets

A key element in solving real-life data science problems is selecting the types of models to use. Tree ensemble models (such as XGBoost) are usually recommended for classification and regression problems with tabular data. However, several deep learning models for tabular data have recently been proposed, claiming to outperform XGBoost for some use cases. This paper explores whether these deep models should be a recommended option for tabular data by rigorously comparing the new deep models to XGBoost on various datasets. In addition to systematically comparing their performance, we consider the tuning and computation they require. Our study shows that XGBoost outperforms these deep models across the datasets, including the datasets used in the papers that proposed the deep models. We also demonstrate that XGBoost requires much less tuning. On the positive side, we show that an ensemble of deep models and XGBoost performs better on these datasets than XGBoost alone.

Ensemble models such as gradient boosting and random forests are proven to offer the best predictive performance on a wide variety of supervised learning problems. The high performance of these black box models, however, comes at a cost of model interpretability. They are also inadequate to meet regulatory demands and explainability needs of organizations. The model interpretability in high performance black-box models is achieved with the help of post-hoc explainable models such as Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP). This paper presents an alternate intrinsic classifier model that extracts a class of higher order patterns and embeds them into an interpretable learning model. More specifically, the proposed model extracts novel High Utility Gain (HUG) patterns that capture higher order interactions, transforms the model input data into a new space, and applies interpretable classifier methods on the transformed space. We conduct rigorous experiments on forty benchmark binary and multi-class classification datasets to evaluate the proposed model against the state-of-the-art ensemble and interpretable classifier models. The proposed model was comprehensively assessed on three key dimensions: 1) quality of predictions using classifier measures such as accuracy, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$F_{1}$ </tex-math></inline-formula>, AUC, H-measure, and logistic loss, 2) computational performance on large and high-dimensional data, and 3) interpretability aspects. The HUG-based learning model was found to deliver performance comparable to that of the state-of-the-art ensemble models. Our model was also found to achieve 2-40% (45%) prediction quality (interpretability) improvements with significantly lower computational requirements over other interpretable classifier models. Furthermore, we present case studies in finance and healthcare domains and generate one- and two-dimensional HUG profiles to illustrate the interpretability aspects of our HUG models. The proposed solution offers an alternate approach to build high performance and transparent machine learning classifier models. We hope that our ML solution help organizations meet their growing regulatory and explainability needs.

In this paper, we suggest an explainable machine learning approach to model the claim frequency of a telematics car dataset. In fact, we use a data-driven method based on tree ensembles, namely, the random forest, to create a claim frequency model. Then, we present a method to build a tree that faithfully synthesizes the predictions of a tree ensemble model such as those derived from the random forest or gradient boosting. This tree serves as a global explanation of the predictions of the black-box. Thanks to this surrogate model, we can extract knowledge from a black-box tree ensemble model. Then, we provide an application to improve the performance of a generalized linear model. Indeed, we integrate this new knowledge into a generalized linear model to increase the predictive power.

Sensor-based cattle monitoring is a growing field within precision livestock farming, where machine learning algorithms are used to detect health issues, monitor behavior, and improve productivity. However, the diversity of evaluation strategies, performance metrics, and algorithmic choices in the current literature complicates comparison and selection of machine learning models. In this study, we conduct a systematic review and quantitative analysis of machine learning approaches applied to sensor-driven cattle monitoring tasks. We evaluate and compare algorithms using a broad spectrum of performance indicators, including accuracy, F1-score, precision, and recall, alongside their variability across studies. Through detailed heatmap visualizations and statistical summaries (mean, median, standard deviation, coefficient of variation), we identify the most consistently high-performing models and the factors contributing to variability. Results show that Convolutional Neural Networks and Extreme Gradient Boosting achieve the highest F1-scores and most balanced precision-recall performance, especially in imbalanced health detection tasks, while tree ensemble models such as Random Forest and Gradient Boosting demonstrate superior accuracy and consistency for behavior classification on balanced datasets. Our findings emphasize the importance of multi-metric evaluation frameworks and context-aware algorithm selection in the development of scalable and reliable livestock monitoring systems.

As COVID-19 crisis is settling down in countries, whether or not a person has been affected personally by the disease, he fights with issues such as anxiety, panic attacks, grief, low mood, and many other psychotic disorders. Mental fitness is one of the major strengths in the development of the individual. Development of social sites turns out to be one platform where the person feels free to vent out their thoughts and to easily interact with people. Extracting useful information from those posts is a part of sentimental analysis, which is the technique of machine learning that helps to know the mental condition of the individual. In this paper, various machine learning algorithms such as random forest, Naive Bayes, decision tree, multilayer perceptron, maximum entropy, KNN, gradient boosted decision tree, adaptive boosting, bagged logistic regression, tree ensemble model, Liblinear, convolutional neural network, and long short-term memory are applied on the dataset, and different mathematical scales such as accuracy, precision, recall, and F1 score concluded that bagged logistic regression has given the better accuracy results.KeywordsDepressionSentimental analysisMental healthSocial sitesDeep learningMachine learningBagged logistic regression

This study provides a comprehensive benchmarking of traditionalsystem identification and modern machine learning (ML) models forthe data-driven modeling of dynamical systems, with a focus on processsystems engineering (PSE) applications. To achieve this, we deployAutoSID, an automated end-to-end framework inspired by Machine LearningOperations (MLOps) principles. While AutoSID facilitates model selection,training, validation, and evaluation, its purpose here is to serveas a platform to investigate how well MLOps-inspired tools can beadapted for system identification tasks in PSE. Our investigationincludes a comparison of 12 diverse model architectures from the systemidentification, machine learning, and deep learning literature, evaluatedacross 11 PSE case studies under varying data regimes. We employ fourmodel search or hyperparameter optimization algorithms and three modelselection criteria to ensure a thorough assessment. Our findings highlightthe importance of model selection as the crucial step in system identification.Specifically, our results demonstrate the effectiveness of Bayesianoptimization with tree-structured parzen estimators (TPE) for balancedmodel selection, while k-fold cross-validation proves to be a robustmetric for performance evaluation during the selection process. Inlarge-scale data scenarios, where performance differences between k-fold cross-validation and information criteria are small,information criteria emerge as a computationally efficient alternative.Once the “best” model structure is decided, in termsof model performance, we find that ML models with balanced complexity,such as tree ensemble models, consistently achieve superior predictiveaccuracy and computational efficiency, outperforming both simplisticand overly complex models. These findings provide actionable insightsinto model selection and performance evaluation for PSE practitionersand demonstrate the potential of incorporating MLOps-inspired workflowsinto the system identification process.

One of the main challenges in support vector machine (SVM) for data mining applications is to obtain explicit knowledge from the solutions of SVM for explaining classification decisions. This paper exploits the fact that the decisions from a non-linear SVM could be decoded into linguistic rules based on the information provided by support vectors and its decision function. Given a support vector of a certain class, cross points between each line, which is extended from the support vector along each axis, and SVM decision hyper-curve are searched first. A hyper-rectangular rule is derived from these cross points. The hyper-rectangle is tuned by a tuning phase in order to exclude those out-class data points. Finally, redundant rules are merged to produce a compact rule set. Simultaneously, important attributes could be highlighted in the extracted rules. Rule extraction results from our proposed method could follow decisions of SVM classifiers very well. Comparisons between our method and other rule extraction methods are also carried out on several benchmark data sets. Higher rule accuracy is obtained in our method with fewer number of premises in each rule.

BackgroundPrevious work from our group has shown that a model using a limited number of common clinical, demographic and MRI features can robustly predict CDR‐SB progression (Hibar et al. CTAD 2021) and that baseline Tau‐PET levels are associated with subsequent cognitive decline (Teng et al., 2021). However, the gain from combining these predictors has not previously been assessed. The aim of this work was to assess the additive prognostic value of including baseline [18F]GTP1 tau‐PET with the other common predictors.MethodBaseline [18F]GTP1 images and MRI features were available from two crenezumab (NCT02670083 and NCT03114657; n = 51) and two semorinemab (NCT03289143; n = 360 and NCT038287747 placebo arm; n = 113) randomized clinical trials in prodromal to moderate AD populations. The benchmark linear model included age, sex, APOE4 status, education, diagnosis, baseline ADAS‐Cog 11, ADCS‐ADL, CDR‐SB and MMSE and hippocampus, ventricles and whole brain volume as predictors. In addition, the extended model included the whole cortical grey and temporal meta‐ROI [18F]GTP1 SUVR. A gradient boosted decision tree ensemble (XGBoost) model aimed to uncover potential nonlinearities and interaction among the features. Model performance was evaluated by r2 using bootstrapped confidence intervals and calibration.ResultOur analyses included data from 524 participants (prodromal AD: 151, mild AD: 318, moderate AD: 55). The benchmark model yielded r2 = 0.175 [CI 0.122 – 0.241] with CDR‐SB in the overall sample. The [18F]GTP1 only model had r2 = 0.122 [0.071 – 0.185]. The extended linear model yielded r2 = 0.226 [0.163 – 0.299] and remained well calibrated. Point estimates of r2 were lower in the prodromal and mild versus the moderate subgroups. The performance of the XGBoost model on the holdout set (r2 = 0.230 [0.123 – 0.382]) was marginally better than the linear model, but results were variable among the diagnosis subgroups and further validation needs to be performed on a larger independent dataset.ConclusionThe extended model including [18F]GTP1 SUVR features showed increased performance relative to the benchmark model. Therefore, [18F]GTP1 SUVR features may provide additional, meaningful information about patient prognosis and may be used as covariate adjustment factors to enhance treatment efficacy estimates in clinical trials. Analysis using [18F]GTP1 SUVR in other brain regions is being evaluated.

We propose a new method for learning ensembles of process-based models for predictive modeling of dynamic systems from data and knowledge. Previous work has shown that ensembles based on sampling data (i.e., bagging), significantly improve predictive performance of process-based models. However, this improvement comes at the cost of a substantial computational overhead needed for learning. On the other hand, methods for constructing ensembles based on sampling knowledge (i.e., random library samples, RLS) allow for efficient learning ensembles of process-based models, while maintaining their long-term predictive performance. This paper aims at checking the conjecture whether the combination of these methods has a potential for further performance improvements. The proposed method, bagging of random library samples for learning ensembles of process-based models combines the afore-mentioned approaches in terms of sampling both data and knowledge. We apply the method to and evaluate its performance on a set of automated predictive modeling tasks in two lake ecosystems from data and library of knowledge for modeling population dynamics. The experimental results serve both to identify the optimal design decisions regarding the proposed method as well as to asses its predictive ability. The results show that such ensembles outperform single process-based model, but also outperform each of the two methods for learning ensembles from data samples (bagging) and knowledge samples (RLS).

Leveraging advanced ensemble models to increase building energy performance prediction accuracy in the residential building sector

An Efficient Cyber Assault Detection System using Feature Optimization for IoT-based Cyberspace

Concept lattice is an effective tool for data analysis and extracting classification rules. However, the classical concept lattice often produce a lot of redundant rules. Closed-label concept lattice realizes reduction of concept intention, which can be used to extract fewer rules than the classical concept lattice. This paper presents a method for classification rules extraction based on the closed-label concept lattice. Examples show that the proposed method is effective for extracting more concise classification rules.

Reinforced concrete shear walls (RCSWs) are complicated to compute their shear capacity due to their large cross-sectional height-to-thickness ratios and the fact that they are subjected to vertical loads. Numerous factors influence RCSWs’ shear strength capacity, and the analytical models find it challenging to fully account for each factor’s impact on RCSWs’ shear-bearing capacity. Machine learning (ML) technology can deeply capture the mapping relationship between each input feature and the target value, and provide a more flexible and effective prediction method for RCSW shear-bearing capacity. To this end, a shear capacity test database containing 583 RCSW specimens was first established and characterized, and then the database was employed to train single, ensemble, and deep learning models for the shear strength of shear walls and combined with hyper-parameter tuning to enhance each model’s prediction performance, after which the prediction performance of each model was compared. Then, the ML models were contrasted with conventional techniques founded on the mechanical premise. Finally, in order to improve the prediction accuracy and reliability of the ML methods, the individually trained models were integrated into a stacking model using the stacking method, and the stacking model’s prediction performance was assessed. The results of this study show that in the single model, the test set R2 of the decision tree (DT) reaches 0.94, showing good trend-capturing ability. Among the ensemble models, Gradient Boosting (GB) performs the best and is comparable to DT in terms of RMSE and R2 and significantly outperforms other ensemble methods, such as Random Forest (RF) and Bagging. Deep Neural Networks (DNNs) show the strongest predictive ability among all models, with the lowest RMSE (263 kN) and a test R2 of 0.95, which is much better than the majority of ensemble models. The ML models show high accuracy and reliability compared to the traditional RC shear wall shear capacity models. The stacking model has an R2 of 0.98 and a CoV of 0.147 in the test set, and it is much better than other independent ML models (R2 = 0.88~0.95, CoV = 0.179~0.651).

Ideally, probabilistic forecasts improve the decision quality, capture upsides and mitigate downsides. Unfortunately, one of the risks associated with probabilistic forecasts is the possible creation of a large number of models with similar properties or responses. Methods to identify representative models from an ensemble of models are thus necessary. In this paper, we extend the use of distance-based methods, developed at Stanford Center for Reservoir Forecasting, and Kernel clustering techniques to the selection of models for economic decision-making. Using mathematical tools, the large set of models in a high-dimensional space is mapped to a low-dimensional space and a distance function based on several prediction-related quantities is used to quantify the similarities and dissimilarities among an ensemble of history-matched models. We then construct a small portfolio of models with a diverse range of prediction performances for less cost-intensive probabilistic forecasts. The approach is applied to an offshore gas condensate field in the Asia-Pacific region. Two possible development plans were competing: continue to produce from existing geological structures, or extend the development to accumulations connected through a saddle in the carbonate reservoir. The decision was impacted by the uncertainty on remaining volumes in place and connectivity between structures. Based on the uncertainty management plans, 925 history-matched models are generated with genetic algorithms. Within the context of distance-based modeling, an application-tailored distance function between models is designed, balancing static (scenario independent) and dynamic (scenario dependent) properties of the models. Then, eight diverse and representative models are selected among the 925 models. We show that the eight selected models, while offering an equally good history-match, cover well the uncertainty ranges of the static and dynamic properties considered. The selected models can be used to evaluate different possible development scenarios for final business decision making.