Developing and Evaluating Machine Learning Models in Recommendation Systems

In this paper, we present a visual analytics tool for enabling hypothesis-based evaluation of machine learning (ML) models. We describe a novel ML-testing framework that combines the traditional statistical hypothesis testing (commonly used in empirical research) with logical reasoning about the conclusions of multiple hypotheses. The framework defines a controlled configuration for testing a number of hypotheses as to whether and how some extra information about a "concept" or "feature" may benefit or hinder an ML model. Because reasoning multiple hypotheses is not always straightforward, we provide HypoML as a visual analysis tool, with which, the multi-thread testing results are first transformed to analytical results using statistical and logical inferences, and then to a visual representation for rapid observation of the conclusions and the logical flow between the testing results and hypotheses. We have applied HypoML to a number of hypothesized concepts, demonstrating the intuitive and explainable nature of the visual analysis.

In last two decades the growing life expectancy has spiralled the prevalence of neurogenerative disorder such as Dementia among elderly population posing a significant challenge to mental health globally. The dementia is generally characterized by progressive cognitive decline which it affects memory, thinking, behaviour, and the ability to perform everyday activities. Early diagnosis and intervention are crucial for improving patient outcomes and managing the societal burden of dementia. The primary aim of this study is to deploy various machine learning models such as Logistic Regression, K-Nearest Neighbour (KNN), Support Vector Machine (SVM), Decision Tree, Random Forest (RF). Extreme Gradient Boosting (XGBoost) for the classification of Dementia. The suitable features required for the classification is determined through efficient filter-based and wrapper-based feature selection techniques. The experimental evaluation of machine learning models is performed using standard metrics such as accuracy, precision, recall and F1 score. Furthermore, Explainable AI (XAI) techniques such as SHAP and LIME are employed to interpret the black-box nature of these models, offering transparency and insights into the contribution of individual features to the model predictions. The thorough evaluation of machine learning models exhibited that Random Forest outperforms other models with an accuracy of 96%, with CDR identified as a key predictor through XAI analysis.

Machine Learning (ML)-based Network Intrusion Detection Systems (NIDSs) have proven to become a reliable intelligence tool to protect networks against cyberattacks. Network data features has a great impact on the performances of ML-based NIDSs. However, evaluating ML models often are not reliable, as each ML-enabled NIDS is trained and validated using different data features that may do not contain security events. Therefore, a common ground feature set from multiple datasets is required to evaluate an ML model's detection accuracy and its ability to generalise across datasets. This paper presents NetFlow features from four benchmark NIDS datasets known as UNSW-NB15, BoT-IoT, ToN-IoT, and CSE-CIC-IDS2018 using their publicly available packet capture files. In a real-world scenario, NetFlow features are relatively easier to extract from network traffic compared to the complex features used in the original datasets, as they are usually extracted from packet headers. The generated Netflow datasets have been labelled for solving binary- and multiclass-based learning challenges. Preliminary results indicate that NetFlow features lead to similar binary-class results and lower multi-class classification results amongst the four datasets compared to their respective original features datasets. The NetFlow datasets are named NF-UNSW-NB15, NF-BoT-IoT, NF-ToN-IoT, NF-CSE-CIC-IDS2018 and NF-UQ-NIDS are published at http://staff.itee.uq.edu.au/marius/NIDS_datasets/ for research purposes.

The high variability and nonlinear relationships between environmental variables (such as temperature, relative humidity, and altitude) in ecological datasets prevent classical statistical models from obtaining accurate predictions. This study aimed to compare and investigate the performance of AI-based machine learning methods in analyzing complex ecological data structures. An agricultural dataset containing meteorological and vegetation variables was used as the representative case study. This dataset is based on population observations of Cimbex quadrimaculata in Diyarbakır (Eğil) and Elazığ (Keban) provinces in Türkiye between 2020 and 2022. Three different modeling approaches (binary classification, multiclass classification, and regression) were applied to the same data. This three-approach design enabled a systematic comparison of model performance, generalizability, and explainability on the same dataset using different definitions of the target variable. For classification tasks, the model performance was evaluated using accuracy, F1 score, and AUC metrics under a stratified 10-fold cross-validation scheme. Regression models, on the other hand, were assessed within a nested cross-validation framework using R², root mean square error (RMSE), mean absolute error (MAE). Ensemble-based boosting AI algorithms (Gradient Boosting, XGBoost, and LightGBM) demonstrated high accuracy and generalizability in characterizing the highly nonlinear relationships, nested effects, and non-additive interactions among multiple variables. Furthermore, the SHAP analysis improved the interpretability of the models and revealed that temperature- and humidity-related variables were consistently among the most influential predictors in the model predictions. Comparative performance evaluations of machine learning models showed that Gradient Boosting (94.3% accuracy, 0.983 AUC) and XGBoost (84.6% accuracy) were the strongest predictors in binary classification scenarios and overall analyses, respectively. In regression analyses, LightGBM and Random Forest algorithms stood out with cross-validation performances of approximately R² ≈ 0.73. In particular, the success of ensemble-based learning methods in capturing multidimensional relationships in ecological datasets explains the high predictive accuracy and robustness of these models across complex ecological data structures.

Machine Learning (ML) models have been applied to solve problems in various fields, which necessarily involves proper evaluation of models to ensure performance. Once deployed, ML models are subject to performance issues, such as those related to changes in data (drift). This type of issue has prompted efforts in model analysis and maintenance, as well as in continual learning, which seeks the ability to continuously learn from a (continuous) stream of data. Therefore, it's important to understand and develop methodologies that can be used to evaluate ML models, making their use in real-world environments feasible. Amongst current areas of application for ML, one that stands out, in particular, is Machine Learning for Healthcare, especially in conjunction with Software for Decision Support of Medical Applications, which presents specific challenges for the evaluation and monitoring of models, particularly given that incorrect prediction or classification can lead to life-threatening situations. This paper presents a systematic literature review that aims at identifying state-of-the-art techniques for evaluating and maintaining ML models for healthcare in effective use in the real world.

Accurate real-time bus arrival information is essential for an efficient public transport network, as it significantly impacts passenger experience, system reliability, reduced waiting times, dwell times, and operational efficiency. In Sri Lanka's public transport system, current bus arrival time computations primarily rely on static data, neglecting real-time information and critical factors influencing travel times. This highlights the need to identify the unique variables affecting bus arrival times within the Sri Lankan context and to develop robust prediction models that account for these influences. While traditional methods such as Historical Average Models, Regression, Time Series Analysis, and Kalman Filtering have been used in previous research for short-term travel time predictions, Machine Learning (ML) approaches have proven to deliver superior accuracy. ML models are regarded as the most effective for heterogeneous, lane-less traffic conditions with varying traffic volumes, such as those found in Sri Lanka. ML techniques excel in processing large, high-quality datasets and provide accurate predictions by accounting for all relevant variables influencing travel times. Although research has been conducted on developing various basic ML models for travel time prediction, there is a noticeable gap in studies comparing these models to determine the most suitable one for the Sri Lankan context. A Long- Short Term Memory (LSTM) neural network is a deep learning model that is capable of handling long-term dependencies. In the context of bus travel time prediction, LSTMs can leverage historical traffic and travel data to capture temporal patterns and fluctuations that influence travel times. By evaluating LSTMs against basic machine learning models, this study seeks to explore the advantages of applying deep learning techniques to transportation forecasting, ultimately contributing to more accurate and efficient predictive systems in transit planning. The ML models selected in this study include two basic traditional models K- Nearest Neighbours (KNN) and Support Vector Regression (SVR) and four advanced models that utilize ensemble techniques and advanced optimization as Random Forest Regression (RFR), Ada Boost, XG Boost and Gradient Boosting Machine (GBM). The performance of these models was compared with the LSTM model to identify the gap in their accuracies.

Nowadays, human overpopulation is stressing our ecosystems in different ways, agriculture being a critical example as different predictions point towards food shortages in the near future. Accordingly, smart farming is becoming key to the optimization of natural resources so that different crops can be grown efficiently, consuming as few resources as possible. In particular, greenhouses have proved to be an effective way of producing a high volume of vegetables/fruits in a reduced space and within a short time span. Hence, optimizing greenhouse functioning results in less water use and nutrient consumption, less energy use, faster growth, and better product quality. In this article, we carry out an in-depth analysis of different machine learning (ML) models to improve climate control in smart greenhouses. As part of the analysis of the techniques we also considered 3 ways of pre-processing the data, as well as 12-hour and 24-hour forecasting. We focus on forecasting the indoor air temperature of an operational smart greenhouse, i.e. assessing the data anomalies that are inherently present in these environments due to the instability of IoT infrastructures. Several ML models are adapted to time series forecasting to provide an overview of these techniques and to find out which one performs better in this particular scenario. Our results show that, after statistically validating the results, the Random Forest Regression technique gives the best overall result with a mean absolute error of less than 1 degree Celsius.

BackgroundWhile machine learning (ML) models are increasingly used to predict outcomes in health care, their practical effect on health care operations, such as bed capacity management, remains underexplored. There is a variety of traditionally used evaluation metrics to analyze ML models; however, decision makers in health care settings require a deeper understanding of their implications for resource management. Traditional performance measures often fail to provide this practical insight.MethodsIn this work, we conduct a simulation study to evaluate the impact of ML-driven length-of-stay (LOS) predictions on intensive care unit (ICU) bed capacity management. Two classification models differing in terms of explainability and interpretability, logistic regression (LR) and extreme gradient boosting (XGB), are applied to predict ICU-LOS. We use the HiRID dataset containing high-frequency data of more than 33,000 patients. The predictions of the ML models are integrated into a simulation framework that replicates real-world ICU bed management, allowing for the assessment of the practical implications of using these algorithms in a clinical setting.ResultsThe application of both classification models results in improved capacity control regarding the key performance indicators in the simulation study, with XGB outperforming LR. While LR leads to slight overoccupancy in the ICU, slight underoccupancy can be observed when XGB is applied.ConclusionOur study bridges the gap between predictive accuracy and practical application by emphasizing the importance of evaluating ML models within the context of ICU capacity management. The simulation-based approach offers a more relevant assessment for health care practitioners, providing actionable insights that go beyond classical performance measures and directly address the needs of decision makers in clinical practice.HighlightsWe apply multiple classification models for ICU-LOS prediction using time-series data. This approach enables an update of the initial prediction resulting in the possibility of efficiently managing intensive care capacities.We present a simulation-based approach to evaluate ML algorithms and their impact on bed capacity management in real-world clinical settings.Our work provides in-depth insights into the impact of using ML techniques as decision support systems in the ICU and can lead to increased acceptance in practice.

Development of a real-time sign recognition system is a critical step in enabling communication between the speech impaired and normal individuals. This paper proposes a vision-based model for sign recognition using Machine Learning models. A local dataset is created by capturing the images under various light and background conditions by employing a webcam. AlexNet architecture and MATLAB are used to extract the features from captured images and to train several Machine Learning models for their performance evaluation. The images captured in real-time are processed to extract the features and then are directed to the classifier to predict the Sign class. The text to voice conversion module plays the speech based on the output of the classifier. An accuracy of 91.7 % is achieved with Support Vector Machine (SVM) learning model on the private dataset. Sign recognition with the proposed model is more reliable and cost-effective when compared to non-vision based approach.

Multiple sclerosis (MS) is a heterogeneous immune-mediated disorder of the central nervous system, and primary progressive MS (PPMS) is characterized by insidious injury independent of clinical relapses from disease onset. Predicting disease progression remains an unmet need across the MS disease spectrum. We evaluated statistical and machine learning (ML) models for predicting progression in 596 patients with PPMS from ORATORIO, a randomized, placebo (PBO)-controlled, phase 3 clinical study of ocrelizumab (OCR). Using baseline clinical, magnetic resonance imaging, and biomarker (neurofilament light chain [NfL]) data, we trained 6 ML models to predict a progression event (logistic regression, support vector machines, random forest, neural network, gradient boosting, and extreme gradient boosting) and the time to progression (Cox proportional hazards [PH]) on upper limb function, as measured by 20% confirmed worsening on the 9-hole peg test (9HPT). Predictive performance was similar across the six ML models (area under the curve = 0.63 to 0.67) and was modest using the Cox PH model (C-index = 0.76, an improvement over previous studies). In the Cox PH model, treatment arm, baseline 9HPT, neurologic functional system scores, and serum NfL levels were identified as factors associated with progression; model performance was similar across the OCR, PBO, and combined OCR + PBO treatment arms. Accounting for potential interactions with treatment arm resulted in similar predictive performance. To our knowledge, this is the first study to systematically evaluate ML models and statistical methods for predicting progression in PPMS using a well-characterized clinical trial cohort.

Examining the condition of groundwater resources and the impact of droughts is valuable for effective water resources management. Today, machine learning (ML) models are recognized as one of the useful tools in time series predictions. In this study, the groundwater condition of one of the most important aquifers in northwest Iran was investigated using MODFLOW, followed by estimating the groundwater resource index (GRI) utilizing the multivariate adaptive regression spline (MARS) and least squares support vector regression (LSSVR) for a period between 2001 and 2019. Meteorological and hydrological drought indicators along with precipitation and flow rate were used as input variables for prediction. The simulation results revealed a groundwater level decrease since the aquifer withdrawal amount is more than the recharge amount. Besides, results showed that there is a limited interaction between surface water and groundwater resources, mainly caused by the decrease in the river flow and aquifer groundwater level drop. Both ML models performed well in GRI estimation, using groundwater flow, streamflow drought index, standardized precipitation index, and runoff as input variables. The performance of the MARS model with RMSE, MAE, and NSE error evaluation criteria of 0.37, − 0.19, and 0.83, respectively, exerted slightly better results than LSSVR with RMSE, MAE, and NSE of 0.48, − 0.06, and 0.80, respectively. The findings reveal the appropriate performance of both models in forecasting drought indicators, highlighting the necessity of using ML models in hydrology and drought prediction problems.

Properties prediction of composites based on machine learning models: A focus on statistical index approaches

Introduction Machine learning (ML) models are increasingly being developed to predict outcomes in patients undergoing kidney transplantation. Little attention has been paid to assessing their performance across diverse patient subgroups, with risk of bias in predictive performance in subgroups of patients with protected characteristics. This study aimed to evaluate and address potential underperformance of a graft and patient survival prediction model in clinically relevant patient subgroups. Methods We assessed the predictive accuracy of a DeepHit survival analysis model, trained on over 25 years of transplant outcome data. The model's performance was evaluated across subgroups defined by patient ethnicity, sex, age, and pre-transplant sensitization status. To mitigate identified performance disparities, we applied several strategies to re-balance the model's training process to ensure all patient populations were adequately considered. Results Analysis revealed that the model's performance for graft survival was consistent across all investigated subgroups. However, a significant disparity was observed in patient survival predictions. The model demonstrated notably worse predictive accuracy for Black patients and patients over 70 years old, indicating a clear underperformance in these specific populations. Using bias mitigation strategies, we were able to partially close this performance gap. Conclusions This study highlights the critical importance of evaluating ML models for subgroup-specific underperformance in a clinical context. Our findings underscore that while overall model performance may appear strong, it can mask significant disparities. The application of mitigation strategies is essential for developing equitable and reliable decision support tools to ensure fair outcomes for all patients.

Forecasting stock prices is crucial in financial markets, with significant ramifications for investors, traders, and financial institutions. Making accurate predictions allows for well-informed decision-making and effective risk management. In recent years, there has been a notable increase in the utilization of machine learning methods within the financial sector to augment the accuracy of stock price forecasting. The expansion is driven by the growing accessibility of historical market data and the processing capacity necessary for its analysis. Using machine learning techniques inside financial markets has brought about a notable shift in the prevailing paradigm. Conventional approaches, such as time series analysis and fundamental analysis, have played a crucial role but had inherent limitations in comprehensively capturing the intricate dynamics of stock markets. Machine learning has promise in capturing complex patterns and correlations present in data, potentially enhancing the precision of predictive outcomes. The primary objective of this research study is to comprehensively evaluate the efficacy of machine learning models in predicting stock prices. The central research question is: "How effective are machine learning models in predicting the price of stocks?" This question is paramount as it directly impacts investment decisions and risk management strategies. Understanding the capabilities and limitations of machine learning models in this context is crucial for financial professionals and individual investors. This research will primarily consider publicly traded stocks in major stock markets (e.g., NYSE, NASDAQ) over the last decade to maintain a focused investigation. It will explore a range of machine learning techniques, including time series models like ARIMA, deep learning models such as LSTM, and ensemble methods like Random Forest. The analysis will encompass various evaluation metrics commonly used in forecasting, such as Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). This paper is structured as follows: Section 2, provides a comprehensive literature review, delving into traditional stock price prediction methods, the application of machine learning in finance, and previous research relevant to the study. Section 3 analyzes the data collection process, feature selection, model selection, and the chosen evaluation metrics. The experimental data are reported in Section 4, followed by a comprehensive analysis and discussion of these results in Section 5. In conclusion, the present study culminates in Section 6, wherein it provides a comprehensive summary of the principal findings, deliberate on the potential ramifications, and provide valuable perspectives on the prospective advancements in the domain of machine learning as applied to the prediction of stock prices. This study illuminates the dynamic terrain of the banking industry, where the utilization of data-driven decision-making, helped by machine learning, assumes a more central position. Through thoroughly evaluating machine learning models, our research aims to enhance our comprehension of their efficacy and constraints in stock price prediction.

Software designers are coming up with tools having machine learning (ML) capabilities integrated into them. Even naive users can build ML-based models for their respective problem domains using these tools. However, a majority of the advanced ML models behave like a black box in the sense that their behavior is not easily interpretable by humans. This lack of human interpretability acts as a hurdle toward acceptance and subsequent deployment of ML-based solutions. This research work is based on an intuitive idea that in an ideal ML-based solution, the provided dataset, the ML model learned using this dataset, and human domain experts must agree in terms of their perception regarding the problem domain. The objective of this work is to propose a framework that first listens to the dataset, the ML model, and human domain experts individually, and then measure the degree of agreement between them. Listening to the dataset refers identifying important characteristics by computing information gain using entropy and Gini index. Listening to the ML model refers interpreting its decision-making behavior by computing variable importance measures. Listening to human experts refers taking feedback in terms of important features as per their perception regarding the problem domain. The task of measuring agreement between dataset, model, and human experts has been modeled as a “two-judges n-participants” rank correlation problem. The proposed approach has been evaluated on a novel problem domain of predicting joining behavior of freshmen students. A positive degree of agreement was observed between the dataset, the ML model, and human experts in terms of their perception regarding the problem domain. The agreement between the provided dataset, the learned model, and human domain experts helps verify learning acquired by the ML model against the prevailing domain knowledge. This work has the potential to form the basis for developing formal quantitative metrics for evaluating ML models in terms of reliable learning and capability to facilitate trust of human users.