A Systematic Analysis of Performance Evaluation Metrics in Machine Learning Models

Explainable hybrid tabular Variational Autoencoder and feature Tokenizer Transformer for depression prediction

A machine learning framework to predict kidney graft failure with class imbalance using Red Deer algorithm

Hardware Trojans (HTs) pose a serious threat to integrated circuit (IC) security. Detection of HTs is extremely challenging due to their stealthy nature. Side-channel analysis techniques have emerged as promising approaches for HT detection by observing anomalies in physical parameters like power or delay. More recently, machine learning (ML) methods have been explored to enhance the accuracy and efficiency of side-channel based HT detection. Background: This paper presents a novel approach using machine learning techniques to detect stealthy hardware Trojans through side-channel analysis. Hardware Trojans, malicious modifications inserted into integrated circuits during manufacturing, pose significant threats to the integrity and security of electronic systems. Traditional methods of detecting these Trojans often rely on known signatures or specific patterns, making them ineffective against subtle and sophisticated attacks. Method: This paper provides a comprehensive review of research advancements in applying ML for side-channel based HT detection. First, an overview of HT attacks, their classification, threat models and detection challenges is presented. Next, various side-channel parameters like power, temperature, delay and electromagnetic emanations used for HT detection are discussed along with their merits and demerits. Furthermore, the application of supervised, unsupervised and semi-supervised ML algorithms for automated feature extraction and intelligent decision making is elucidated in detail. Result: Specifically, the data collection strategies, feature extraction techniques, ML models and performance evaluation metrics adopted in existing literature are critically reviewed. In addition, the limitations of current approaches and promising future research directions like on-chip ML implementation, hierarchical ML and explainable ML models tailored for HT detection are highlighted. Conclusion: Case studies on benchmark circuits are also presented to demonstrate the efficacy of ML-based side-channel HT detection methods. Through an extensive literature review and incisive analysis, this paper provides contemporary insights on the advancement of ML techniques to enable robust side-channel based HT detection for securing next-generation ICs.

Objectives: The primary objective of the study is to classify the blood pressure (BP) levels using advanced machine learning (ML) techniques for predictive purposes. The study assesses the efficacy of the Naïve Bayes, AdaBoost, feedforward neural networks (FNNs), and long short-term memory (LSTM) algorithms over the conventional multinomial logistic model using standard performance evaluation metrics. Methods: The dataset comprised 15,000 entries obtained from the National Health Service, England, each containing eight variables. The variables include BP, age, weight, height, gender, smoking habit, alcohol consumption, and fitness level. The Naïve Bayes, AdaBoost, FNN, LSTM, and multinomial logistic models were employed in the study. Each model underwent training, testing, validation, and evaluation using suitable metrics such as accuracy, F1-Score, kappa statistics, sensitivity, specificity, and area under the curve score. Results: The FNN model gives the highest test accuracy of 89.47% and balanced performance, making it the most appropriate model for predicting BP levels. The LSTM model demonstrated strong proficiency in capturing temporal patterns. AdaBoost was highly effective for dealing with class imbalance, but Naïve Bayes was a dependable benchmark. The multinomial logistic model established a reliable and stable reference point. The results represented a notable improvement over previous research, which typically reported median accuracy rates in the 80–85% range. Conclusion: The study reveals that knowing an individual’s age, weight, height, gender, smoking habit, alcohol consumption, and fitness level is useful in predicting his/her BP level. Thus, the advanced ML algorithms demonstrate potential in accurately classifying BP levels and can aid in the prevention, detection, and management of hypertension.

Aims This research paper aims to check the effectiveness of a variety of machine learning models in classifying esophageal cancer through MRI scans. The current study encompasses Convolutional Neural Network (CNN), K-Nearest Neighbor (KNN), Recurrent Neural Network (RNN), and Visual Geometry Group 16 (VGG16), among others which are elaborated in this paper. This paper aims to identify the most accurate model to facilitate increased, improved diagnostic accuracy to revolutionize early detection methods for this dreadful disease. The ultimate goal is, therefore, to improve the clinical practice performance and its results with advanced machine learning techniques in medical diagnosis. Background Esophageal cancer poses a critical problem for medical oncologists since its pathology is quite complex, and the death rate is exceptionally high. Proper early detection is essential for effective treatment and improved survival. The results are positive, but the conventional diagnostic methods are not sensitive and have low specificity. Recent progress in machine learning methods brings a new possibility to high sensitivity and specificity in the diagnosis. This paper explores the potentiality of different machine-learning models in classifying esophageal cancer through MRI scans to complement the constraints of the traditional diagnostics approach. Objective This study is aimed at verifying whether CNN, KNN, RNN, and VGG16, amongst other advanced machine learning models, are effective in correctly classifying esophageal cancer from MRI scans. This review aims at establishing the diagnostic accuracy of all these models, with the best among all. It plays a role in developing early detection mechanisms that increase patient outcome confidence in the clinical setting. Methods This study applies the approach of comparative analysis by using four unique machine learning models to classify esophageal cancer from MRI scans. This was made possible through the intensive training and validation of the model using a standardized set of MRI data. The model’s effectiveness was assessed using performance evaluation metrics, which included accuracy, precision, recall, and F1 score. Results In classifying esophageal cancers from MRI scans, the current study found VGG16 to be an adequate model, with a high accuracy of 96.66%. CNN took the second position, with an accuracy of 94.5%, showing efficient results for spatial pattern recognition. The model of KNN and RNN also showed commendable performance, with accuracies of 91.44% and 88.97%, respectively, portraying their strengths in proximity-based learning and handling sequential data. These findings underline the potential to add significant value to the processes of esophageal cancer diagnosis using machine learning models. Conclusion The study concluded that machine learning techniques, mainly VGG16 and CNN, had a high potential for escalated diagnostic precision in classifying esophageal cancer from MRI imaging. VGG16 showed great accuracy, while CNN displayed advanced spatial detection, followed by KNN and RNN. Thus, the results set new opportunities for introducing advanced computational models to the clinics, which might transform strategies for early detection to improve patient-centered outcomes in oncology.

<p>Accurate water level (WL) forecasting is important for water resources management and planning purposes in the Great Lakes. The objectives of this research are two-fold.  The first objective is to apply machine learning (ML) (i.e., random forest (RF) and support vector regression (SVR)) and hybrid convolutional neural network(CNN)-long-short term memory (LSTM) deep learning (DL) models for multi-step (i.e., one-, two- and three-monthly step ahead) WL forecasting in the Great Lakes (Michigan and Ontario). The second objective is to integrate the boundary corrected (BC) maximal overlap discrete wavelet transform (MODWT) with SVR, RF, and CNN-LSTM models to improve the performance of the individual models. By employing a BC-wavelet decomposition method, the ‘future data’ issue (i.e., data from the future that is not available), often overlooked in the literature and a major barrier to achieving realistic forecasting performance is overcome. </p><p>For Lakes Michigan and Ontario, 1212 monthly WL (m) records (spanning Jan 1918–Dec 2018) were used to develop the models. For the non-wavelet-based models (SVR, RF, and CNN-LSTM), candidate model inputs included the WL recorded over the previous 12 months.  For the BC-MODWT-based models (BC-MODWT-SVR, BC-MODWT-RF, and BC-MODWT-CNN-LSTM), the lagged input time series were decomposed into BC-wavelet and scaling coefficients by using different mother wavelets (Haar, Daubechies, Symlets, Fejer-Korovkin and Coiflets), filter lengths (from two up to 12) and decomposition levels (from one up to seven).  For each method (SVR, RF, and CNN-LSTM), mother wavelet, and decomposition level a model was generated.  For both wavelet- and non-wavelet-based models, the particle swarm optimization (PSO) method was used to select the most appropriate inputs to include in the proposed multi-step WL forecasting models.</p><p>The datasets were partitioned into calibration and validation subsets. After calibrating the models, various performance evaluation metrics, e.g., coefficient of determination (R<sup>2</sup>), root mean square error (RMSE), mean absolute error (MAE), root mean square percentage error (RMSPE), mean absolute percentage error (MAPE) and the Nash-Sutcliffe efficiency coefficient (NSC) were used to assess model accuracy.</p><p>Of the ML models, the SVR outperformed RF while the DL models outperformed the ML models for each forecast lead time (one-, two-, and three-step(s) ahead). Results from this case study indicate that not all wavelet families and decomposition levels perform equally and in some cases, the wavelet-based models do not improve performance over the non-wavelet-based models. However, the BC-MODWT-CNN-LSTM using suitable mother wavelets (e.g., Haar) outperforms the individual ML and BC-MODWT-ML-based models. More accurate forecasts were obtained for Lake Michigan although the performance in both Great Lakes was accurate. The outcomes of this research indicate that the BC-MODWT-CNN-LSTM model is a promising tool for generating accurate WL forecasts.</p>

Sickle cell disease (SCD) is a common serious genetic disease, which has a severe impact due to red blood cell (RBCs) abnormality. According to the World Health Organisation, 7 million newborn babies each year suffer either from the congenital anomaly or from an inherited disease, primarily from thalassemia and sickle cell disease. In the case of SCD, recent research has shown the beneficial effects of a drug called hydroxyurea/hydroxycarbamide in modifying the disease phenotype. The clinical management of this disease-modifying therapy is difficult and time consuming for clinical staff. This includes finding an optimal classifier that can help to solve the issues with missing values, multi-class datasets, and features selection. For the classification and discriminant analysis of SCD datasets, 7 classifiers based on machine learning models are selected representing linear and non-linear methods. After running these classifiers with a single model, the results revealed that a single classifier has provided us with effective outcomes in terms of the classification performance evaluation metric. In order to produce such an optimal outcome, this research proposed and designed combined classifiers (ensemble classifiers) among the neural network’s models, the random forest classifier, and the K-nearest neighbour classifier. In this aspect, combining the levenberg-marquardt algorithm, the voted perceptron classifier, the radial basis neural classifier, and random forest classifier obtain the highest rate of performance and accuracy. This ensemble classifier receives better results during the training set and testing set process. Recent technology advances based on smart devices have improved the medical facilities and become increasingly popular in association with real-time health monitoring and remote/personal health-care. The web-based system developed under the supervision of the haematology specialist at the Alder Hey Children’s Hospital in order to produce such an effective and useful system for both patients and clinicians. To sum up, the simulation experiment concludes that using machine learning and the web-based system platforms represents an alternative procedure that could assist healthcare professionals, particularly for the specialist nurse and junior doctor to improve the quality of care with sickle cell disorder.

Performance evaluation metrics for ring-oscillator-based temperature sensors on FPGAs: A quality factor

Researchers are interested in genomic level cancer classification because it offers a thorough, precise, and reliable diagnosis for various cancer types. The gene expression data set is made up of hundreds of samples, each containing thousands of genes. As a result, accurately and effectively classifying such samples is difficult. To analyze gene expression patterns efficiently, recent studies have used artificial intelligence, data mining, and statistical techniques. Furthermore, machine learning methods have been extensively used to construct robust and accurate classification frameworks. In this paper, we examined the effectiveness of machine learning-based classification techniques such as Support Vector Machine, Logistic Regression, Naïve Bayes, Decision Tree, K-Nearest Neighbors, and Random Forest methods. In a total of 72 patients, 7129 distinct gene expressions and their degrees were evaluated to determine their relevance. Features have been normalized and PCA was used to decrease the dataset's dimensionality and complexity. The dataset was divided into training and test sets, with 90% of the occurrences in the training set and 10% in the test set. And finally, SMOTE was employed to oversample the minority class. In this study, gene expression data was utilized to differentiate Acute Myeloid Leukemia (AML) from Acute Lymphoblastic Leukemia (ALL). To evaluate the performance of the six machine learning models we used, we examined their associated confusion matrices and calculated nine performance evaluation metrics for each model's prediction on the test dataset. Results demonstrate that SVM, Logistic Regression, and Naïve Bayes outperformed the other ML techniques with an accuracy of 100%.

Motivated to address the inconsistency between the essential i.i.d. assumption in machine learning theory and the data heterogeneity in real-world applications, we propose a novel calibrated ensemble (CE) algorithm to facilitate learning with diverse data subgroups. Unlike the traditional ensemble framework in which each learner is trained independently using the entire dataset, our method exploits the strengths of various machine learning models by training them simultaneously and forming model-ergonomic data subgroups as part of the training process. Consequently, each learner is calibrated to a unique subset of data based on their individualized predictive strength. Clinically, we can interpret each model as an expert specializing in treating patients with particular disease manifestations. We evaluate the CE model in our motivating domain of identifying lupus patients with severe SLE flares using 1541 clinical encounters in the Mass General Brigham (MGB) Lupus Cohort. Our experimental results demonstrate the efficacy of our CE model across seven performance evaluation metrics compared to five individual machine learning models and regular ensemble approaches. We further utilize ANOVA and Tukey HSD post-hoc statistical analysis to discover characteristic features of individual model clusters for clinical interpretations.

Machine learning models to predict the residual tensile strength of glass fiber reinforced polymer bars in strong alkaline environments: A comparative study

—Stress is a common psychological state that significantly impacts human well-being, productivity, and overall health. The ability to accurately detect stress in individuals is crucial for mitigating its harmful effects. In recent years, machine learning (ML) and deep learning (DL) have emerged as powerful tools for stress detection, utilizing physiological data, behavioral cues, and other relevant information. This survey paper provides a comprehensive review of the existing ML and DL approaches used for stress detection, exploring a wide range of models, datasets, and applications. We highlight various techniques, including supervised and unsupervised learning, feature extraction methods, and performance evaluation metrics. Additionally, the challenges in the field, such as data heterogeneity, real- time detection, and model interpretability, are discussed, along with future research directions that could further enhance the effectiveness of stress detection systems.

Early detection of sexually transmitted infections from skin lesions with deep learning: a systematic review and meta-analysis.

The prevalence of nitrates (NO3−) in groundwater due to the extensive application of fertilizers and anthropogenic sources pollutes the groundwater. Machine learning (ML) techniques are now being increasingly deployed to achieve high precision in predicting water quality. This study assesses the efficacy of nine distinct ML algorithms, namely, linear regression, polynomial regression, decision tree, random forest (RF), support vector machine, multilayer perceptron regressor, eXtreme gradient boosting (XGB), light gradient boosting (LGB), and K‐nearest neighbors to predict nitrate concentration in the groundwater in Thiruvannamalai District, Tamil Nadu. Overall, 360 water samples for 1 year and 14 groundwater variables were determined to predict nitrate. Performance evaluation metrics such as root mean square error (RMSE), moving average error (MAE), and correlation coefficient (R2) were evaluated for pre‐monsoon, monsoon, and post‐monsoon seasons. For all three seasons, RF predicted the nitrate concentration with low values of RMSE, MAE, and higher values of R2. The results show values for RF with: RSME: 0.49, MAE: 1.30, and R2: 0.94, which has a higher prediction tailed by LGB and XGB and is true for all the seasons. The results from the study will aid the policymakers in planning the strategy for remediation.

The increasing popularity of internet of things (IoT), dissimilar networks, distributed devices, and applications has turned out to be a major call for the identification of novel security threats and tracing malicious network behaviours. An intrusion detection system (IDS) is a self-defense tool for preventing several types of cyberattacks. Latest machine learning (ML) methods are becoming the backbone for constructing intelligent IDS that are highly data driven. This chapter proposes decision tree based IDS for the dataset NSL-KDD. A novel approach has been developed for the ranking of security features. The proposed system has been validated against performance evaluation metrics consisting of recall, precision, accuracy, and F score. The results produced by the proposed system are compared with well-known ML methods including logistic regression, support vector machines and K-nearest neighbor in order to analyse the efficiency.