Logistic regression for data mining and high-dimensional classification

This paper carries on research surrounding the influences produced by dimensionality reduction on machine learning classification effect. Firstly, paper constructs the analysis architecture of data dimension reduction classification, combines the two different unsupervised dimension reduction methods, locally linear embedding (LLE) and principal component analysis (PCA) with the five machine learning classification methods: Gradient Boosting Decision Tree (GBDT), Random Forest, Support Vector Machine (SVM), K-Nearest Neighbor (KNN) and Logistic Regression. And then uses the handwritten digital identification dataset to analyze the classification performance of these five classification methods on different dimension datasets by different dimensionality reduction methods. The analysis shows that using the appropriate dimensionality reduction method for dimensionality reduction classification can effectively improve the classification accuracy; the dimensionality reduction classification effect of non-linear dimensionality reduction method is generally better than the linear dimensionality reduction method; different machine learning classification algorithms have significant differences in the sensitivity of dimensions.

Gesture recognition is a topic in computer science and language technology that aims to interpret human gestures with computer programs and many different algorithms. It can be seen as the way computers can understand human body language. Today, the main interaction tools between computers and humans are still the keyboard and mouse. Gesture recognition can be used as a tool for communication with the machine and interaction without any mechanical device such as a keyboard or mouse. In this paper, we present the results of a comparison of eight different machine learning (ML) classifiers in the task of human hand gesture recognition and classification to explore how to efficiently implement one or more tested ML algorithms on an 8-bit AVR microcontroller for on-line human gesture recognition with the intention to gesturally control the mobile robot. The 8-bit AVR microcontrollers are still widely used in the industry, but due to their lack of computational power and limited memory, it is a challenging task to efficiently implement ML algorithms on them for on-line classification. Gestures were recorded by using inertial sensors, gyroscopes, and accelerometers placed at the wrist and index finger. One thousand and eight hundred (1800) hand gestures were recorded and labelled. Six important features were defined for the identification of nine different hand gestures using eight different machine learning classifiers: Decision Tree (DT), Random Forests (RF), Logistic Regression (LR), Linear Discriminant Analysis (LDA), Support Vector Machine (SVM) with linear kernel, Naïve Bayes classifier (NB), K-Nearest Neighbours (KNN), and Stochastic Gradient Descent (SGD). All tested algorithms were ranged according to Precision, Recall, and F1-score (abb.: P-R-F1). The best algorithms were SVM (P-R-F1: 0.9865, 0.9861, and 0.0863), and RF (P-R-F1: 0.9863, 0.9861, and 0.0862), but their main disadvantage is their unusability for on-line implementations in 8-bit AVR microcontrollers, as proven in the paper. The next best algorithms have had only slightly poorer performance than SVM and RF: KNN (P-R-F1: 0.9835, 0.9833, and 0.9834) and LR (P-R-F1: 0.9810, 0.9810, and 0.9810). Regarding the implementation on 8-bit microcontrollers, KNN has proven to be inadequate, like SVM and RF. However, the analysis for LR has proved that this classifier could be efficiently implemented on targeted microcontrollers. Having in mind its high F1-score (comparable to SVM, RF, and KNN), this leads to the conclusion that the LR is the most suitable classifier among tested for on-line applications in resource-constrained environments, such as embedded devices based on 8-bit AVR microcontrollers, due to its lower computational complexity in comparison with other tested algorithms.

This dissertation addresses two significant issues in the effective application of machine learning algorithms and models for the physical and engineering sciences. The first is the broad challenge of automated modeling of data across different processes in a physical system. The second is the dilemma of obtaining insightful interpretations on the relationships between the inputs and outcome of a system as inferred from complex, black box machine learning models.Automated Geometric Shape Deviation Modeling for Additive Manufacturing SystemsAdditive manufacturing systems possess an intrinsic capability for one-of-a-kind manufacturing of a vast variety of shapes across a wide spectrum of processes. One major issue in AM systems is geometric accuracy control for the inevitable shape deviations that arise in AM processes. Current effective approaches for shape deviation control in AM involve the specification of statistical or machine learning deviation models for additively manufactured products. However, this task is challenging due to the constraints on the number of test shapes that can be manufactured in practice, and limitations on user efforts that can be devoted for learning deviation models across different shape classes and processes in an AM system. We develop an automated, Bayesian neural network methodology for comprehensive shape deviation modeling in an AM system. A fundamental innovation in this machine learning method is our new and connectable neural network structures that facilitate the transfer of prior knowledge and models on deviations across different shape classes and AM processes. Several case studies on in-plane and out-of-plane deviations, regular and free-form shapes, and different settings of lurking variables serve to validate the power and broad scope of our methodology, and its potential to advance high-quality manufacturing in an AM system.Interpretable Machine LearningMachine learning algorithms and models constitute the dominant set of predictive methods for a wide range of complex, real-world processes. However, interpreting what such methods effectively infer from data is difficult in general. This is because their typical black box natures possess a limited ability to directly yield insights on the underlying relationships between inputs and the outcome for a process. We develop methodologies based on new predictive comparison estimands that effectively enable one to ``mine’’ machine learning models, in the sense of (a) interpreting their inferred associations between inputs and/or functional forms of inputs with the outcome, (b) identifying the inputs that they effectively consider relevant, and (c) interpreting the inferred conditional and two-way associations of the inputs with the outcome. We establish Fisher consistent estimators, and their corresponding standard errors, for our new estimands under a condition on the inputs' distributions. The significance of our predictive comparison methodology is demonstrated with a wide range of simulation and case studies that involve Bayesian additive regression trees, neural networks, and support vector machines. Our extended study of interpretable machine learning for AM systems demonstrates how our method can contribute to smarter advanced manufacturing systems, especially as current machine learning methods for AM are lacking in their ability to yield meaningful engineering knowledge on AM processes.

This thesis presents a clear conceptual basis for theoretically studying machine learning problems. Machine learning methods afford means to automate the discovery of relationships in data sets. A relationship between quantities X and Y allows the prediction of one quantity given information of the other. It is these relationships that we make the central object of study. We call these relationships transitions. A transition from a set X to a set Y is a function from X into the probability distributions on Y. Beginning with this simple notion, the thesis proceeds as follows: • Utilizing tools from statistical decision theory, we develop an abstract language for quantifying the information present in a transition. • We attack the problem of generalized supervision. Generalized supervision is the learning of classifiers from non-ideal data. An important example of this is the learning of classifiers from noisily labelled data. We demonstrate the virtues of our abstract treatment by producing generic methods for solving these problems, as well as producing generic upper bounds for our methods as well as lower bounds for any method that attempts to solve these problems. • As a result of our study in generalized supervision, we produce means to define procedures that are robust to certain forms of corruption. We explore, in detail, procedures for learning classifiers that are robust to the effects of symmetric label noise. The result is a classification algorithm that is easier to understand, implement and parallelize than standard kernel based classification schemes, such as the support vector machine and logistic regression. Furthermore, we demonstrate the uniqueness of this method. • Finally, we show how many feature learning schemes can be understood via our language. We present well motivated objectives for the task of learning features from unlabelled data, before showing how many standard feature learning methods (such as PCA, sparse coding, auto-encoders and so on) can be seen as minimizing surrogates to our objective functions.

Background and aimsThe birth weight of a newborn is a crucial factor that affects their overall health and future well-being. Low birth weight (LBW) is a widespread global issue, which the World Health Organization defines as weighing less than 2,500 g. LBW can have severe negative consequences on an individual’s health, including neonatal mortality and various health concerns throughout their life. To address this problem, this study has been conducted using BDHS 2017–2018 data to uncover important aspects of LBW using a variety of machine learning (ML) approaches and to determine the best feature selection technique and best predictive ML model.MethodsTo pick out the key features, the Boruta algorithm and wrapper method were used. Logistic Regression (LR) used as traditional method and several machine learning classifiers were then used, including, DT (Decision Tree), SVM (Support Vector Machine), NB (Naïve Bayes), RF (Random Forest), XGBoost (eXtreme Gradient Boosting), and AdaBoost (Adaptive Boosting), to determine the best model for predicting LBW. The model’s performance was evaluated based on the specificity, sensitivity, accuracy, F1 score and AUC value.ResultsResult shows, Boruta algorithm identifies eleven significant features including respondent’s age, highest education level, educational attainment, wealth index, age at first birth, weight, height, BMI, age at first sexual intercourse, birth order number, and whether the child is a twin. Incorporating Boruta algorithm’s significant features, the performance of traditional LR and ML methods including DT, SVM, NB, RF, XGBoost, and AB were evaluated where LR, had a specificity, sensitivity, accuracy and F1 score of 0.85, 0.5, 85.15% and 0.915. While the ML methods DT, SVM, NB, RF, XGBoost, and AB model’s respective accuracy values were 85.35%, 85.15%, 84.54%, 81.18%, and 84.41%. Based on the specificity, sensitivity, accuracy, F1 score and AUC, RF (specificity = 0.99, sensitivity = 0.58, accuracy = 85.86%, F1 score = 0.9243, AUC = 0.549) outperformed the other methods. Both the classical (LR) and machine learning (ML) models’ performance has improved dramatically when important characteristics are extracted using the wrapper method. The LR method identified five significant features with a specificity, sensitivity, accuracy and F1 score of 0.87, 0.33, 87.12% and 0.9309. The region, whether the infant is a twin, and cesarean delivery were the three key features discovered by the DT and RF models, which were implemented using the wrapper technique. All three models had the identical F1 score of 0.9318. However, “child is twin” was recognized as a significant feature by the SVM, NB, and AB models, with an F1 score of 0.9315. Ultimately, with an F1 score of 0.9315, the XGBoost model recognized “child is twin” and “age at first sex” as relevant features. Random Forest again beat the other approaches in this instance.ConclusionsThe study reveals Wrapper method as the optimal feature selection technique. The ML method outperforms traditional methods, with Random Forest (RF) being the most effective predictive model for Low-Birth-Weight prediction. The study suggests that policymakers in Bangladesh can mitigate low birth weight newborns by considering identified risk factors.

e17570 Background: Ovarian cancer(OC) is the leading cause of death from gynecologic malignancy. Current challenges include lack of diagnostic tools, predictive biomarkers, and identifying appropriate surgical candidates. Machine learning(ML) is an emerging field that can make accurate projections by making inferences on data and may play a crucial role in OC.The objective of the current study was to review the literature on application of ML in OC and report the most commonly used algorithms and their performance in comparison to existing prediction tools and traditional regression models. Methods: This is a systematic review of published literature from January 1985 to March 2021 on the use of ML in OC. An extensive search of electronic library databases was conducted. Four independent reviewers screened the articles initially by title then full text. Quality was assessed using the MINORS criteria. P-values were generated using the Pearson’s Chi-squared(x2) test to compare performance of ML models with traditional statistics. No p-values were reported if only one study was available. Results: Among 4,295 articles screened, 88 studies on ML in OC were included. The mean age of OC patients was 54.7 years(11-90) and the most common stages at diagnosis were:Stage III (39.9%) and IV (34%). Applications of ML were in clinical datasets(33%, n = 29), preoperative diagnostics(30.7%, n = 27), serum biomarkers (21.6%, n = 19), genomics (12.5%, n = 11), and prediction of cytoreductive outcomes (2.3%, n = 2). The most commonly applied algorithms were Support Vector Machine [SVM](28%, n = 33)and Neural Networks[NN] (25.28%). Over the past decades, the number of publications on ML in OC increased three-fold from 20(1994-2010) to 67 (2011–2021). Only 9 (10%) studies compared ML techniques with existing prediction tools, or traditional regression models. Among 29 clinical dataset studies, 4 compared ML with traditional logistic regression(LR). Two studies reported better performance with ML compared to LR but not significant(accuracy: 0.88 vs 0.84, p = 0.15), one study performed comparably(accuracy: 0.1 vs 0.1) while one study performed worse(accuracy: 0.1 vs 0.97). Only one preoperative diagnostic study compared ML techniques with LR. SVM classifiers outperformed LR in classifying ovarian masses as benign or malignant(sensitivity: 0.88 vs. 0.70). One serum biomarker study compared LR with ML algorithms; LR performed better using two biomarkers for predicting OC(accuracy: 0.97 vs. 0.94). Among five studies reporting overall survival outcomes, only one study compared survival ML techniques using NN with LR and showed that NN classifiers outperformed LR in predicting overall survival(AUC: 0.72 vs. 0.62). Conclusions: This is the first systematic review exploring the literature on ML algorithms in OC. Most ML models outperformed traditional models. However, larger datasets would be required to validate findings.

Integrated Circuits (ICs) are suffering severer threats caused by Hardware Trojans (HTs), some of which hide in routine operations by coercing firmware or hardware. Along with conventional side-channel detection not always getting golden-chip, HTs become more difficult to detect. An improved Support Vector Machine (SVM) machine learning frameworks for this is proposed using system-level side-channel analysis. Cross validation experimental results on Field Programmable Gate Array (FPGA) show that in the condition of golden-chip, supervised SVM achieves 85.8% test accuracy in average. After grouping, outlier-removing and normalization, it rises by 4%. Even if golden-chip is out of hand, semi-supervised SVM has accuracy to judge HTs existence, averaging in 52.9%-79.5% under different test modes. Comparing with existing researches, this work verifies the efficiency of SVM for HT detection in instruction level, and points out the relationship between diversified learning conditions with detection performance.

Probabilistic programming refers to the idea of using standard programming constructs for specifying probabilistic models from machine learning and statistics, and employing generic inference algorithms for answering various queries on these models, such as posterior inference and estimation of model evidence. Although this idea itself is not new and was, in fact, explored by several programming-language and statistics researchers in the early 2000, it is only in the last few years that probabilistic programming has gained a large amount of attention among researchers in machine learning and programming languages, and that expressive and efficient probabilistic programming systems (such as Anglican, Church, Figaro, Infer.net, PSI, PyMC, Stan, and Venture) started to appear and have been taken up by a nontrivial number of users. The primary goal of my talk is to introduce probabilistic programming to the CONCUR/QUEST/FORMATS audience. At the end of my talk, I want the audience to understand basic results and techniques in probabilistic programming and to feel that these results and techniques are relevant or at least related to what she or he studies, although they typically come from foreign research areas, such as machine learning and statistics. My talk will contain both technical materials and lessons that I learnt from my machine-learning colleagues in Oxford, who are developing a highly-expressive higher-order probabilistic programming language, called Anglican. It will also include my work on the denotational semantics of higher-order probabilistic programming languages and their inference algorithms, which are jointly pursued with colleagues in Cambridge, Edinburgh, Oxford and Tubingen.

General Background: Heart failure remains a global health concern due to its high prevalence and mortality rates. Specific Background: In Indonesia, heart disease ranks as a leading cause of death, emphasizing the need for reliable predictive tools. Knowledge Gap: Despite the availability of machine learning algorithms, limited studies provide a comparative evaluation using updated clinical datasets in the Indonesian context. Aims: This study aims to compare the performance of Logistic Regression and Support Vector Machine (SVM) in predicting heart failure. Results: Using a dataset of 918 samples and rigorous preprocessing, SVM achieved 90% accuracy with an AUC of 0.93, outperforming Logistic Regression, which scored 88% accuracy and an AUC of 0.9304. SVM demonstrated superior sensitivity and robustness in handling non-linear data, while Logistic Regression offered better calibration for risk interpretation. Novelty: The study’s novelty lies in its integrated open-source pipeline, use of biomedical signal features, and statistical validation via McNemar’s test. Implications: These findings support the implementation of SVM in automated clinical decision systems for early heart failure detection, while highlighting Logistic Regression's value in interpretability-focused clinical settings.Highlight : SVM has higher accuracy and F1-score, indicating superior classification performance. Logistic Regression remains superior in probability calibration, important for clinical risk interpretation. Both models effectively support the development of machine learning-based health prediction systems. Keywords : Logistic Regression, Support Vector Machine (SVM), Heart Failure, Machine Learning Classification, Prediction Calibration

Total shoulder arthroplasty (TSA) for primary glenohumeral osteoarthritis with B2-B3 glenoids is challenging due to the relatively high rate of postoperative complications, such as glenoid implant loosening. Machine learning (ML) is a promising method for predicting outcomes in shoulder arthroplasty. However, no studies have included preoperative radiological data to predict surgical complications using ML. The present study evaluated the potential of ML in predicting the occurrence of complications after TSA in patients treated for glenohumeral osteoarthritis with B2-B3 glenoids by integrating various prognostic factors, such as radiological features. We hypothesized that ML would accurately predict postoperative complications and identify the variables that are most strongly associated with these complications. This retrospective study included 60 patients with primary osteoarthritis and type B2-B3 glenoids from our institutional TSA database. Prognostic factors, including patient characteristics, clinical scores, radiological features, and surgical techniques, were recorded. Outcomes at a minimum of 2 years of follow-up were characterized by the Aldinger complication scale (scored 0-III). Of the 60 patients, 13 (21.7%) experienced complications, with 8 (13.3%) classified as Aldinger I and 5 (8.3%) as Aldinger III. These data were used to train and test four ML methods: logistic regression (LR), gradient boosting classifier (GBC), support vector machine (SVM), and multilayer perceptron classifier (MLPC). We considered a binary outcome: no complication vs. Aldinger I-III. The data were split into a training set (75%) and a testing set (25%). Among the four ML models evaluated, LR and GBC correctly identified all complication cases (3/12), whereas SVM and MLPC missed one complication. The number of false positives was lower with GBC (2/12) and LR (3/12). Younger age, glenoid version and inclination were the main variables associated with complications. Using a posteriorly augmented glenoid implant was associated with lower complication rates. ML can efficiently predict TSA complications, even with a limited dataset. Glenoid retroversion was identified as a critical radiological feature associated with outcomes, as supported by the literature. In addition, younger age is associated with increased complication risks, likely due to increased functional demand. Thus, ML is potentially a valuable tool for forecasting complications in the surgical decision-making process.

In the field of computer vision and image processing CBIR (Content based image retrieval) system become an active research area. The CBIR technique involves feature extraction based on image features (like color, shape and texture) and defining rules for comparing image. A classifier is also used by authors for image classification. For this purpose an image database taken from James Z. Wang research group website is used by many authors. Due to enormous increase in the size of database traditional methods of image retrieval by using keywords does not provide accurate result. Thus the concept of CBIR system is emerged. In last couple of years many method of feature extraction is developed but since now researcher are not able to develop a system which has gained high accuracy of human visual perception in distinguishing pictures. Therefore earlier researchers have concentrated to increase the retrieval accuracy of CBIR system by using different combination of features. The objective of this paper is to present review on a CBIR technique which combined four kinds of features set including every kind of feature for developing feature vector. Color based features (histogram), statistical features (mean, median and standard deviation), spatial features (Gabor feature and wavelet features) and texture features (GLCM) is used for forming feature vector. A SVM classifier is used with CBIR system to increase retrieval accuracy. SVM (support vector machine) is a machine learning classifier which provides good classification accuracy. Keywords: CBIR, SVM, NN, GLCM, HSV Color space, Euclidean distance matrix.

Improving autocoding performance of rare categories in injury classification: Is more training data or filtering the solution?

This dissertation contributes with novel predictive modelling approaches to data-driven computational psychiatry and offers alternative analyses frameworks to the standard statistical analyses in psychiatric research. In particular, this document advances research in medical data mining, especially psychiatry, via two phases. In the first phase, this document promotes research by proposing synergistic machine learning and statistical approaches for detecting patterns and developing predictive models in clinical psychiatry data to classify diseases, predict treatment outcomes or improve treatment selections. In particular, these data-driven approaches are built upon several machine learning techniques whose predictive models have been pre-processed, trained, optimised, post-processed and tested in novel computationally intensive frameworks. In the second phase, this document advances research in medical data mining by proposing several novel extensions in the area of data classification by offering a novel decision tree algorithm, which we call PIDT, based on parameterised impurities and statistical pruning approaches toward building more accurate decision trees classifiers and developing new ensemblebased classification methods. In particular, the experimental results show that by building predictive models with the novel PIDT algorithm, these models primarily led to better performance regarding accuracy and tree size than those built with traditional decision trees. The contributions of the proposed dissertation can be summarised as follow. Firstly, several statistical and machine learning algorithms, plus techniques to improve these algorithms, are explored. Secondly, prediction modelling and pattern detection approaches for the first-episode psychosis associated with cannabis use are developed. Thirdly, a new computationally intensive machine learning framework for understanding the link between cannabis use and first-episode psychosis was introduced. Then, complementary and equally sophisticated prediction models for the first-episode psychosis associated with cannabis use were developed using artificial neural networks and deep learning within the proposed novel computationally intensive framework. Lastly, an efficient novel decision tree algorithm (PIDT) based on novel parameterised impurities and statistical pruning approaches is proposed and tested with several medical datasets. These contributions can be used to guide future theory, experiment, and treatment development in medical data mining, especially psychiatry.

Logistic regression was as good as machine learning for predicting major chronic diseases

Breast cancer is a frequent cancer that develops when normal cells in the breast transform into malignant cells. Breast cancer can arise from glandular tissue, muscular tissue, or fatty tissue in the breast. Many variables contribute to the risk of breast cancer, including genetics, environmental exposure, food, and lifestyle. Breast cancer should be detected early through breast self-examination, regular clinical evaluation, and mammography to identify any abnormal changes, In recent years, early detection of breast cancer in women has emerged as a beacon of hope and a pivotal point in the treatment of this dangerous disease, and its timely identification has become paramount. Modern advancements in technology, especially artificial intelligence algorithms, have played a vital role in developing systems that facilitate automated disease detection, diagnosis, rapid response, and a reduced risk of fatalities. This paper delves into a comparative study of various machine learning (ML) techniques, namely logistic regression (LR), support vector machines (SVM), linear SVM, Gaussian Naive Bayes (GNB), and artificial neural networks (ANNs). The evaluation metrics used in this study are accuracy and elapsed time. The results show that Gaussian Naive Bayes achieved the highest accuracy of 94.07% in just 0.005495 seconds, outperforming SVM (91.85%), linear SVM (90.19%), logistic regression (87.04%), and ANN (37.04%). These findings highlight the potential of Gaussian Naive Bayes in aiding the early detection of breast cancer, leading to more effective and timely interventions, ultimately improving patient outcomes. Keywords: Breast Cancer, Machine learning (ML), Logistic Regression (LR), Support Vector Machine (SVM), Linear SVM, Gaussian Naive Bayes (GNB) and Artificial Neural Networks (ANNs).