Learning from the machine: interpreting machine learning algorithms for point- and extended-source classification

Machine learning has achieved remarkable success across diverse domains. Nevertheless, concerns about interpretability in black-box models, especially within Deep Neural Networks (DNNs), have become pronounced in safety-critical fields like healthcare and finance. Classical machine learning (ML) classifiers, known for their higher interpretability, are preferred in these domains. Similar to DNNs, classical ML classifiers can exhibit bugs that could lead to severe consequences in practice. Test input prioritization has emerged as a promising approach to ensure the quality of an ML system, which prioritizes potentially misclassified tests so that such tests can be identified earlier with limited manual labeling costs. However, when applying to classical ML classifiers, existing DNN test prioritization methods are constrained from three perspectives: 1) Coverage-based methods are inefficient and time-consuming; 2) Mutation-based methods cannot be adapted to classical ML models due to mismatched model mutation rules; 3) Confidence-based methods are restricted to a single dimension when applying to binary ML classifiers, solely depending on the model’s prediction probability for one class. To overcome the challenges, we propose MLPrior, a test prioritization approach specifically tailored for classical ML models. MLPrior leverages the characteristics of classical ML classifiers (i.e., interpretable models and carefully engineered attribute features) to prioritize test inputs. The foundational principles are: 1) tests more sensitive to mutations are more likely to be misclassified, and 2) tests closer to the model’s decision boundary are more likely to be misclassified. Building on the first concept, we design mutation rules to generate two types of mutation features (i.e., <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">model mutation features</b> and <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">input mutation features</b> ) for each test. Drawing from the second notion, MLPrior generates <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">attribute features</b> of each test based on its attribute values, which can indirectly reveal the proximity between the test and the decision boundary. For each test, MLPrior combines all three types of features of it into a final vector. Subsequently, MLPrior employs a pre-trained ranking model to predict the misclassification probability of each test based on its final vector and ranks tests accordingly. We conducted an extensive study to evaluate MLPrior based on 185 subjects, encompassing natural datasets, mixed noisy datasets, and fairness datasets. The results demonstrate that MLPrior outperforms all the compared test prioritization approaches, with an average improvement of 14.74%∼66.93% on natural datasets, 18.55%∼67.73% on mixed noisy datasets, and 15.34%∼62.72% on fairness datasets.

&lt;p&gt;Thesis examined at the University of Nottingham in April 2018,&nbsp;accepted in May 2018.&nbsp;&lt;/p&gt;

We discuss the statistical foundations of morphological star–galaxy separation. We show that many of the star–galaxy separation metrics in common use today (e.g., by Sloan Digital Sky Survey or SExtractor) are closely related both to each other, and to the model odds ratio derived in a Bayesian framework by Sebok. While the scaling of these algorithms with the noise properties of the sources varies, these differences do not strongly differentiate their performance. We construct a model of the performance of a star–galaxy separator in a realistic survey to understand the impact of observational signal-to-noise ratio (S/N) (or equivalently, 5σ limiting depth) and seeing on classification performance. The model quantitatively demonstrates that, assuming realistic densities and angular sizes of stars and galaxies, 10% worse seeing can be compensated for by approximately 0.4 mag deeper data to achieve the same star–galaxy classification performance. We discuss how to probabilistically combine multiple measurements, either of the same type (e.g., subsequent exposures), or differing types (e.g., multiple bandpasses), or differing methodologies (e.g., morphological and color-based classification). These methods are increasingly important for observations at faint magnitudes, where the rapidly rising number density of small galaxies makes star–galaxy classification a challenging problem. However, because of the significant role that the S/N plays in resolving small galaxies, surveys with large-aperture telescopes, such as LSST, will continue to see improving star–galaxy separation as they push to these fainter magnitudes.

Using a machine learning (ML) method, we mine DB white dwarfs (DBWDs) from the Sloan Digital Sky Survey (SDSS) Data Release (DR) 12 and DR14. The ML method consists of two parts: feature extraction and classification. The least absolute shrinkage and selection operator (LASSO) is used for the spectral feature extraction by comparing high quality data of a positive sample group with negative sample groups. In both the training and testing sets, the positive sample group is composed of a selection of 300 known DBWDs, while the negative sample groups are obtained from all types of SDSS spectra. In the space of the LASSO detected features, a support vector machine is then employed to build classifiers that are used to separate the DBWDs from the non-DBWDs for each individual type. Depending on the classifiers, the DBWD candidates are selected from the entire SDSS data set. After visual inspection, 2808 spectra (2029 objects) are spectroscopically confirmed. By checking the samples with the literature, there are 58 objects with 60 spectra that are newly identified, including a newly discovered AM CVn. Finally, we measure their effective temperatures (Teff), surface gravities (log g), and radial velocities, before compiling them into a catalog.

We investigate the geometric complexity of decision boundaries for robust training compared to standard training. By considering the local geometry of nearest neighbour sets, we study them in a model-agnostic way and theoretically derive a lower-bound \(R^*\in \mathbb {R}\) on the perturbation magnitude \(\delta \in \mathbb {R}\) for which robust training provably requires a geometrically more complex decision boundary than accurate training. We show that state-of-the-art robust models learn more complex decision boundaries than their non-robust counterparts, confirming previous hypotheses. Then, we compute \(R^*\) for common image benchmarks and find that it also empirically serves as an upper bound over which label noise is introduced. We demonstrate for deep neural network classifiers that perturbation magnitudes \(\delta \ge R^*\) lead to reduced robustness and generalization performance. Therefore, \(R^*\) bounds the maximum feasible perturbation magnitude for norm-bounded robust training and data augmentation. Finally, we show that \(R^*< 0.5R\) for common benchmarks, where R is a distribution’s minimum nearest neighbour distance. Thus, we improve previous work on determining a distribution’s maximum robust radius.

Purpose. This study is aimed at introducing a new hybrid machine learning model to enhance the accuracy and speed in detecting botnet attacks in Internet of Things networks. The new model is derived from an integration of decision tree algorithm and feature selection algorithms to produce a novel hybrid machine learning for better performance in IoT botnet detection. Methodology. The study adopts a six steps research methodology. It consists of dataset collection, dataset preprocessing, applying machine learning, comparing feature selection algorithms, combining both machine learning and feature selection algorithms, and finally comparing the results. Findings. A novel hybrid machine learning (ML) model called FoSDeT has been obtained as a result of combination of decision tree algorithm and feature selection algorithm called Forward Selection which shows a significant improvement in IoT botnet detection in comparison to standard decision tree model. Originality. The paper proposes a simple yet powerful hybrid approach which integrates Decision Tree algorithm with two pre-defined feature selection algorithms namely, Forward Selection and Backward Elimination. The new hybrid model called FoSDeT shows a significant enhancement in terms of IoT botnet detection. Practical value. The hybrid model obtained from this study might be used by IT security practitioners in developing real intrusion detection system for defending IoT networks from botnet attacks.

Mental disorder is an illness that are more common in technical employees and are increasing among working professionals. It is an important and challenging issue in the world as the stress among the technical employees are high due to work culture. Mental health in the healthcare is necessary to recognize individual’s situation and continue to predict diverse situations accurately. Therefore, this study proposes the mental health disorder prediction using various feature selection algorithms for a specific dataset called Tech survey Datasets. We applied multiple machine learning classification algorithms on the best features of RFE, RFECV and LASSO to obtain the performance metrics of the model and to decide the best accuracy, precision and recall of the respective models. There are 61 features in the Tech survey Dataset consisting of the data in the technical workplace worldwide that provides mental health attribute and frequency. The results are discussed and an aggregated table is developed using the performance metrics to understand the percentage of technical employees undergoes mental disorder. The proposed research evaluates various classification algorithms along with feature selection algorithm to forecast the mental disorder in a particular workplace and found to be 79% accuracy.KeywordsMental disorderFeature selectionRFERFECVLASSOAssociative propertyDistributive property

Photometric redshifts (photo- z ’s) are crucial for the cosmology, galaxy evolution, and transient science drivers of next-generation imaging facilities like the Euclid Mission, the Vera C. Rubin Observatory, and the Nancy Grace Roman Space Telescope. Previous work has shown that image-based deep learning photo- z methods produce smaller scatter than photometry-based classical machine learning (ML) methods on the Sloan Digital Sky Survey (SDSS. Main Galaxy Sample, a test bed photo- z dataset. However, global assessments can obscure local trends. To explore this possibility, we used a self-organizing map (SOM) to cluster SDSS galaxies based on their ugriz colors. Deep learning methods achieve lower photo- z scatter than classical ML methods for all SOM cells. The fractional reduction in scatter is roughly constant across most of color space with the exception of the most bulge-dominated and reddest cells where it is smaller in magnitude. Interestingly, classical ML photo- z ’s suffer from a significant color-dependent attenuation bias, where photo- z ’s for galaxies within an SOM cell are systematically biased towards the cell’s mean spectroscopic redshift and away from extreme values, which is not readily apparent when all objects are considered. In contrast, deep learning photo- z ’s suffer from very little color-dependent attenuation bias. The increased attenuation bias for classical ML photo- z methods is the primary reason why they exhibit larger scatter than deep learning methods. This difference can be explained by the deep learning methods weighting redshift information from the individual pixels of a galaxy image more optimally than integrated photometry.

Active learning is one form of supervised machine learning. In supervised learning, a set of labeled samples is passed to a learning algorithm for training a classifier. However, labeling large amounts of training samples can be costly and error-prone. Active learning deals with the development of algorithms that interactively select a subset of the available unlabeled samples for labeling, and aims at minimizing the labeling effort while maintaining classification performance. The key challenge for the development of so-called active learning strategies is the balance between exploitation and exploration: On the one hand, the estimated decision boundary needs to be refined in feature space regions where it has already been established, while, on the other hand, the feature space needs to be scanned carefully for unexpected class distributions. In this thesis, two approaches to active learning are presented that consider these two aspects in a novel way. In order to lay the foundations for the first one, it is proposed to express the uncertainty in class prediction of a classifier at a test point in terms of a second-order distribution. The mean of this distribution corresponds to the common estimate of the posterior class probabilities and thus is related to the distance of the test point to the decision boundary, whereas the spread of the distribution indicates the degree of exploration in the corresponding region of feature space. This allows for the evaluation of the utility of labeling a yet unlabeled point with respect to classifier improvement in a principled way and leads to a completely novel approach to active learning. The proposed strategy is then implemented and evaluated based on kernel density classification. The generic active learning strategy can be combined with any other classifier, but it performs best if the derived second-order distributions are sufficiently good approximations to the sampling distribution. Although second-order distributions for random forests are derived in this thesis, they do not approximate sufficiently well the sampling distribution and mainly allow only for the relative comparison of prediction uncertainty between test points. In order to combine the state of the art classification performance of random forests with the principal ideas of the first active learning approach, a related second approach for random forests is derived. It is, in addition, tailored to the demands in industrial optical inspection: bag-wise labeling with weak labels and strongly imbalanced classes. Moreover, an outlier detection scheme based on random forests is derived that is used by the proposed active learning algorithm. Finally, a new computational scheme for Gaussian process classification is presented. It is compared to two standard methods in geostatistics, both with respect to theoretical consistency and practical performance. The method evolved as a by-product when considering using Gaussian process models for active learning.

In the field of data mining applications, the most challenging research work is to classify imbalanced data in supervised learning. From the beginning of the machine learning era, a huge number of researches has done by inventing a lot of successful data balancing methods, however, these are not yet fully compatible to learn from imbalanced data. The key reason is data is producing in every moment and randomly by various sources, like humans, machines, sensors, robots, and so on. To date, the existing machine learning algorithms mostly biased to the majority class instances and neglecting the minority class instances, therefore they have a crucial effect on the prediction capacity and exactness on the final results. To deal with imbalanced data the most popular classifiers are under-sampling and over-sampling, cost-sensitive learning and ensemble learning. In this paper, we have presented a novel data balancing method by employing clustering technique with SVM especially decision boundary instances for creating balanced data. The proposed approach selects the most informative majority class and minority class support vectors nearest to the decision boundary or hyperplane, and finally, combines them to establishes a novel method. We have experimented with the performance of our proposed technique with some popular single classifiers such as C4.5 Decision Tree (DT) and Naive Bayes (NB) classifier and ensemble classifiers such as Random Forest and AdaBoost on thirteen standard real-life imbalanced datasets. The experimental result shows apparently, that the proposed idea generates improved accuracy when classifying both the minority and majority class instances compared to other existing methods.

Cosmology is the study of the universe comprising stars and galaxies. Advancement in the telescope has made it possible to capture high-resolution images, which can be analyzed using machine learning (ML) algorithms. This paper classifies the star galaxy dataset into two classes: star and galaxy using ML algorithms and compares their classification performance. It is observed that random forest provides better accuracy of 78% as compared to other ML classifiers. Further to improve the classification accuracy, we proposed a CNN (Convolution Neural Network) model and achieved an accuracy of 92.44%. Since the CNN model itself extracts the characteristics, it exhibits superior classification accuracy.

In the present state of affairs, the motive behind every educational organization is to uplift the academic achievement of students. Educational data mining (EDM) is an upward field of research, and it is very helpful for academic institutions to predict the academic performance of the students. Educational datasets are the basis of various predictive models. Quality of these models can be improved by using feature selection (FS). To get the required benefits from the available data, there must be some tools for analysis and prediction. In lieu of the above, machine learning/data mining are most suitable. In educational data mining, for better accuracy of prediction models’ and quality of various educational datasets, feature selection (FS) plays a vital role. Feature selection (FS) algorithms abolish inappropriate information from the repositories of educational background so that performance of classifier in terms of accuracy could be increased and the same could be used for better decision. In lieu of the above, a best feature selection algorithm must be selected. In this paper, two filter selection approaches namely correlation feature selection (CFS) and wrapper-based feature selection have been used to demonstrate the importance of selection of a feature subset for a classification problem. The present paper aims to find the detailed investigation of filter feature selection algorithms along with the classification algorithms on a given dataset. We found result with numerous numbers of features from various Feature selection algorithms and classifiers which will help the researcher to discover the most excellent mixture of filter feature selection algorithms and its associated classifiers. The result indicates that SMO and J48 have the highest accuracy measures with the correlation feature selection algorithms, while Naïve Bayes has the highest accuracy measures with the wrapper subset feature selection algorithms for predicting high, medium and low grade for the students.

This paper examines the enhancement of security measures for the Internet of Things (IoT) systems through the application of Machine Learning (ML) techniques. As the number of IoT devices continues to rise, ensuring their security has become increasingly critical, given that conventional methods frequently struggle to identify advanced threats. This study explores the implementation of several ML algorithms, including Random Forest (RF), Decision Trees (DT), Support Vector Machines (SVM), and Convolutional Neural Networks (CNN), to identify anomalies and intrusions within IoT networks. By conducting a comprehensive review of existing research and experiments, it highlights the effectiveness of ML in enhancing IoT security, with high detection rates for various threats, including botnet attacks, Denial-of-Service (DoS) and Distributed Denial-of-Service (DDoS) incidents, and intrusion attempts. DoS/DDoS attacks and many types of botnets are the most devastating attacks that have been spreading for a long time, and they are still branching out in new ways against IoT networks. They can damage IoT services and prevent these services from being used by legitimate users. Therefore, securing IoT networks becomes a significant concern. The proposed model is used to increasingly monitor network traffic for any deviations from standard patterns IoT networks. This paper also stresses the necessity of utilising suitable datasets and feature selection techniques to enhance the efficacy of ML models. To train our model, we have utilized a dataset called the IoT23 dataset, which is one of the most recent datasets that has many IoT scenarios and anomalous activities. Further-more, we utilised two types of feature selection algorithms, the Correlation-based Feature Selection (CFS) algorithm and the Genetic Algorithm (GA), and then we compared the results of these algorithms when training our model. The best performances were obtained with DT and RF classifiers when they were trained with features selected by CFS However, for training and testing times metrics, DT performance was superior across both feature selection methods.

Compressed signal processing (CSP) is a branch of compressive sensing (CS), which gives a direction to solve a class of signal processing problems directly from the compressive measurements of a signal. CSP utilizes the information preserved in the compressive measurements of a signal to solve certain inference problems like: classification, detection, and estimation, without reconstructing the original signal. It further simplifies the signal processing compared to conventional CS by omitting their complex reconstruction stage. This, in turn, reduces the implementation complexity of signal processing systems. This paper investigates the performance of CSP for classification application. After extracting the features from compressive measurements, these features or the data instances are used for classification purpose. Through experimental analysis, it has been found that as the CS undersampling factor is increased, the overlapping among the data instances predominates. This results in a complex decision boundary, which in turn degrades the classification accuracy at higher undersampling factors. To overcome the above issue, this paper proposes the use of a machine learning method known as overlap aware learning along with CSP. This generates a smoother decision boundary and hence improves the classification accuracy at higher undersampling factors. The simulation results show the trend of improved classification accuracy using the proposed method. An analysis of the proposed method has been done on different datasets and based on run-time complexity and complexity vs gain analysis to verify the effectiveness of proposed method.

Background Interpretable machine learning (ML) for early detection of cancer has the potential to improve risk assessment and early intervention. Methods Data from 261 proteins related to inflammation and/or tumor processes in 123 blood samples collected from healthy persons, but of whom a sub‐group later developed squamous cell carcinoma of the oral tongue (SCCOT), were analyzed. Samples from people who developed SCCOT within less than 5 years were classified as tumor‐to‐be and all other samples as tumor‐free. The optimal ML algorithm for feature selection was identified and feature importance computed by the SHapley Additive exPlanations (SHAP) method. Five popular ML algorithms (AdaBoost, Artificial neural networks [ANNs], Decision Tree [DT], eXtreme Gradient Boosting [XGBoost], and Support Vector Machine [SVM]) were applied to establish prediction models, and decisions of the optimal models were interpreted by SHAP. Results Using the 22 selected features, the SVM prediction model showed the best performance (sensitivity = 0.867, specificity = 0.859, balanced accuracy = 0.863, area under the receiver operating characteristic curve [ROC‐AUC] = 0.924). SHAP analysis revealed that the 22 features rendered varying person‐specific impacts on model decision and the top three contributors to prediction were Interleukin 10 (IL10), TNF Receptor Associated Factor 2 (TRAF2), and Kallikrein Related Peptidase 12 (KLK12). Conclusion Using multidimensional plasma protein analysis and interpretable ML, we outline a systematic approach for early detection of SCCOT before the appearance of clinical signs.