Boosting feature selection for Neural Network based regression

• Neural Network feature selection methodology for both numerical and categorical data. • Embedded feature selection provides model and feature subset in a single training. • Novel feature embedding for dealing with high-cardinality categorical features. • Validation of the methodology on several open and real industry dataset. • Comparison with several state-of-the-art feature selection methodologie. In an era of effortless data collection, the impact of machine learning — especially neural networks (NNs) — is undeniable. As datasets grow in size and complexity, efficiently handling mixed data types, including categorical and numerical features, becomes critical. Feature encoding and selection play a key role in improving NN performance, efficiency, interpretability, and generalisation. This paper presents GLEm-Net (Grouped Lasso with Embeddings Network), a novel NN-based approach that seamlessly integrates feature encoding and selection directly into the training process. GLEm-Net uses embedding layers to process categorical features with high cardinality, simplifying the model and improving generalisation. By extending the grouped Lasso regularisation to explicitly consider categorical features, GLEm-Net automatically identifies the most relevant features during training and returns them to the analyst. We evaluate GLEm-Net on open and proprietary industry datasets and compare it to state-of-the-art feature selection methodologies. Results show that GLEm-Net adapts to each dataset by allowing the NN to directly select subsets of most important features, offering on par performance with the best state-of-the-art feature selection methods, while eliminating the need for the external feature encoding and selection steps that are now incorporated in the NN training stage.

Prediction of the stage of cancer plays an important role in planning the course of treatment and has been largely reliant on imaging tools which do not capture molecular events that cause cancer progression. Gene-expression data-based analyses are able to identify these events, allowing RNA-sequence and microarray cancer data to be used for cancer analyses. Breast cancer is the most common cancer worldwide, and is classified into four stages - stages 1, 2, 3, and 4 [2]. While machine learning models have previously been explored to perform stage classification with limited success, multi-class stage classification has not had significant progress. There is a need for improved multi-class classification models, such as by investigating deep learning models. Gene-expression-based cancer data is characterised by the small size of available datasets, class imbalance, and high dimensionality. Class balancing methods must be applied to the dataset. Since all the genes are not necessary for stage prediction, retaining only the necessary genes can improve classification accuracy. The breast cancer samples are to be classified into 4 classes of stages 1 to 4. Invasive ductal carcinoma breast cancer samples are obtained from The Cancer Genome Atlas (TCGA) and Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) datasets and combined. Two class balancing techniques are explored, synthetic minority oversampling technique (SMOTE) and SMOTE followed by random undersampling. A hybrid feature selection pipeline is proposed, with three pipelines explored involving combinations of filter and embedded feature selection methods: Pipeline 1 - minimum-redundancy maximum-relevancy (mRMR) and correlation feature selection (CFS), Pipeline 2 - mRMR, mutual information (MI) and CFS, and Pipeline 3 - mRMR and support vector machine-recursive feature elimination (SVM-RFE). The classification is done using deep learning models, namely deep neural network, convolutional neural network, recurrent neural network, a modified deep neural network, and an AutoKeras generated model. Classification performance post class-balancing and various feature selection techniques show marked improvement over classification prior to feature selection. The best multiclass classification was found to be by a deep neural network post SMOTE and random undersampling, and feature selection using mRMR and recursive feature elimination, with a Cohen-Kappa score of 0.303 and a classification accuracy of 53.1%. For binary classification into early and late-stage cancer, the best performance is obtained by a modified deep neural network (DNN) post SMOTE and random undersampling, and feature selection using mRMR and recursive feature elimination, with an accuracy of 81.0% and a Cohen-Kappa score (CKS) of 0.280. This pipeline also showed improved multiclass classification performance on neuroblastoma cancer data, with a best area under the receiver operating characteristic (auROC) curve score of 0.872, as compared to 0.71 obtained in previous work, an improvement of 22.81%. The results and analysis reveal that feature selection techniques play a vital role in gene-expression data-based classification, and the proposed hybrid feature selection pipeline improves classification performance. Multi-class classification is possible using deep learning models, though further improvement particularly in late-stage classification is necessary and should be explored further.

Estimation of complete mutual information exploiting nonlinear magnitude-phase dependence: Application to spatial FNC for complex-valued fMRI data

In this paper, we propose a fast and versatile blind source separation including closed-form estimation of sources' probability density functions (PDFs), where the ICA's activation function is automatically adapted to various noise conditions. In the proposed method, closed-form second-order ICA and closed-form PDF estimation are introduced as a computational-cost-efficient preprocessing to extract sources' PDFs. Compared with various type of conventional ICAs, e.g., fixed activation-function type and ML-based type, our proposed algorithm can give a faster and higher convergence. Experimental assessment reveals that the proposed method is versatile for handling non-speech sound sources.

The non-line-of-sight (NLOS) propagation caused by intermittently blocking of the direct path between the base station and the mobile station is the main difficulty for mobile user location in cellular wireless communication system. So it is indispensable to investigate the techniques about the NLOS error identification and correction. In this paper, we suggest a robust and accurate approach using probability density function (PDF) estimator to identify the NLOS situation, and then an improved biased Kalman filter which depends on PDF estimator as the nonlinearity weighting coefficient in recursive operation is used to mitigate the NLOS error for time of arrival (TOA) measurements. As the PDF estimator is calculated from actual TOA samples, it is adaptive for the complex practical NLOS environments. The simulation results also indicate that with the less prior information about communication environments, the approach we present has a good location performance even in severe NLOS situations.

The error estimate of Borgonovo's moment-independent index δi is considered, and it shows that the possible computational complexity of δi is mainly due to the probability density function (PDF) estimate because the PDF estimate is an ill-posed problem and its convergence rate is quite slow. So it reminds us to compute Borgonovo's index using other methods. To avoid the PDF estimate, δi, which is based on the PDF, is first approximatively represented by the cumulative distribution function (CDF). The CDF estimate is well posed and its convergence rate is always faster than that of the PDF estimate. From the representation, a stable approach is proposed to compute δi with an adaptive procedure. Since the small probability multidimensional integral needs to be computed in this procedure, a computational strategy named asymptotic space integration is introduced to reduce a high-dimensional integral to a one-dimensional integral. Then we can compute the small probability multidimensional integral by adaptive numerical integration in one dimension with an improved convergence rate. From the comparison of numerical error analysis of some examples, it can be shown that the proposed method is an effective approach to uncertainty importance measure computation.

An efficient data fusion model based on Bayesian model averaging for robust water quality prediction using deep learning strategies

The efficiency of a binary support vector machine- (SVM-) based classifier depends on the combination and the number of input features extracted from raw signals. Sometimes, a combination of individual good features does not perform well in discriminating a class due to a high level of relevance to a second class also. Moreover, an increase in the dimensions of an input vector also degrades the performance of a classifier in most cases. To get efficient results, it is needed to input a combination of the lowest possible number of discriminating features to a classifier. In this paper, we propose a framework to improve the performance of an SVM-based classifier for sensor fault classification in two ways: firstly, by selecting the best combination of features for a target class from a feature pool and, secondly, by minimizing the dimensionality of input vectors. To obtain the best combination of features, we propose a novel feature selection algorithm that selects m out of M features having the maximum mutual information (or relevance) with a target class and the minimum mutual information with nontarget classes. This technique ensures to select the features sensitive to the target class exclusively. Furthermore, we propose a diversified-input SVM (DI-SVM) model for multiclass classification problems to achieve our second objective which is to reduce the dimensions of the input vector. In this model, the number of SVM-based classifiers is the same as the number of classes in the dataset. However, each classifier is fed with a unique combination of features selected by a feature selection scheme for a target class. The efficiency of the proposed feature selection algorithm is shown by comparing the results obtained from experiments performed with and without feature selection. Furthermore, the experimental results in terms of accuracy, receiver operating characteristics (ROC), and the area under the ROC curve (AUC-ROC) show that the proposed DI-SVM model outperforms the conventional model of SVM, the neural network, and the k-nearest neighbor algorithm for sensor fault detection and classification.

The Sixth International Symposium on Neural Networks (ISNN 2009)

Abstract. Automatic registration of multi-sensor data is a basic step in data fusion for photogrammetric and remote sensing applications. The effectiveness of intensity-based methods such as Mutual Information (MI) for automated registration of multi-sensor image has been previously reported for medical and remote sensing applications. In this paper, a new multivariable MI approach that exploits complementary information of inherently registered LiDAR DSM and intensity data to improve the robustness of registering optical imagery and LiDAR point cloud, is presented. LiDAR DSM and intensity information has been utilised in measuring the similarity of LiDAR and optical imagery via the Combined MI. An effective histogramming technique is adopted to facilitate estimation of a 3D probability density function (pdf). In addition, a local similarity measure is introduced to decrease the complexity of optimisation at higher dimensions and computation cost. Therefore, the reliability of registration is improved due to the use of redundant observations of similarity. The performance of the proposed method for registration of satellite and aerial images with LiDAR data in urban and rural areas is experimentally evaluated and the results obtained are discussed.

Multivariate online kernel density estimation with Gaussian kernels

Measuring entropy and entropy based informative functionals plays a vital role in many aspects of modern machine learning. However, recent practices often comes across problems such as ineffective of high-dimensional samples, or expensive computation and storage. These challenges in measuring could limit the implementation of machine learning methods based on information theory. To address, we proposed LogDet estimator – a reliable matrix-based entropy estimator based on the logarithm determinant of the distance matrix. We construct informative functionals of multivariate samples and design tests to ensure it is free from saturation problems in high dimensions. Besides, our method scales linearly in storage with the same computation complexity as the recently proposed α-Rényi entropy, which makes it more suitable for real-world practices of modern machine learning,such as neural network analysis, feature selection and design optmization objectives. For application, we utilize this method to analyze theinformative behaviour of neural networks, which gives a novel empirical interpretation of neural networks’ Information Bottleneck Theory.

Probability density function (PDF) estimation is a fundamentally important problem for statistical pattern recognition. Independent component analysis (ICA) can be applied to the feature vectors so that the PDF estimation of a high dimensional vector can be converted to the PDF estimation of several 1-dimensional variables. However in practice we find that this PDF is in poor generalization ability for pattern classification because of the implied noise. Hence, this paper proposes an improvement of ICA based PDF estimation method. A latent variable model is built to separate the noise from the feature vector so that the pattern information and the noise can be dealt with respectively. Based on the latent variable model, a modified ICA based PDF is deduced. The validity of our proposed method is demonstrated by the experiments of off-line handwritten numeral recognition.

Cardiovascular Disease (CVD) is one of the primary diseases that causes death every year. An approximation of roughly about 17.5 million people dies due to CVD, signifying about 31% of global deaths. Based on the statistics, every 34 seconds, a person dies due to heart disease. Various classification algorithms have been developed and utilized as classifiers to support doctors who are ineffectually diagnosed with heart disease. The main aim of this work is to improve the performance of the heart disease approach using image processing algorithm. To improve the effectiveness and efficiency of classification performance for heart disease diagnosis, an optimized neural network was proposed based on the feature extraction and selection approach for handling features. The objective of this investigation is to diagnose heart diseases using feature extraction, reduction based classification and image processing methods. The proposed model comprises of two subsets: Feature extraction using gray scale properties and Moth flame optimization (MFO) for effectual feature selection, followed by a classification technique using Generalized Regression Neural Network. The first system includes three stages: (i) Pre-processing of the dataset (ii) feature extraction (iii) performing MFO for efficient selection. In the second method, GRNN is proposed. The heart data set obtained from ACDC Challenge, was utilized for performing the computation. The image obtained from the MRI-scanner is in the NIfTI image format. The pre-process step used in this is to convert the image type from INT16 to uint8 to improve the quality of image viewing and for feature extraction process. In this phase, the texture properties from the pre-processed image are calculated and the value is in the numeric format. These values are the feature attributes of the dataset. The feature attributes of the image are given as input for the moth flame optimization process and output is the feature selected from the optimization process. The whole process is performed on the feature attributes of the image by determining the optimal feature for the classifier by reducing its error rate. The optimal feature from the moth flame optimization is used for training and testing the network. The classifier used in this approach is a single neural network classifier with a regression nature. Due to the regression property, the network is well trained with the feature. The Generalized regression neural network is used for classifying heart diseases. The proposed method achieves the accuracy of 96.23%, sensitivity of 95.41% and specificity of 96.75%. These values are calculated based on the confusion matrix of the classifier. In this, the feature extraction using the gray scale properties plays an important role to determine the feature attributes from the MRI heart images. The Moth flame optimization able to produce an accuracy of 97.23% using GRNN for classification with minimum single attribute mean of the image. It also outperformed the other methods, either the feature extraction based classification or the feature reduction based classification.

<p>Classification aims to identify a class label of an instance according to the information from its characteristics or features. Unfortunately, many classification problems have a large feature set containing irrelevant and redundant features, which reduce the classification performance. In order to address the above problem, feature selection is proposed to select a small subset of relevant features. There are three main types of feature selection methods, i.e. wrapper, embedded and filter approaches. Wrappers use a classification algorithm to evaluate candidate feature subsets. In embedded approaches, the selection process is embedded in the training process of a classification algorithm. Different from the other two approaches, filters do not involve any classification algorithm during the selection process. Feature selection is an important process but it is not an easy task due to its large search space and complex feature interactions. Because of the potential global search ability, Evolutionary Computation (EC), especially Particle Swarm Optimization (PSO), has been widely and successfully applied to feature selection. However, there is potential to improve the effectiveness and efficiency of EC-based feature selection. The overall goal of this thesis is to investigate and improve the capability of EC for feature selection to select small feature subsets while maintaining or even improving the classification performance compared to using all features. Different aspects of feature selection are considered in this thesis such as the number of objectives (single-objective/multi-objective), the fitness function (filter/wrapper), and the searching mechanism. This thesis introduces a new fitness function based on mutual information which is calculated by an estimation approach instead of the traditional counting approach. Results show that the estimation approach works well on both continuous and discrete data. More importantly, mutual information calculated by the estimation approach can capture feature interactions better than the traditional counting approach. This thesis develops a novel binary PSO algorithm, which is the first work to redefine some core concepts of PSO such as velocity and momentum to suit the characteristics of binary search spaces. Experimental results show that the proposed binary PSO algorithm evolve better solutions than other binary EC algorithms when the search spaces are large and complex. Specifically, on feature selection, the proposed binary PSO algorithm can select smaller feature subsets with similar or better classification accuracies, especially when there are a large number of features. This thesis proposes surrogate models for wrapper-based feature selection. The surrogate models use surrogate training sets which are subsets of informative instances selected from the training set. Experimental results show that the proposed surrogate models assist PSO to reduce the computational cost while maintaining or even improving the classification performance compared to using only the original training set. The thesis develops the first wrapper-based multi-objective feature selection algorithm using MOEA/D. A new decomposition strategy using multiple reference points for MOEA/D is designed, which can deal with different characteristics of multi-objective feature selection such as highly discontinuous Pareto fronts and complex relationships between objectives. The experimental results show that the proposed algorithm can evolve more diverse non-dominated sets than other multi-objective algorithms. This thesis introduces the first PSO-based feature selection algorithm for transfer learning. In the proposed algorithm, the fitness function uses classification performance to reduce the differences between domains while maintaining the discriminative ability on the target domain. The experimental results show that the proposed algorithm can select feature subsets which achieve better classification performance than four state-of-the-art feature-based transfer learning algorithms.</p>