Kernel ridge regression for out-of-sample mapping in supervised manifold learning

This study contributes to classification and recognition of hyperspectral image in industrial sorting, mineral identification, geological exploration, environmental survey, and other industrial production fields. The finding eliminates the redundancy of hyperspectral data and improves the classification accuracy of hyperspectral image. Hyperspectral remote sensing data are inherently nonlinear, and traditional linear dimensionality reduction methods cannot effectively reveal the nonlinear structure contained in the data. Manifold learning is a nonlinear dimensionality reduction method which has been developed rapidly in recent years. The intrinsic structure of the data is preserved by Isometric Feature Mapping (ISOMAP), but it is sensitive to noise and is not conducive to final classification. Prior studies neglected a higher dimensional manifold real structure in hyperspectral data dimensionality reduction due to lack of consideration credibility information and class information of samples in calculating manifold distance. This study proposes a Robust Supervised ISOMAP method. Data sample credibility mode is constructed, and a data similarity measure named triple geodesic distance is defined by introducing samples’ credibility information, class information, and neighborhood information. Triple geodesic distance fitting is proven. Multidimensional scaling analysis (MDS) and generalized regression neural network are used to construct the embedded coordinates of training samples and test samples. The results show that the proposed method has higher anti-noise performance than the traditional one. Also, the runtime is feasible, and can effectively improve the classification precision of the hyperspectral images, in comparison to traditional ISOMAP methods. The conclusion is that Robust Supervised ISOMAP is an effective nonlinear dimensionality reduction method for hyperspectral remote sensing image.

In recent years, a variety of supervised manifold learning techniques have been proposed to outperform their unsupervised alternative versions in terms of classification accuracy and data structure capturing. Some dissimilarity measures have been used in these techniques to guide the dimensionality reduction process. Their good performance was empirically demonstrated; however, the relevant analysis is still missing. This paper contributes to a theoretical analysis on a) how dissimilarity measures affect maintaining manifold neighbourhood structure and b) how supervised manifold learning techniques could contribute to the reduction of classification error. This paper also provides a cross-comparison between supervised and unsupervised manifold learning approaches in terms of structure capturing using Kendall's Tau coefficients and co-ranking matrices. Four different metrics (including three dissimilarity measures and Euclidean distance) have been considered along with manifold learning methods such as Isomap, t-Stochastic Neighbour Embedding ( t-SNE), and Laplacian Eigenmaps (LE), in two datasets: Breast Cancer and Swiss-Roll. This paper concludes that although the dissimilarity measures used in the manifold learning techniques can reduce classification error, they do not learn well or preserve the structure of the hidden manifold in the high dimensional space, but instead, they destroy the structure of the data. Based on the findings of this paper, it is advisable to use supervised manifold learning techniques as a pre-processing step in classification. In addition, it is not advisable to apply supervised manifold learning for visualization purposes since the two-dimensional representation using supervised manifold learning does not improve the preservation of data structure.

A mixed-integer programming approach to GRNN parameter estimation

This paper carries on research surrounding the influences produced by dimensionality reduction on machine learning classification effect. Firstly, paper constructs the analysis architecture of data dimension reduction classification, combines the two different unsupervised dimension reduction methods, locally linear embedding (LLE) and principal component analysis (PCA) with the five machine learning classification methods: Gradient Boosting Decision Tree (GBDT), Random Forest, Support Vector Machine (SVM), K-Nearest Neighbor (KNN) and Logistic Regression. And then uses the handwritten digital identification dataset to analyze the classification performance of these five classification methods on different dimension datasets by different dimensionality reduction methods. The analysis shows that using the appropriate dimensionality reduction method for dimensionality reduction classification can effectively improve the classification accuracy; the dimensionality reduction classification effect of non-linear dimensionality reduction method is generally better than the linear dimensionality reduction method; different machine learning classification algorithms have significant differences in the sensitivity of dimensions.

Electronic noses (E-Noses) are becoming popular for food and fruit quality assessment due to their robustness and repeated usability without fatigue, unlike human experts. An E-Nose equipped with classification algorithms and having open ended classification boundaries such as the k-nearest neighbor (k-NN), support vector machine (SVM), and multilayer perceptron neural network (MLPNN), are found to suffer from false classification errors of irrelevant odor data. To reduce false classification and misclassification errors, and to improve correct rejection performance; algorithms with a hyperspheric boundary, such as a radial basis function neural network (RBFNN) and generalized regression neural network (GRNN) with a Gaussian activation function in the hidden layer should be used. The simulation results presented in this paper show that GRNN has more correct classification efficiency and false alarm reduction capability compared to RBFNN. As the design of a GRNN and RBFNN is complex and expensive due to large numbers of neuron requirements, a simple hyperspheric classification method based on minimum, maximum, and mean (MMM) values of each class of the training dataset was presented. The MMM algorithm was simple and found to be fast and efficient in correctly classifying data of training classes, and correctly rejecting data of extraneous odors, and thereby reduced false alarms.

Machines learning models have recently been proposed for predicting rivers water temperature (Tw) using only air temperature (Ta). The proposed models relied on a nonlinear relationship between the Tw and Ta and they have proven to be robust modelling tools. The main motivation for this study was to evaluate how the variational mode decomposition (VMD) contributed to the improvement of machines learning performances for river Tw modelling. Measured data collected at five stations located in Poland from 1987 to 2014 were acquired and used for the analysis. Six machines learning models were used and compared namely, K-nearest neighbor's regression (KNNR), least square support vector machine (LSSVM), generalized regression neural network (GRNN), cascade correlation artificial neural networks (CCNN), relevance vector machine (RVM), and locally weighted polynomials regression (LWPR). The six models were developed according to three scenarios. First, the models were calibrated using only the Ta as input and obtained results show that the models were able to predict consistently water temperature, showing a high determination coefficient (R2) and Nash-Sutcliffe efficiency (NSE) with values near or above 0.910 and 0.915, respectively, and in overall the six models worked equally without clear superiority of one above another. Second, the air temperature was combined with the periodicity (i.e., day, month and year number) as input variable and a significant improvement was achieved. Both models show their ability to accurately predict river Tw with an overall accuracy of 0.956 for R2 and 0.955 for NSE values, but the LSSVM2 have some advantages such as a small errors metrics, and high fitting capabilities and it slightly surpasses the others models. Thirdly, air temperature was decomposed into several intrinsic mode functions (IMF) using the VMD method and the performances of the models were evaluated. The VMD parameters appeared to cause much influence on the prediction accuracy, exhibiting an improvement of about 40.50% and 39.12% in terms of RMSE and MAE between the first and the third scenarios, however, some models, i.e., GRNN and KNNR have not benefited from the VMD. This research has demonstrated the high capability of the VMD algorithm as a preprocessing approach in improving the accuracies of the machine learning models for river water temperature prediction.

Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression

Manifold learning approaches seek the intrinsic, low-dimensional data structure within a high-dimensional space. Mainstream manifold learning algorithms, such as Isomap, UMAP, t-SNE, Diffusion Map, and Laplacian Eigenmaps do not use data labels and are thus considered unsupervised. Existing supervised extensions of these methods are limited to classification problems and fall short of uncovering meaningful embeddings due to their construction using order non-preserving, class-conditional distances. In this paper, we show the weaknesses of class-conditional manifold learning quantitatively and visually and propose an alternate choice of kernel for supervised dimensionality reduction using a data-geometry-preserving variant of random forest proximities as an initialization for manifold learning methods. We show that local structure preservation using these proximities is near universal across manifold learning approaches and global structure is properly maintained using diffusion-based algorithms.

This paper proposes an approach for phoneme classification based on supervised manifold learning. It has been shown that speech sounds exist on a low dimensional manifold nonlinearly embedded in high dimensional space and the manifold learning technique can get high phoneme classification accuracy. To improve the performance of phoneme classification, the proposed algorithm calculates the supervised geodesic distance using the minimum distance and the set distance of different class points to enhance the discriminability of low-dimensional embedded data. Experiments show that the proposed algorithm can significantly improve phoneme classification compared to the baseline features.

Unsupervised dimensionality reduction by jointing dynamic hypergraph and low-rank embedding for classification and clustering

Spinal cord injuries (SCI) occur when the spinal cord is damaged due to any trauma. Treatment of this condition typically involves a long and challenging rehabilitation process. Brain-computer interface (BCI) controlled rehabilitation systems show promise for people with SCI, as they offer repetitive and controlled treatment in the home environment without requiring a specialist. There is a growing demand for electroencephalography (EEG)–based BCI rehabilitation systems, particularly for patients with SCI. In this study, EEG signals from ten SCI patients were analyzed. At the same time, they imagined performing rehabilitation movements for five different hand and arm actions (pronation, supination, palmar grasp, lateral grasp, and hand opening). The study tested manifold learning algorithms, feature extraction, and classification methods in EEG analysis to improve usability in real-time applications. Manifold learning algorithms were used to represent complex and high-dimensional data in a lower-dimensional space, allowing for better representation and separation of the temporal and spatial characteristics of brain activity. The spectral embedding algorithm was used in this study, which, to the best of our knowledge, is the first time this algorithm has been applied to the data of SCI patients. Additionally, we conducted a comparative analysis of commonly encountered methods in the literature, including multi-dimensional scaling (MDS), isometric feature mapping (ISOMAP), local linear embedding (LLE), and t-distributed stochastic neighbor embedding (t-SNE). Machine learning algorithms, such as k-nearest neighbor (k-NN), support vector machine (SVM), and Naive Bayes methods, were used to obtain classification results for both multi-class and binary classes. The combination of pronation and hand open was found to yield the best performance for binary classification of the movements. The study determined that the ISOMAP manifold learning method with the k-NN algorithm is the optimal method for processing times of both the train and the test. The method also demonstrated a high accuracy value of 0.967 and a short time of 0.088 units in multiple classifications, which is promising. The study utilized the spectral embedding method for the first time and achieved an accuracy rate of 0.649 in multi-class classification and 0.933 ± 0.049 in binary-class classification.

Improving MRI-based analysis of brain structural changes in patients with hypertension via a privileged information learning algorithm.

Zelenkov et al. [1] proposed a two-step classification method (TSCM) based on genetic algorithm to predict the bankruptcy of Russian companies. This current study was conducted to do more comprehensive evaluation than Zelenkov et al. [1] did. We involved more datasets, compared to a greater number of competitive methods, and developed the ensemble using more base classifiers. The datasets consisted of nine datasets and then the result of prediction compared to previous studies from 13 papers that were published between 1996 and 2009. In addition, the previous method using five base classifiers, this study involved seven base classifiers. The purpose of this study is to compare the accuracy of prediction from previous studies with this method. In this study, we used k-Nearest Neighbour (k-NN), Logistic Regression (LR), Naïve Bayes (NB), Decision Tree (DT), support vector machine (SVM), random forest (RF), and boosting as the base classifiers of ensemble model. Genetic algorithm is used to find the best predictor variables and the best weight for each base classifier. The result of TSCM shows that the accuracy of prediction can increase about 0-34% for some datasets compared to the previous studies.

While manifold learning algorithms can discover intrinsic low-dimensional manifold embedded in the high-dimensional Euclidean space, the discriminant ability of the low-dimensional subspaces obtained by the algorithms is often lower than those obtained by the conventional dimensionality reduction approaches. Furthermore, the original feature vectors may include redundancy such as high-order correlation which cannot be removed by manifold learning algorithms. To address the two problems, we first employ Gabor wavelet to remove intrinsic redundancies of images and obtain a set of over-completed feature vectors. Then a supervised manifold learning algorithm (ULLELDA) is applied to project Gabor-based data and out-of-the-samples into a common low-dimensional subspace. Experiments in two FERET face databases indicate that Gabors indeed help supervised manifold learning to remarkably improve the discriminant ability of low-dimensional subspaces.KeywordsFace RecognitionLinear Discriminant AnalysisDiscriminant AbilityFace DatabaseGabor WaveletThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Laser-induced breakdown spectroscopy (LIBS) is an important analysis technique with applications in many industrial branches and fields of scientific research. Nowadays, the advantages of LIBS are impaired by the main drawback in the interpretation of obtained spectra and identification of observed spectral lines. This procedure is highly time-consuming since it is essentially based on the comparison of lines present in the spectrum with the literature database. This paper proposes the use of various computational intelligence methods to develop a reliable and fast classification of quasi-destructively acquired LIBS spectra into a set of predefined classes. We focus on a specific problem of classification of paper-ink samples into 30 separate, predefined classes. For each of 30 classes (10 pens of each of 5 ink types combined with 10 sheets of 5 paper types plus empty pages), 100 LIBS spectra are collected. Four variants of preprocessing, seven classifiers (decision trees, random forest, k-nearest neighbor, support vector machine, probabilistic neural network, multi-layer perceptron, and generalized regression neural network), 5-fold stratified cross-validation, and a test on an independent set (for methods evaluation) scenarios are employed. Our developed system yielded an accuracy of 99.08%, obtained using the random forest classifier. Our results clearly demonstrates that machine learning methods can be used to identify the paper-ink samples based on LIBS reliably at a faster rate.