Recovery of linear components: Reduced complexity autoencoder designs

In recent years, the data tends to be very large and complex and it becomes very difficult and tedious to work with large datasets containing huge number of features. That's where Dimensionality Reduction comes into play. Dimensionality Reduction is a pre-processing step in various fields such as machine learning, data mining, statistics etc. and is effective in removing irrelevant and highly redundant data. In this paper, the author's performed a vast literature survey and aims to provide an adequate application based understanding of various dimensionality reduction techniques and to work as a guide to choose right approach of Dimensionality Reduction for better performance in different applications. Here, the authors have also performed detailed experiments on two different datasets for comparative analysis between various linear and non-linear dimensionality reduction techniques to figure out the effectiveness of the techniques used. PCA, a linear dimensionality reduction technique, outperformed all other techniques used in the experiments. In fact, almost all the linear dimensionality reduction techniques outperformed the non-linear techniques on both datasets by a huge error percentage margin.

Analysis of population genetic variation and structure is a common practice for genome-wide studies, including association mapping, ecology, and evolution studies in several crop species. In this study, machine learning (ML) clustering methods, K-means (KM), and hierarchical clustering (HC), in combination with non-linear and linear dimensionality reduction techniques, deep autoencoder (DeepAE) and principal component analysis (PCA), were used to infer population structure and individual assignment of maize inbred lines, i.e., dent field corn (n = 97) and popcorn (n = 86). The results revealed that the HC method in combination with DeepAE-based data preprocessing (DeepAE-HC) was the most effective method to assign individuals to clusters (with 96% of correct individual assignments), whereas DeepAE-KM, PCA-HC, and PCA-KM were assigned correctly 92, 89, and 81% of the lines, respectively. These findings were consistent with both Silhouette Coefficient (SC) and Davies–Bouldin validation indexes. Notably, DeepAE-HC also had better accuracy than the Bayesian clustering method implemented in InStruct. The results of this study showed that deep learning (DL)-based dimensional reduction combined with ML clustering methods is a useful tool to determine genetically differentiated groups and to assign individuals into subpopulations in genome-wide studies without having to consider previous genetic assumptions.

Linear dimensionality reduction by maximizing the Chernoff distance in the transformed space

A theoretical analysis for comparing two linear dimensionality reduction (LDR) techniques, namely Fisher's discriminant (FD) and Loog-Duin (LD) dimensionality reduciton, is presented. The necessary and sufficient conditions for which FD and LD provide the same linear transformation are discussed and proved. To derive these conditions, it is first shown that the two criteria preserve the same maximum value after a diagonalization process is applied, and then the necessary and sufficient conditions for various cases, including coincident covariance matrices, coincident prior probabilities, and for when one of the covariances is the identity matrix. A measure for comparing the two criteria is derived from the necessary and sufficient conditions, and used to empirically show that the conditions are statistically related to the classification error for a post-processing quadratic classifier and the Chernoff distance in the transformed space.

Embedding the original high dimensional data in a low dimensional space helps to overcome the curse of dimensionality and removes noise. The aim of this work is to evaluate the performance of three different linear dimensionality reduction techniques (DR) techniques namely principal component analysis (PCA), multi dimensional scaling (MDS) and linear discriminant analysis (LDA) on classification of cardiac arrhythmias using probabilistic neural network classifier (PNN). The design phase of classification model comprises of the following stages: preprocessing of the cardiac signal by eliminating detail coefficients that contain noise, feature extraction through daubechies wavelet transform, dimensionality reduction through linear DR techniques specified, and arrhythmia classification using PNN. Linear dimensionality reduction techniques have simple geometric representations and simple computational properties. Entire MIT-BIH arrhythmia database is used for experimentation. The experimental results demonstrates that combination of PNN classifier (spread parameter, σ = 0.08) and PCA DR technique exhibits highest sensitivity and F score of 78.84% and 78.82% respectively with a minimum of 8 dimensions.

Unsupervised dimensionality reduction by jointing dynamic hypergraph and low-rank embedding for classification and clustering

In this paper, we assess whether using non-linear dimension reduction techniques pays off for forecasting inflation in real-time. Several recent methods from the machine learning literature are adopted to map a large dimensional dataset into a lower-dimensional set of latent factors. We model the relationship between inflation and the latent factors using constant and time-varying parameter (TVP) regressions with shrinkage priors. Our models are then used to forecast monthly US inflation in real-time. The results suggest that sophisticated dimension reduction methods yield inflation forecasts that are highly competitive with linear approaches based on principal components. Among the techniques considered, the Autoencoder and squared principal components yield factors that have high predictive power for one-month- and one-quarter-ahead inflation. Zooming into model performance over time reveals that controlling for non-linear relations in the data is of particular importance during recessionary episodes of the business cycle or the current COVID-19 pandemic.

A new linear dimensionality reduction (LDR) technique for pattern classification and machine learning is presented, which, though linear, aims at maximizing the Chernoff distance in the transformed space. The corresponding two-class criterion, which is maximized via a gradient-based algorithm, is presented and initialization procedures are also discussed. Empirical results of this and traditional LDR approaches combined with two well-known classifiers, linear and quadratic, on synthetic and real-life data show that the proposed criterion outperforms the traditional schemes.

Unlike classical linear dimensionality reduction techniques, nonlinear ones are capable of discovering the nonlinear degrees of freedom that are present in natural observations, by assuming that the data lie on an embedded nonlinear manifold within an observed high dimensional feature space. Nevertheless, when measured objects are actually functional data, nonlinear dimensionality reduction techniques do not produce suitable unfolded results, particularly in the locally linear embedding (LLE) algorithm. In this case, the Euclidean distance employed in cost function does not correctly represent the similarity between objects, because this distance does not take into account the intrinsical relations presented in functional data, besides, it is easily distorted by non-gaussian noise, such as an artifacts, impulsive noise, etc. The main contribution in this paper is to use (inside the LLE algorithm) a localized similarity measure called correntropy, which has a particular metric that allows it to conform to suitable neighborhoods, and indeed convenient representations of the objects, avoiding distortions.

In this paper we introduce an advanced supervised training method for neural networks. It is based on Jacobian rank deficiency and it is formulated, in some sense, in the spirit of the Gauss-Newton algorithm. The Levenberg-Marquardt algorithm, as a modified Gauss-Newton, has been used successfully in solving nonlinear least squares problems including neural-network training. It outperforms (in terms of training accuracy, convergence properties, overall training time, etc.) the basic backpropagation and its variations with variable learning rate significantly, however, with higher computation and memory complexities within each iteration. The new method developed in this paper is aiming at improving convergence properties, while reducing the memory and computation complexities in supervised training of neural networks. Extensive simulation results are provided to demonstrate the superior performance of the new algorithm over the Levenberg-Marquardt algorithm.

For purposes of dimensionality reduction in classification Linear Discriminant Analysis (LDA) is probably the most common approach. In fact, LDA is a linear dimension reduction technique that also returns a classification rule. In the case of heteroscedasticity of the classes, Quadratic Discriminant Analysis (QDA) can be used to determine an appropriate classification rule, but QDA does not serve for dimensionality reduction. Sliced Average Variance Estimation (SAVE) has been shown to be adequate in such situations as implemented in R in the package dr. This paper presents an alternative approach for linear dimensionality reduction for situations of heteroscedastic intraclass covariances, namely Heteroscedastic Discriminant Analysis (HDA) as well as its R implementation. Furthermore, tests are suggested in order to determine the dimension for the discriminative data subspace and a generalization of HDA by regularization of the covariance matrix estimates is proposed. Examples for application of HDA in R are demonstrated as well as a small simulation study turning out that HDA is preferable to SAVE in a situation where the classes differ in both means and covariances.

Background Prediction of inflammatory bowel disease (IBD) development is the subject of intense study. We have previously shown that significant changes occur in routine laboratory tests in the 5 years preceding IBD diagnosis. Using routinely collected laboratory results, we developed machine learning models to predict development of IBD. Methods We extracted data from the electronic medical records of Maccabi Health Services and included patients with IBD ≥ 16 years of age with a minimum of 5 years follow up and an age and sex matched healthy cohort. The outcome measured was entry into the IBD registry. The study population was split into training and validation cohorts and using laboratory values in the 5 years prior to IBD diagnosis as features, we assessed the performance of supervised learning models and performed interpretability analysis with unsupervised dimensionality reduction. Results 5643 patients with IBD and 17199 healthy people were included in this study. 3039 (53.8%) had Crohn’s disease (CD), 2322 (41.1%) ulcerative colitis (UC) and 282 (5%) indeterminate colitis. The mean age of the IBD and healthy cohorts at study inclusion was 39.3 ± 16.5 and 38.5 ± 15.6 years, respectively. Weighted nearest neighbour imputation was used to correct for data missingness. Unsupervised Uniform Manifold Approximation Projection dimensionality reduction correctly separated the cohort into healthy and IBD clusters (Figure 1A). Furthermore, this algorithm identified two distinct groups correlating to CD or UC diagnosis (Figure 1B). Supervised learning models, namely neural net (AUC 0.95), linear support vector machine (SVM) (AUC 0.93), radial basis function SVM (AUC 0.85) and logistic regression (AUC 0.93) models, accurately predicted IBD diagnosis (Figure 2). These models were repeated without C-reactive protein and the predictive value was not significantly changed. Moreover, we excluded data in the 2 years preceding IBD diagnosis to determine whether model performance was reduced and the predictive value remained the same. Interestingly, using only data from 3-5 years preceding IBD diagnosis, the model could better distinguish between CD and UC (Figure 3). Conclusion In this study, machine learning models could accurately predict a diagnosis of IBD using laboratory data from the 5 years preceding IBD diagnosis. Prospective studies are needed to test these models in a real world setting.

Chapter 19 - Dimensionality Reduction and Latent Variable Modeling

The majority of the classical dimensionality reduction methods can be unified into a graph-embedding-based framework. A fixed graph constructed in a high-dimensional space has been extensively employed in the graph-embedding-based dimensionality reduction methods. However, a fixed graph often cannot characterize the structure of high-dimensional data owing to the curse of dimensionality. To solve this problem, we combine graph construction and dimensionality reduction into a coherent framework. Thus, the constructed graph can be updated dynamically in dimensionality reduction. In the existing methods based on the coherent framework, graphs are usually constructed by a type of neighborhood relationship and single clustering result. This study proposes an unsupervised dimensionality reduction method guided by fusing multiple clustering results. In the proposed method, multiple clustering results are first obtained by the k-means algorithm, and then a graph is constructed using a weighted co-association matrix of fusing the clustering results to capture data distribution information. Based on the graph, we present an objective function of combining graph construction and dimensionality reduction to implement mutual guidance between them. Numerical experiments on real data sets illustrate that the proposed method achieves significant improvement over some representative and state-of-the-art unsupervised dimensionality reduction methods.

Abstract. Linear dimensionality reduction techniques, notably principal component analysis, are widely used in climate data analysis as a means to aid in the interpretation of datasets of high dimensionality. These linear methods may not be appropriate for the analysis of data arising from nonlinear processes occurring in the climate system. Numerous techniques for nonlinear dimensionality reduction have been developed recently that may provide a potentially useful tool for the identification of low-dimensional manifolds in climate data sets arising from nonlinear dynamics. Here, we apply Isomap, one such technique, to the study of El Niño/Southern Oscillation variability in tropical Pacific sea surface temperatures, comparing observational data with simulations from a number of current coupled atmosphere-ocean general circulation models. We use Isomap to examine El Niño variability in the different datasets and assess the suitability of the Isomap approach for climate data analysis. We conclude that, for the application presented here, analysis using Isomap does not provide additional information beyond that already provided by principal component analysis.