Improving the Architecture of an Autoencoder for Dimension Reduction

This paper carries on research surrounding the influences produced by dimensionality reduction on machine learning classification effect. Firstly, paper constructs the analysis architecture of data dimension reduction classification, combines the two different unsupervised dimension reduction methods, locally linear embedding (LLE) and principal component analysis (PCA) with the five machine learning classification methods: Gradient Boosting Decision Tree (GBDT), Random Forest, Support Vector Machine (SVM), K-Nearest Neighbor (KNN) and Logistic Regression. And then uses the handwritten digital identification dataset to analyze the classification performance of these five classification methods on different dimension datasets by different dimensionality reduction methods. The analysis shows that using the appropriate dimensionality reduction method for dimensionality reduction classification can effectively improve the classification accuracy; the dimensionality reduction classification effect of non-linear dimensionality reduction method is generally better than the linear dimensionality reduction method; different machine learning classification algorithms have significant differences in the sensitivity of dimensions.

Recently, many dimensionality reduction algorithms, including local methods and global methods, have been presented. The representative local linear methods are locally linear embedding (LLE) and linear preserving projections (LPP), which seek to find an embedding space that preserves local information to explore the intrinsic characteristics of high dimensional data. However, both of them still fail to nicely deal with the sparsely sampled or noise contaminated datasets, where the local neighborhood structure is critically distorted. On the contrary, principal component analysis (PCA), the most frequently used global method, preserves the total variance by maximizing the trace of feature variance matrix. But PCA cannot preserve local information due to pursuing maximal variance. In order to integrate the locality and globality together and avoid the drawback in LLE and PCA, in this paper, inspired by the dimensionality reduction methods of LLE and PCA, we propose a new dimensionality reduction method for face recognition, namely, unsupervised linear difference projection (ULDP). This approach can be regarded as the integration of a local approach (LLE) and a global approach (PCA), so that it has better performance and robustness in applications. Experimental results on the ORL, YALE and AR face databases show the effectiveness of the proposed method on face recognition.

One of the main challenges that the industry faces when dealing with massive data for failure diagnosis is high dimensionality of such data. This can be tackled by dimensionality reduction method such as principal components analysis, which usually results in an improved fault diagnosis. Other available techniques include auto-encoders and its variants denoising auto-encoders and sparse auto-encoders. Most recently, variational auto-encoders are one of the most promising techniques for unsupervised learning with successful applications in image processing and speech recognition. Differently from other auto-encoder methods, variational auto-encoders use variational inference to generate a latent representation of the data and impose a distribution over the latent variables and the data itself. In this article, we propose a fully unsupervised deep variational auto-encoder-based approach for dimensionality reduction in fault diagnosis and explore the variational auto-encoder capabilities for such a task. This is achieved by comparing the latent representations provided by variational auto-encoders to the ones from principal components analysis as well as when no reduction is performed in ball bearings’ fault classification using vibration signals. To tackle massive sensor data, we propose different architectures for the variational auto-encoder’s encoder and decoder that are based on deep neural networks and deep convolutional neural networks. Experiments are also carried out by varying the data preprocessing methods for generating spectrograms and hand-engineering features as well as the use of raw vibration data, the architecture of the neural networks fault classifiers operating on the latent representations from variational auto-encoder and principal components analysis, the degree of data dimension reduction, and the size of the available labeled data used for training the fault classifiers. The results show that variational auto-encoders are a competent and promising tool for dimensionality reduction for use in fault diagnosis and worth further exploring their capabilities beyond vibration signals of ball bearing elements.

Divergent Projection Analysis for Unsupervised Dimensionality Reduction

Dimensionality reduction plays a significant role for the performance of large-scale image retrieval. In this paper, various dimensionality reduction methods are compared to validate their own performance in image retrieval. For this purpose, first, the Scale Invariant Feature Transform (SIFT) features and HSV (Hue, Saturation, Value) histogram are extracted as image features. Second, the Principal Component Analysis (PCA), Fisher Linear Discriminant Analysis (FLDA), Local Fisher Discriminant Analysis (LFDA), Isometric Mapping (ISOMAP), Locally Linear Embedding (LLE), and Locality Preserving Projections (LPP) are respectively applied to reduce the dimensions of SIFT feature descriptors and color information, which can be used to generate vocabulary trees. Finally, through setting the match weights of vocabulary trees, large-scale image retrieval scheme is implemented. By comparing multiple sets of experimental data from several platforms, it can be concluded that dimensionality reduction method of LLE and LPP can effectively reduce the computational cost of image features, and maintain the high retrieval performance as well.

Dimensionality reduction is an unsupervised learning task aimed at creating a low-dimensional summary and/or extracting the most salient features of a dataset. Principal component analysis is a linear dimensionality reduction method in the sense that each principal component is a linear combination of the input variables. To allow features that are nonlinear functions of the input variables, many nonlinear dimensionality reduction (NLDR) methods have been proposed. In this article, we propose novel NLDR methods based on bottleneck deep autoencoders. Our contributions are 2-fold: (1) We introduce a monotonicity constraint into bottleneck deep autoencoders for estimating a single nonlinear component and propose two methods for fitting the model. (2) We propose a new, forward stepwise deep learning architecture for estimating multiple nonlinear components. The former helps extract interpretable, monotone components when the assumption of monotonicity holds, and the latter helps evaluate reconstruction errors in the original data space for a range of components. We conduct numerical studies to compare different model fitting methods and use two real data examples from the studies of human immune responses to HIV to illustrate the proposed methods. Supplementary materials for this article are available online.

Modern manufacturing processes are often characterized by high dimensionality and nonlinearity; dimension reduction is usually required to identify the critical features and help improve the process yield. Dimension reduction and analysis of data with high nonlinearity has been a challenging task. Many methods have been proposed based on manifold learning, which aims to learn a lower-dimensional manifold from the original high-dimensional space. However, since the mappings are usually implicit, it is difficult to build a connection between the classification results and the original features. Apart from the prediction results, industry data analysis also focus on the features themselves. Therefore, methods that are able to combine feature selection with dimension reduction is often needed. This paper proposed a hybrid method for nonlinear dimension reduction and feature selection based on Locally Linear Embedding (LLE). LLE and many filter feature selection shares a common procedure of neighbor search. By integrating nearest neighbor search for both LLE and ReliefF, and adjusting the distance measure in supervised LLE with results from feature selection, the proposed method could connect the feature selection process with the nonlinear model, extract the critical features for further analysis, and improve the performance of process modelling. Two dataset from real industry processes are also illustrated as case studies.

Underground metal target detection technology has been widely applied in industrial production, resource exploration, and engineering construction, etc. However, due to the influence of non-negligible noise and high dimensionality in collected data, achieving efficient and accurate underground target classification remains a grand challenge for further applications of underground metal target detection on portable devices with limited computing capability and energy supply. This study aimed to seek out robust and efficient data-based strategies to classify the underground metal targets of different shapes and materials based on electromagnetic induction detection. We investigated thirty-three classification strategies based on eleven dimensionality reduction methods, namely, the least absolute shrinkage and selection operator (LASSO), genetic algorithm-support vector machine (GA-SVM), Pearson correlation coefficient (PCC), mutual information (MI), maximal relevance minimal redundancy Pearson correlation (mRMRP), maximal relevance minimal redundancy mutual information (mRMRMI), statistical features (SF), principal component analysis (PCA), kernel principal component analysis (KPCA), locally linear embedding (LLE), and stacked denoising autoencoder (SDAE), and three machine learning models, namely, artificial neural network (ANN), linear support vector machine (L-SVM), and Gaussian Naïve Bayes (GNB). Several parameters, including classification accuracy, the number of features after dimensionality, the feature type importance, and the time consumption were considered to evaluate the data-based classification strategies. Among the classification strategies investigated and considering the above evaluation parameters, the artificial neural network (ANN) classifier assisted with the kernel principal component analysis (KPCA) feature extraction method yielded the best performance in the material-based classification (accuracy:0.99) and the shape-based classification (accuracy:0.99). The locally linear embedding (LLE) improved the robustness of machine learning classifiers and efficiency of the artificial neural network in the material-based classification (improvement of average accuracy:0.17, reduction of classification time cost:14%) and shape-based classification (improvement of average accuracy:0.16, reduction of classification time cost:22%). Our comparative investigation provides a robust and efficient data-based strategy for underground metal target classification, which is significant for applications of underground metal target detection on portable devices with limited computing capability and energy supply. The cross-combination strategy of dimensionality reduction methods and machine learning models provides a way to find the optimal machine learning model for underground target detection.

Identifying collective variables (CVs) for chemical reactions is essential to reduce the 3N-dimensional energy landscape into lower dimensional basins and barriers of interest. However, in condensed phase processes, the nonmeaningful motions of bulk solvent often overpower the ability of dimensionality reduction methods to identify correlated motions that underpin collective variables. Yet solvent can play important indirect or direct roles in reactivity, and much can be lost through treatments that remove or dampen solvent motion. This has been amply demonstrated within principal component analysis (PCA), although less is known about the behavior of nonlinear dimensionality reduction methods, e.g., uniform manifold approximation and projection (UMAP), that have become recently utilized. The latter presents an interesting alternative to linear methods though often at the expense of interpretability. This work presents distance-attenuated projection methods of atomic coordinates that facilitate the application of both PCA and UMAP to identify collective variables in the presence of explicit solvent and further the specific identity of solvent molecules that participate in chemical reactions. The performance of both methods is examined in detail for two reactions where the explicit solvent plays very different roles within the collective variables. When applied to raw molecular dynamics data in solution, both PCA and UMAP representations are dominated by bulk solvent motions. On the other hand, when applied to data preprocessed by our attenuated projection methods, both PCA and UMAP identify the appropriate collective variables (though varying sensitivity is observed due to the presence of explicit solvent that results from the projection method). Importantly, this approach allows identification of specific solvent molecules that are relevant to the CVs and their importance.

Statistical methods have been getting constant development since 1970s. However, the statistical methods of the big data are no longer restricted with these methods which are listed in the textbook. This paper mainly demonstrates the Discrimination Analysis of Multivariate Statistical Analysis, Linear Dimensionality Reduction and Nonlinear Dimensionality Reduction Method under the circumstances of the wide range of applications of high-dimensional data. This paper includes three parts. To begin with, the paper illustrates a developing trend from the data to the high-dimensional. Meanwhile, it analyzes the impacts of the high-dimensional data on discriminate analysis methods. The second part represents the necessity of the dimensionality reduction studies in the era of the high-dimensional data overflowing. Then, the paper focuses on introducing the main methods of the linear dimensionality reduction. In addition, this paper covers the basic idea of the nonlinear dimensionality reduction. Moreover, it systematically analyzes the breakthrough of the traditional methods. Furthermore, it chronologically demonstrates the developing trend of the dimensionality reduction method. The final part shows a comprehensive and systematic conclusion to the whole essay and describes a developing prospect of the dimension reduction methods in the future. The purpose of this essay is to design a framework of a performance system which is subject to the characteristics of China High-tech enterprises. It based on the analysing the principles and significance of the performance system of High-tech enterprises. The framework will promote the standardize management of High-tech enterprises of China.

Stacked generalization (SG) is a hierarchical architecture, which combines classifiers in order to boost the performance of overall system by integrating the individual classifiers. Two-layered version of SG has been utilized in many image analysis researches and shown to be an effective tool for enhancing the performance measure of the individual classifiers in a number of various cases. However, due to its layered architecture and integration approach, it inherits some problems such as curse of dimensionality, i.e. necessity for more samples, and higher computational time for classification. In this work, the effect of two different dimensionality reduction methods, namely linear and non-linear, between layers of SG are analyzed to attack the curse of dimensionality problem. In the experimental part, the comparisons of these approaches with respect to each other and SG-without-dimensionality-reduction are presented. According to test results, it is concluded that dimensionality reduction has a positive effect on the performance of SG, i.e. both linear and nonlinear dimensionality reduction approaches performs better than SG architecture, and nonlinear dimensionality reduction achieves better classification accuracy than linear dimensionality reduction.

During the COVID-19 pandemic, researchers have made efforts to detect COVID-19 through various methods. In the dataset used for this study, COVID-19 patients were identified using chest computed tomography (CT) images. High dimensionality is frequently an issue in machine learning image classification. Accordingly, this study implemented three dimensionality reduction methods in combination with various machine learning algorithms for improved classification. Principal component analysis (PCA), uniform manifold approximation and projection (UMAP), and diffusion maps were applied to the dataset to extract the most important features of the chest CT images. The extracted features were given as input either to logistic regression or the extreme gradient boosting (XGBoost) algorithm to perform classification. The strongest model identified from this study was diffusion maps in combination with logistic regression. This model, evaluated against existing models from similar studies in recent years, yielded strong performance for detecting COVID-19 cases using chest CT images. Our proposed model achieved 97.35% accuracy, 92.16% sensitivity, and 98.59% specificity on the held-out test set in differentiating between COVID-19-positive cases and healthy, non-COVID-19 cases. This study aimed to detect COVID-19 without the use of viral testing. Importantly, this method could assist clinicians in making an initial diagnosis, especially when viral testing is not available or timely enough for the patient’s case. This study also provides deeper insight into various dimensionality reduction methods and how compatible they are with biomedical imaging data. Models were trained using stratified cross-validation on the training set, with final performance evaluated on a held-out test set at the patient level to prevent data leakage. Additional imbalance-aware metrics were used to assess robustness given class distribution differences.

Local appearance-based methods have been proposed recently for face recognition. We analyze the effects of different dimension reduction and normalization methods on local appearance-based face recognition in this paper. Each image is divided into equal sized blocks and six different dimension reduction methods are implemented for each block separately to create local visual feature vectors. On these local features, several normalization methods are applied in an attempt to eliminate the changes in lighting conditions and contrast differences among blocks of different face images. The experimental results show the improvements in recognition rates due to the effects of dimension reduction and normalization for three different classifiers. Usage of trainable dimension reduction methods instead of DCT and a new normalization method in our work (within-block normalization as referred in this paper) are two factors that makes difference from previous works in literature. The best performance is achieved using a block size of 16times16, performing dimension reduction using approximate pairwise accuracy criterion (aPAC) and applying within-block mean and variance normalization.

Detection of maize seed germination rate based on improved locally linear embedding

Local Linear Embedding with Adaptive Neighbors