Utilization of dimensionality reduction in stacked generalization architecture

This paper carries on research surrounding the influences produced by dimensionality reduction on machine learning classification effect. Firstly, paper constructs the analysis architecture of data dimension reduction classification, combines the two different unsupervised dimension reduction methods, locally linear embedding (LLE) and principal component analysis (PCA) with the five machine learning classification methods: Gradient Boosting Decision Tree (GBDT), Random Forest, Support Vector Machine (SVM), K-Nearest Neighbor (KNN) and Logistic Regression. And then uses the handwritten digital identification dataset to analyze the classification performance of these five classification methods on different dimension datasets by different dimensionality reduction methods. The analysis shows that using the appropriate dimensionality reduction method for dimensionality reduction classification can effectively improve the classification accuracy; the dimensionality reduction classification effect of non-linear dimensionality reduction method is generally better than the linear dimensionality reduction method; different machine learning classification algorithms have significant differences in the sensitivity of dimensions.

Statistical methods have been getting constant development since 1970s. However, the statistical methods of the big data are no longer restricted with these methods which are listed in the textbook. This paper mainly demonstrates the Discrimination Analysis of Multivariate Statistical Analysis, Linear Dimensionality Reduction and Nonlinear Dimensionality Reduction Method under the circumstances of the wide range of applications of high-dimensional data. This paper includes three parts. To begin with, the paper illustrates a developing trend from the data to the high-dimensional. Meanwhile, it analyzes the impacts of the high-dimensional data on discriminate analysis methods. The second part represents the necessity of the dimensionality reduction studies in the era of the high-dimensional data overflowing. Then, the paper focuses on introducing the main methods of the linear dimensionality reduction. In addition, this paper covers the basic idea of the nonlinear dimensionality reduction. Moreover, it systematically analyzes the breakthrough of the traditional methods. Furthermore, it chronologically demonstrates the developing trend of the dimensionality reduction method. The final part shows a comprehensive and systematic conclusion to the whole essay and describes a developing prospect of the dimension reduction methods in the future. The purpose of this essay is to design a framework of a performance system which is subject to the characteristics of China High-tech enterprises. It based on the analysing the principles and significance of the performance system of High-tech enterprises. The framework will promote the standardize management of High-tech enterprises of China.

Detection of maize seed germination rate based on improved locally linear embedding

Underground metal target detection technology has been widely applied in industrial production, resource exploration, and engineering construction, etc. However, due to the influence of non-negligible noise and high dimensionality in collected data, achieving efficient and accurate underground target classification remains a grand challenge for further applications of underground metal target detection on portable devices with limited computing capability and energy supply. This study aimed to seek out robust and efficient data-based strategies to classify the underground metal targets of different shapes and materials based on electromagnetic induction detection. We investigated thirty-three classification strategies based on eleven dimensionality reduction methods, namely, the least absolute shrinkage and selection operator (LASSO), genetic algorithm-support vector machine (GA-SVM), Pearson correlation coefficient (PCC), mutual information (MI), maximal relevance minimal redundancy Pearson correlation (mRMRP), maximal relevance minimal redundancy mutual information (mRMRMI), statistical features (SF), principal component analysis (PCA), kernel principal component analysis (KPCA), locally linear embedding (LLE), and stacked denoising autoencoder (SDAE), and three machine learning models, namely, artificial neural network (ANN), linear support vector machine (L-SVM), and Gaussian Naïve Bayes (GNB). Several parameters, including classification accuracy, the number of features after dimensionality, the feature type importance, and the time consumption were considered to evaluate the data-based classification strategies. Among the classification strategies investigated and considering the above evaluation parameters, the artificial neural network (ANN) classifier assisted with the kernel principal component analysis (KPCA) feature extraction method yielded the best performance in the material-based classification (accuracy:0.99) and the shape-based classification (accuracy:0.99). The locally linear embedding (LLE) improved the robustness of machine learning classifiers and efficiency of the artificial neural network in the material-based classification (improvement of average accuracy:0.17, reduction of classification time cost:14%) and shape-based classification (improvement of average accuracy:0.16, reduction of classification time cost:22%). Our comparative investigation provides a robust and efficient data-based strategy for underground metal target classification, which is significant for applications of underground metal target detection on portable devices with limited computing capability and energy supply. The cross-combination strategy of dimensionality reduction methods and machine learning models provides a way to find the optimal machine learning model for underground target detection.

Modern manufacturing processes are often characterized by high dimensionality and nonlinearity; dimension reduction is usually required to identify the critical features and help improve the process yield. Dimension reduction and analysis of data with high nonlinearity has been a challenging task. Many methods have been proposed based on manifold learning, which aims to learn a lower-dimensional manifold from the original high-dimensional space. However, since the mappings are usually implicit, it is difficult to build a connection between the classification results and the original features. Apart from the prediction results, industry data analysis also focus on the features themselves. Therefore, methods that are able to combine feature selection with dimension reduction is often needed. This paper proposed a hybrid method for nonlinear dimension reduction and feature selection based on Locally Linear Embedding (LLE). LLE and many filter feature selection shares a common procedure of neighbor search. By integrating nearest neighbor search for both LLE and ReliefF, and adjusting the distance measure in supervised LLE with results from feature selection, the proposed method could connect the feature selection process with the nonlinear model, extract the critical features for further analysis, and improve the performance of process modelling. Two dataset from real industry processes are also illustrated as case studies.

Recently, many dimensionality reduction algorithms, including local methods and global methods, have been presented. The representative local linear methods are locally linear embedding (LLE) and linear preserving projections (LPP), which seek to find an embedding space that preserves local information to explore the intrinsic characteristics of high dimensional data. However, both of them still fail to nicely deal with the sparsely sampled or noise contaminated datasets, where the local neighborhood structure is critically distorted. On the contrary, principal component analysis (PCA), the most frequently used global method, preserves the total variance by maximizing the trace of feature variance matrix. But PCA cannot preserve local information due to pursuing maximal variance. In order to integrate the locality and globality together and avoid the drawback in LLE and PCA, in this paper, inspired by the dimensionality reduction methods of LLE and PCA, we propose a new dimensionality reduction method for face recognition, namely, unsupervised linear difference projection (ULDP). This approach can be regarded as the integration of a local approach (LLE) and a global approach (PCA), so that it has better performance and robustness in applications. Experimental results on the ORL, YALE and AR face databases show the effectiveness of the proposed method on face recognition.

The recent explosion in procurement and availability of high-dimensional gene- and protein-expression profile datasets for cancer diagnostics has necessitated the development of sophisticated machine learning tools with which to analyze them. A major limitation in the ability to accurate classify these high-dimensional datasets stems from the 'curse of dimensionality', occurring in situations where the number of genes or peptides significantly exceeds the total number of patient samples. Previous attempts at dealing with this issue have mostly centered on the use of a dimensionality reduction (DR) scheme, Principal Component Analysis (PCA), to obtain a low-dimensional projection of the high-dimensional data. However, linear PCA and other linear DR methods, which rely on Euclidean distances to estimate object similarity, do not account for the inherent underlying nonlinear structure associated with most biomedical data. The motivation behind this work is to identify the appropriate DR methods for analysis of high-dimensional gene- and protein-expression studies. Towards this end, we empirically and rigorously compare three nonlinear (Isomap, Locally Linear Embedding, Laplacian Eigenmaps) and three linear DR schemes (PCA, Linear Discriminant Analysis, Multidimensional Scaling) with the intent of determining a reduced subspace representation in which the individual object classes are more easily discriminable.

Divergent Projection Analysis for Unsupervised Dimensionality Reduction

Dimensionality reduction is an unsupervised learning task aimed at creating a low-dimensional summary and/or extracting the most salient features of a dataset. Principal component analysis is a linear dimensionality reduction method in the sense that each principal component is a linear combination of the input variables. To allow features that are nonlinear functions of the input variables, many nonlinear dimensionality reduction (NLDR) methods have been proposed. In this article, we propose novel NLDR methods based on bottleneck deep autoencoders. Our contributions are 2-fold: (1) We introduce a monotonicity constraint into bottleneck deep autoencoders for estimating a single nonlinear component and propose two methods for fitting the model. (2) We propose a new, forward stepwise deep learning architecture for estimating multiple nonlinear components. The former helps extract interpretable, monotone components when the assumption of monotonicity holds, and the latter helps evaluate reconstruction errors in the original data space for a range of components. We conduct numerical studies to compare different model fitting methods and use two real data examples from the studies of human immune responses to HIV to illustrate the proposed methods. Supplementary materials for this article are available online.

Local Linear Embedding with Adaptive Neighbors

Dimension reduction is used by scientists to deal with huge amount of high-dimensional data because of the "curse of dimensionality". There exist many methods of dimension reduction, such as principal components analysis (PCA), Locally Linear Embedding (LLE), Stochastic Neighbor Embedding (SNE), etc. Auto encoder is also applied for dimension reduction recently. It uses deep learning to train the network and has been applied in image reconstruction successfully. However, one important problem in auto encoder application is how to find the best architecture of the network. In this paper, we propose an improved architecture of the auto encoder for dimension reduction. The experimental results show the effectiveness of the proposed method.

High-dimensional features often cause computational complexity and dimensionality curse. Feature selection and feature extraction are the two mainstream methods for dimensionality reduction. Feature selection but not feature extraction can preserve the critical information and maintain the physical meaning simultaneously. Herein, we proposed a dimensionality reduction method based on rough set theory (DRM-RST) for feature selection. We defined the hyperspectral image as a decision system, extracted the features as decision attributes, and selected the effective features based on information entropy. We used the Washington D.C. Mall dataset and New York dataset to evaluate the performance of DRM-RST on dimensionality reduction. Compared with full band classification, 184 or 185 redundant bands were removed in DRM-RST, respectively. DRM-RST achieved similar accuracy (overall accuracy >94%) by SVM classifier and reduced computing time by about 85%. We further compared the dimensionality reduction efficiency of DRM-RST against other popular methods, including ReliefF, Sequential Backward Elimination (SBE) and Information Gain (IG). The Producer’s accuracy (PA) and User’s accuracy (UA) of DRM-RST was greater than that of ReliefF and IG. DRM-RST showed greater stability of accuracy than SBE in dimensionality reduction when using for different datasets. Collectively, this study provides a new method for dimensionality reduction that can reduce computational complexity and alleviate dimensionality curse.

Dimensionality reduction plays a significant role for the performance of large-scale image retrieval. In this paper, various dimensionality reduction methods are compared to validate their own performance in image retrieval. For this purpose, first, the Scale Invariant Feature Transform (SIFT) features and HSV (Hue, Saturation, Value) histogram are extracted as image features. Second, the Principal Component Analysis (PCA), Fisher Linear Discriminant Analysis (FLDA), Local Fisher Discriminant Analysis (LFDA), Isometric Mapping (ISOMAP), Locally Linear Embedding (LLE), and Locality Preserving Projections (LPP) are respectively applied to reduce the dimensions of SIFT feature descriptors and color information, which can be used to generate vocabulary trees. Finally, through setting the match weights of vocabulary trees, large-scale image retrieval scheme is implemented. By comparing multiple sets of experimental data from several platforms, it can be concluded that dimensionality reduction method of LLE and LPP can effectively reduce the computational cost of image features, and maintain the high retrieval performance as well.

Dimension reduction has become an important tool for dealing with high dimensional data. Locally linear embedding (LLE) is a nonlinear dimension reduction method which can preserve local configurations of nearest neighbors. However, finding the nearest neighbors requires the definition of a distance measure, which is a critical step in LLE. In this paper, the Rank-order distance measure is used to substitute the traditional Euclidean distance measure in order to find better nearest neighbor candidates for preserving local configurations of the manifolds. The Rank-order distance between the data points is calculated using their neighbors’ ranking orders, and is shown to be able to improve the clustering of high dimensional data. The proposed method is called Rank-order based LLE (RLLE). The RLLE method is evaluated by comparing with the original LLE, ISO-LLE and IED-LLE on two handwritten datasets. It is shown that the effectiveness of a distance measure in the LLE method is closely related to whether it can be used to find good nearest neighbors. The experimental results show that the proposed RLLE method can improve the process of dimension reduction effectively, and C-index is another good candidate for evaluating the dimension reduction results.

Identifying collective variables (CVs) for chemical reactions is essential to reduce the 3N-dimensional energy landscape into lower dimensional basins and barriers of interest. However, in condensed phase processes, the nonmeaningful motions of bulk solvent often overpower the ability of dimensionality reduction methods to identify correlated motions that underpin collective variables. Yet solvent can play important indirect or direct roles in reactivity, and much can be lost through treatments that remove or dampen solvent motion. This has been amply demonstrated within principal component analysis (PCA), although less is known about the behavior of nonlinear dimensionality reduction methods, e.g., uniform manifold approximation and projection (UMAP), that have become recently utilized. The latter presents an interesting alternative to linear methods though often at the expense of interpretability. This work presents distance-attenuated projection methods of atomic coordinates that facilitate the application of both PCA and UMAP to identify collective variables in the presence of explicit solvent and further the specific identity of solvent molecules that participate in chemical reactions. The performance of both methods is examined in detail for two reactions where the explicit solvent plays very different roles within the collective variables. When applied to raw molecular dynamics data in solution, both PCA and UMAP representations are dominated by bulk solvent motions. On the other hand, when applied to data preprocessed by our attenuated projection methods, both PCA and UMAP identify the appropriate collective variables (though varying sensitivity is observed due to the presence of explicit solvent that results from the projection method). Importantly, this approach allows identification of specific solvent molecules that are relevant to the CVs and their importance.