Supervised data transformation and dimensionality reduction with a 3-layer multi-layer perceptron for classification problems

Target detection (TD) is a major task in hyperspectral image (HSI) processing which, due to the high spectral resolution, requires dealing with the curse of dimensionality. The integrated feature extraction and selection is a well-known solution for dimensionality reduction of HSIs. In this study, a new method is presented to improve the performance of TD algorithms based on principal component analysis (PCA) feature space. In this method, using the implantation of the target spectrum (TS) in the HSI and following the simulated targets in the PCA feature space, the best principal components (PCs) are selected. Then, using the mixing and unmixing coefficients of the PCs, a new TS and a new image in the PCA feature space are created. Afterwards, using the new spectrum of the target, the TD algorithm is run on the new HSI. The performance of the proposed method is compared to nine counterpart algorithms on Hymap and Hyperion HSI. All the comparisons are performed using adaptive coherence estimator (ACE) TD algorithm. Experimental results illustrate that the proposed method, compared to its counterparts, yields superior performance based on the false alarm rate (FAR) measure. It gives an average FAR value of about 16, which is approximately 9% better than that of its best counterparts.

This paper carries on research surrounding the influences produced by dimensionality reduction on machine learning classification effect. Firstly, paper constructs the analysis architecture of data dimension reduction classification, combines the two different unsupervised dimension reduction methods, locally linear embedding (LLE) and principal component analysis (PCA) with the five machine learning classification methods: Gradient Boosting Decision Tree (GBDT), Random Forest, Support Vector Machine (SVM), K-Nearest Neighbor (KNN) and Logistic Regression. And then uses the handwritten digital identification dataset to analyze the classification performance of these five classification methods on different dimension datasets by different dimensionality reduction methods. The analysis shows that using the appropriate dimensionality reduction method for dimensionality reduction classification can effectively improve the classification accuracy; the dimensionality reduction classification effect of non-linear dimensionality reduction method is generally better than the linear dimensionality reduction method; different machine learning classification algorithms have significant differences in the sensitivity of dimensions.

Principal component analysis is an unsupervised linear dimension reduction method in machine learning. The basic idea of principal component analysis is to project the data of feature space into low dimensional space, and ensure the maximum variance of projection point, which turns the problem into the optimal parameter problem that can make the maximum variance. In the field of machine learning, the gradient ascent method is usually used to solve the optimal parameters, but it is easy to fall into the local maximum trap. In this paper, a principal component analysis method with high- order truncated gradient ascent method is proposed. High- order truncated gradient ascent method is used to replace the gradient ascent method so as to better retain the gradient information and have better global convergence.

Unsupervised learning is a branch of machine learning that tries to find hidden structures within unlabeled data and derive insights from it. Clustering, data dimensionality-reduction techniques, noise reduction, segmentation, anomaly detection, fraud detection, and other rich methods rely on unsupervised learning to drive analytics. Today, with so much data around us, it is impossible to label all data for supervised learning. This makes unsupervised learning all the more important. Restricted Boltzmann machines and auto-encoders are unsupervised methods that are based on artificial neural networks. They have a wide range of uses in data compression and dimensionality reduction, noise reduction from data, anomaly detection, generative modeling, collaborative filtering, and initialization of deep neural networks, among other things. We will go through these topics in detail and then touch upon a couple of unsupervised pre-processing techniques for images, namely PCA (principal component analysis) whitening and ZCA (Mahalanobis) whitening. Also, since restricted Boltzmann machines use sampling techniques during training, I have briefly touched upon Bayesian inference and Markov Chain Monte Carlo sampling for reader's benefit.

The paper is focused on the idea to demonstrate the advantages of deep learning approaches over ordinary shallow neural network on their comparative applications to image classifying from such popular benchmark databases as FERET and MNIST. An autoassociative neural network is used as a standalone program realized the nonlinear principal component analysis for prior extracting the most informative features of input data for neural networks to be compared further as classifiers. A special study of the optimal choice of activation function and the normalization transformation of input data allows to improve efficiency of the autoassociative program. One more study devoted to denoising properties of this program demonstrates its high efficiency even on noisy data. Three types of neural networks are compared: feed-forward neural net with one hidden layer, deep network with several hidden layers and deep belief network with several pretraining layers realized restricted Boltzmann machine. The number of hidden layer and the number of hidden neurons in them were chosen by cross-validation procedure to keep balance between number of layers and hidden neurons and classification efficiency. Results of our comparative study demonstrate the undoubted advantage of deep networks, as well as denoising power of autoencoders. In our work we use both multiprocessor graphic card and cloud services to speed up our calculations. The paper is oriented to specialists in concrete fields of scientific or experimental applications, who have already some knowledge about artificial neural networks, probability theory and numerical methods.

Aiming at the features that modem industrial processes always have some characteristics of complexity and nonlinearity and the process data usually contain both Gaussion and non-Gaussion information at the same time, a new process performance monitoring and fault diagnosis method based on subsystem division and kernel entropy component analysis (Sub-KECA) is proposed in this paper. KECA as a new method for data transformation and dimensionality reduction, which chooses the best principal component vector according to the maximal Renyi entropy rather than judging by the top eigenvalue and eigenvector of the kernel matrix simply. Besides, it can be optimized and anti-disturb due to the application of subsystem division. The proposed method is applied to process monitoring of the Tennessee Eastman(TE) process. The positive simulation results indicate that this method is more feasible and efficient when comparing with KPCA method and original KECA method.

Considering that kernel entropy component analysis (KECA) is a promising new method of nonlinear data transformation and dimensionality reduction, a KECA based method is proposed for nonlinear chemical process monitoring. In this method, an angle-based statistic is designed because KECA reveals structure related to the Renyi entropy of input space data set, and the transformed data sets are produced with a distinct angle-based structure. Based on the angle difference between normal status and current sample data, the current status can be monitored effectively. And, the confidence limit of the angle-based statistics is determined by kernel density estimation based on sample data of the normal status. The effectiveness of the proposed method is demonstrated by case studies on both a numerical process and a simulated continuous stirred tank reactor (CSTR) process. The KECA based method can be an effective method for nonlinear chemical process monitoring.

The rapid evolution of information and communication technologies leads to a big networks security problem. For this reason, the Intrusion Detection System (IDS) has been developed in order to detect and prevent computer network attacks. However, the majority of IDSs operate on huge network traffic data with many useless and redundant features. Consequently, the IDS generates a lot of false alarms and the intrusion detection process becomes difficult and imprecise. To improve the performance of an IDS, many data dimensionality reduction methods, such as Principal Component Analysis (PCA), have been proposed. However, the classical PCA approach, that is based on the covariance matrix of the data, is very sensitive to outliers. In order to overcome this problem, we propose to introduce a new variant of PCA namely L1-norm PCA. This new method is based on the L1-norm maximization, which is more robust to outliers, instead of the Euclidean norm in the classical PCA. Extensive experiments on the well-known KDDcup99 dataset are exploited for testing the effectiveness of the proposed approach. Obtained results confirm the superiority of L1-norm PCA over the traditional PCA in terms of network attacks detection and false alarms reduction.

One of the main challenges that the industry faces when dealing with massive data for failure diagnosis is high dimensionality of such data. This can be tackled by dimensionality reduction method such as principal components analysis, which usually results in an improved fault diagnosis. Other available techniques include auto-encoders and its variants denoising auto-encoders and sparse auto-encoders. Most recently, variational auto-encoders are one of the most promising techniques for unsupervised learning with successful applications in image processing and speech recognition. Differently from other auto-encoder methods, variational auto-encoders use variational inference to generate a latent representation of the data and impose a distribution over the latent variables and the data itself. In this article, we propose a fully unsupervised deep variational auto-encoder-based approach for dimensionality reduction in fault diagnosis and explore the variational auto-encoder capabilities for such a task. This is achieved by comparing the latent representations provided by variational auto-encoders to the ones from principal components analysis as well as when no reduction is performed in ball bearings’ fault classification using vibration signals. To tackle massive sensor data, we propose different architectures for the variational auto-encoder’s encoder and decoder that are based on deep neural networks and deep convolutional neural networks. Experiments are also carried out by varying the data preprocessing methods for generating spectrograms and hand-engineering features as well as the use of raw vibration data, the architecture of the neural networks fault classifiers operating on the latent representations from variational auto-encoder and principal components analysis, the degree of data dimension reduction, and the size of the available labeled data used for training the fault classifiers. The results show that variational auto-encoders are a competent and promising tool for dimensionality reduction for use in fault diagnosis and worth further exploring their capabilities beyond vibration signals of ball bearing elements.

The performance of gas-sensor array systems is greatly influenced by the pattern recognition scheme applied on the instrument's measurement data. The traditional method of choice is principal component analysis (PCA), aiming for reduction in dimensionality and visualization of multivariate measurement data. PCA, as a second-order statistical tool, performs well in many cases, but lacks the ability to give meaningful representations for non-Gaussian data, which often is a property of gas-sensor array measurement data. If, instead, higher order statistical methods are considered for data analysis, more useful information can be extracted from the data. This paper introduces the higher order statistical method called independent component analysis (ICA) as a novel tool for analysis of gas-sensor array measurement data. A comparison between the performances of PCA and ICA is illustrated both in theory and for two sets of practical measurement data. The described experiments show that ICA is capable of handling sensor drift combined with improved discrimination, dimensionality reduction, and more adequate data representation when compared to PCA.

Data dimension reduction (DDR) is all about mapping data from high dimensions to low dimensions, various techniques of DDR are being used for image dimension reduction like Random Projections, Principal Component Analysis (PCA), the Variance approach, LSA-Transform, the Combined and Direct approaches, and the New Random Approach. Auto-encoders (AE) are used to learn end-to-end mapping. In this paper, we demonstrate that pre-processing not only speeds up the algorithms but also improves accuracy in both supervised and unsupervised learning. In pre-processing of DDR, first PCA based DDR is used for supervised learning, then we explore AE based DDR for unsupervised learning. In PCA based DDR, we first compare supervised learning algorithms accuracy and time before and after applying PCA. Similarly, in AE based DDR, we compare unsupervised learning algorithm accuracy and time before and after AE representation learning. Supervised learning algorithms including support-vector machines (SVM), Decision Tree with GINI index, Decision Tree with entropy and Stochastic Gradient Descent classifier (SGDC) and unsupervised learning algorithm including K-means clustering, are used for classification purpose. We used two datasets MNIST and FashionMNIST Our experiment shows that there is massive improvement in accuracy and time reduction after pre-processing in both supervised and unsupervised learning.

Recently, many dimensionality reduction algorithms, including local methods and global methods, have been presented. The representative local linear methods are locally linear embedding (LLE) and linear preserving projections (LPP), which seek to find an embedding space that preserves local information to explore the intrinsic characteristics of high dimensional data. However, both of them still fail to nicely deal with the sparsely sampled or noise contaminated datasets, where the local neighborhood structure is critically distorted. On the contrary, principal component analysis (PCA), the most frequently used global method, preserves the total variance by maximizing the trace of feature variance matrix. But PCA cannot preserve local information due to pursuing maximal variance. In order to integrate the locality and globality together and avoid the drawback in LLE and PCA, in this paper, inspired by the dimensionality reduction methods of LLE and PCA, we propose a new dimensionality reduction method for face recognition, namely, unsupervised linear difference projection (ULDP). This approach can be regarded as the integration of a local approach (LLE) and a global approach (PCA), so that it has better performance and robustness in applications. Experimental results on the ORL, YALE and AR face databases show the effectiveness of the proposed method on face recognition.

Utilization of existing clinical data for improving patient outcomes poses a number of challenging and complex problems involving lack of data integration, the absence of standardization across inhomogeneous data sources and computationally-demanding and time-consuming exploration of very large datasets. In this paper, we will present a robust semantic data integration, standardization and dimensionality reduction method to tackle and solve these problems. Our approach enables the integration of clinical data from diverse sources by resolving canonical inconsistencies and semantic heterogeneity as required by the National Library of Medicine's Unified Medical Language System (UMLS) to produce standardized medical data. Through a combined application of rule-based semantic networks and machine learning, our approach enables a large reduction in dimensionality of the data and thus allows for fast and efficient application of data mining techniques to large clinical datasets. An example application of the techniques developed in our study is presented for the prediction of bariatric surgery outcomes.

Principal Component Analysis (PCA) is a widely used dimensional reduction method that aims to find a low dimension sub space of highly correlated data for its major information to be used in further analysis. Machine learning methods based on PCA are popular in high dimensional data analysis, such as video and image processing. In video processing, the Robust PCA (RPCA), which is a modified method of the traditional PCA, has good properties in separating moving objects from the background, but it may have difficulties in separating those when light intensity of the background varies significantly in time. To overcome the difficulties, a modified PCA method, called Regression PCA (RegPCA), is proposed. The method is developed by combining the traditional PCA and regression approaches together, and it can be easily combined with RPCA for video processing. We focus the presentation of RegPCA with the combination of RPCA on video processing and find that it is more reliable than RPCA only. We use RegPCA to separate moving object from the background in a color video and get a better result than that given by RPCA. In the implementation, we first derive the explanatory variables by the background information. we then process a number of frames of the video and use those as a set of response variables. We remove the impact of the background by regressing the response against the explanatory variables by a regression model. The regression model provides a set of residuals, which can be further analyzed by RPCA. We compare the results of RegRPCA against those of RPCA only. It is evident that the moving objects can be completely removed from the background using our method but not in RPCA. Note that our result is based on a combination of RegPCA with RPCA. Our proposed method provides a new implementation of RPCA under the framework of regression approaches, which can be used to account for the impact of risk factors. This problem cannot be addressed by the application of RPCA only.

Telecom Companies logs customer’s actions which generate a huge amount of data that can bring important findings related to customer’s behavior and needs. The main characteristics of such data are the large number of features and the high sparsity that impose challenges to the analytics steps. This paper aims to explore dimensionality reduction on a real telecom dataset and evaluate customers’ clustering in reduced and latent space, compared to original space in order to achieve better quality clustering results. The original dataset contains 220 features that belonging to 100,000 customers. However, dimensionality reduction is an important data preprocessing step in the data mining process specially with the presence of curse of dimensionality. In particular, the aim of data reduction techniques is to filter out irrelevant features and noisy data samples. To reduce the high dimensional data, we projected it down to a subspace using well known Principal Component Analysis (PCA) decomposition and a novel approach based on Autoencoder Neural Network, performing in this way dimensionality reduction of original data. Then K-Means Clustering is applied on both-original and reduced data set. Different internal measures were performed to evaluate clustering for different numbers of dimensions and then we evaluated how the reduction method impacts the clustering task.