Learning Linear and Nonlinear PCA with Linear Programming

The authors provide a didactic treatment of nonlinear (categorical) principal components analysis (PCA). This method is the nonlinear equivalent of standard PCA and reduces the observed variables to a number of uncorrelated principal components. The most important advantages of nonlinear over linear PCA are that it incorporates nominal and ordinal variables and that it can handle and discover nonlinear relationships between variables. Also, nonlinear PCA can deal with variables at their appropriate measurement level; for example, it can treat Likert-type scales ordinally instead of numerically. Every observed value of a variable can be referred to as a category. While performing PCA, nonlinear PCA converts every category to a numeric value, in accordance with the variable's analysis level, using optimal quantification. The authors discuss how optimal quantification is carried out, what analysis levels are, which decisions have to be made when applying nonlinear PCA, and how the results can be interpreted. The strengths and limitations of the method are discussed. An example applying nonlinear PCA to empirical data using the program CATPCA (J. J. Meulman, W. J. Heiser, & SPSS, 2004) is provided.

Complex-valued neural networks for nonlinear complex principal component analysis

Nonlinear principal component analysis (NLPCA) is known as a nonlinear generalization of the standard principal component analysis (PCA). Since nonlinear PCA is a non-unique concept, it is discussed, how nonlinear PCA can be defined as a nonlinear feature extraction technique most similar to linear PCA. The nonlinear reduction of a data set from its original dimension to the intrinsic dimension of the data is one aspect, but we also request that the nonlinear features spanning this intrinsic data space are hierarchically arranged similar to the linear features of PCA. Thus, such nonlinear PCA is a powerful pre-processing step. It can be used as nonlinear sphering (whitening) or it can be considered as a smoothing method which removes nonlinear correlations between variables. Nonlinear PCA can be performed by minimizing a hierarchical error function. This error function is applied to the autoencoder which is a multi-layer perceptron used in auto-associative mode to perform the identity mapping.

Besides the performance and range requirements, the breakthrough of electromobility depends crucially on the safety of battery systems. Cell faults that can lead to thermal runaway of the energy storage reduce customer acceptance. Thermal runaways are often preceded by an Internal Short Circuit (ISC). Thus, there is a necessity for a method that is able to detect the formation of ISCs, before a significant temperature rise is observed and the risk of thermal propagation arises. To achieve this, a data-driven method that enables early detection of soft ISCs is presented. The unsupervised machine-learning method is based on linear Principal Component Analysis (PCA) and nonlinear Kernel PCA (KPCA). Since the method only requires fault-free voltage measurements for training, it is directly applicable in conventional battery systems. The nonlinear KPCA is thoroughly compared with the linear PCA using experimental data. The data originates from a module consisting of twelve automotive cells. While the linear method has advantages in computational complexity, the nonlinear method detects ISCs earlier due to its high sensitivity and specificity.

The identification of between-group differences and changes in gait mechanics are useful for injury prevention. Previous studies suggest the differences in gait biomechanical variables may interact in a complex non-linear fashion rather than a simple linear fashion. A non-linear multivariate analysis technique is therefore necessary to unravel the inherent structure in the gait data. Kernel principal component analysis (KPCA) is an extension of principal component analysis (PCA), the most widely used method in this field, and can provide inside into non-linear relationships of the variables of interest. Despite the growing use of PCA in running gait research, no prior studies have evaluated the utility of KPCA in determining the between-group differences in running gait patterns. Therefore, the objective of this study was to compare the performance of KPCA and PCA in identifying differences between male and female runners and as well between young and older runners in running gait kinematics. Running kinematic data were analysed on a gender group (female, n = 50; male, n = 50) and an age group (young, n = 50; older, n = 50) for discrete variables and waveform data. The following key results were obtained: (1) the performance of the KPCA for the identification of gender and age differences in running gait kinematics increased as compared to using the linear PCA; (2) the Gaussian function often performed better than the polynomial function for these two experiments; (3) there was no consistent optimal value of either the width σ of the Gaussian kernel or the degree d of the polynomial kernel for different data types and experiments. These results suggest that non-linear features extracted by KPCA could be potentially useful in identifying and discriminating between-group differences and changes in running gait patterns, and could provide useful information about the intrinsic non-linear dynamics of running movement.

Morphological profiles (MPs) have been proposed in recent literature as aiding tools to achieve better results for classification of remotely sensed data. MPs are in general built using features containing most of the information content of the data, such as the components derived from principal component analysis (PCA). Recently, nonlinear PCA (NLPCA), performed by autoassociative neural network, has emerged as a good unsupervised technique to fit the information content of hyperspectral data into few components. The aim of this letter is to investigate the classification accuracies obtained using extended MPs built from the features of NPCA. A comparison of the two approaches has been validated on two different data sets having different spatial and spectral resolutions/coverages, over the same ground truth, and also using two different classification algorithms. The results show that NLPCA permits one to obtain better classification accuracies than using linear PCA.

In order to improve the operation and management level of thermal power units, and to achieve the goal of saving energy and reducing consumption, the nonlinear principal component analysis (PCA) was proposed to assess the operating condition of fossil-fired power plants comprehensively, which can resolve the problem on the evaluation of multiple indicators. This method improves the traditional PCA by eliminating the relativity. Moreover, this paper applies the nonlinear PCA to evaluate the operating state of power plants, which can reduce the dimensions of the indicators and abstract the principal components. It can provide the high reference value for the power generation group on how to implement saving energy, optimal operation, indicators evaluation and competition in thermal power units. The practical example shows that the proposed method is effective.

A kernel based nonlinear subspace projection (KNSP) method is proposed for reduction of hyperspectral image dimensionality. This method involves three steps: subspace partition of full data space, feature extraction based on kernel principal component analysis (KPCA) in subspace and feature selection based on class separability criterion. The main merit of the proposed method is that it is more suitable for feature extraction than linear principal component analysis (PCA) and segmented principal component: transformation (SPCT), in particular, when hyperspectral data have nonlinear characteristics. In order to testify the effectiveness of the KNSP method for reduction of hyperspectral image dimensionality, hyperspectral image classification is performed on AVIRIS data. Experimental results show that when the hyperspectral dimensionality is reduced to a few features, the average classification accuracy of the new method is higher than those of PCA and SPCT methods.

Principal component analysis (PCA) has been proven to be an efficient method in dimensionality reduction, feature extraction and pattern recognition. Kernel principal component analysis (KPCA) can be considered as a natural nonlinear generalization of PCA, which performs linear PCA in a high dimensional space implicitly by using kernel trick. However, both conventional PCA and KPCA suffer from the deficiency of being sensitive to outliers. Existing robust KPCA has to eigen-decompose the gram matrix directly in each step and is much more computationally infeasible due to the large size of the matrix when the number of training samples is large. By extending existing robust PCA algorithm using kernel methods, we present a novel robust adaptive algorithm for calculating the kernel principal components. The proposed method not only preserves the characteristic of capturing underlying nonlinear structure of KPCA but also is robust against outliers by restraining the effect of outlying samples. Compared with existing robust KPCA methods, our method is performed without having to store the kernel matrix, which can reduce significantly the storage burden. In addition, our method shows the potential of expansibility to the incremental learning version. Experimental results on synthetic data indicate that our improved algorithm is effective and promising.

Artificial neural networks (ANNs) are used to determine the state-of-health (SOH) of the Automated Radioxenon Analyzer/Sampler (ARSA). ARSA is a gas collection and analysis system used for non-proliferation monitoring in detecting radioxenon released during nuclear tests. SOH diagnostics are important for automated, unmanned sensing systems so that remote detection and identification of problems can be made without onsite staff. Both recurrent and feed-forward ANNs are presented. The recurrent ANN is trained to predict sensor values based on current valve states, which control air flow, so that with only valve states the normal SOH sensor values can be predicted. Deviation between modeled value and actual is an indication of a potential problem. The feed-forward ANN acts as a nonlinear version of principal components analysis (PCA) and is trained to replicate the normal SOH sensor values. Because of ARSA's complexity, this nonlinear PCA is better able to capture the relationships among the sensors than standard linear PCA and is applicable to both sensor validation and recognizing off-normal operating conditions. Both models provide valuable information to detect impending malfunctions before they occur to avoid unscheduled shutdown. Finally, the ability of ANN methods to predict the system state is presented.

Eigenvoice-based methods have been shown to be effective for fast speaker adaptation when the amount of adaptation data is small, say, less than 10 seconds. In traditional eigenvoice (EV) speaker adaptation, linear principal component analysis (PCA) is used to derive the eigenvoices. Recently, we proposed that eigenvoices found by nonlinear kernel PCA could be more effective, and the eigenvoices thus derived were called kernel eigenvoices (KEV). One of our novelties is the use of composite kernel that makes it possible to compute state observation likelihoods via kernel functions. In this paper, we investigate two different composite kernels: direct sum kernel and tensor product kernel for KEV adaptation. In an evaluation on the TIDIGITS task, it is found that KEV speaker adaptation using both forms of composite kernel are equally effective, and they outperform a speaker-independent model and the adapted models from EV, MAP, or MLLR adaptation using 2.1s and 4.1s of speech. For example, with 2.1s of adaptation data, KEV adaptation outperforms the speaker-independent model by 27.5%, whereas EV, MAP, or MLLR adaptation are not effective at all. 1.

This paper introduces a new unsupervised method for dimensionality reduction via regression (DRR). The algorithm belongs to the family of invertible transforms that generalize Principal Component Analysis (PCA) by using curvilinear instead of linear features. DRR identifies the nonlinear features through multivariate regression to ensure the reduction in redundancy between he PCA coefficients, the reduction of the variance of the scores, and the reduction in the reconstruction error. More importantly, unlike other nonlinear dimensionality reduction methods, the invertibility, volume-preservation, and straightforward out-of-sample extension, makes DRR interpretable and easy to apply. The properties of DRR enable learning a more broader class of data manifolds than the recently proposed Non-linear Principal Components Analysis (NLPCA) and Principal Polynomial Analysis (PPA). We illustrate the performance of the representation in reducing the dimensionality of remote sensing data. In particular, we tackle two common problems: processing very high dimensional spectral information such as in hyperspectral image sounding data, and dealing with spatial-spectral image patches of multispectral images. Both settings pose collinearity and ill-determination problems. Evaluation of the expressive power of the features is assessed in terms of truncation error, estimating atmospheric variables, and surface land cover classification error. Results show that DRR outperforms linear PCA and recently proposed invertible extensions based on neural networks (NLPCA) and univariate regressions (PPA).

This research uses eigenvalue characteristics of nonlinear principal component analysis (NLPCA) and principal component analysis (PCA) to investigate total electron content (TEC) anomalies associated with Taiwan’s Chi-Chi earthquake of 21 September 1999 (LT) (M w=7.6). The transforms are used for ionospheric TEC from 01 August to 20 September 1999 (local time) using data from 13 GPS receivers. The data were collected at 22°N–26°N Lat. and 120°E–122°E Long.. Applying the NLPCA to the multi-channel total electron content records of GPS receivers, the earthquake-associated TEC anomalies were represented by large principal eigenvalues of NLPCA (>0.5 in a normalized set) on 14 August and 17, 18, and 20 September, with allowance given for the Dst index, which was quiet for the study period. Comparisons were then made with other researchers who also found TEC anomalies on September 17, 18, and 19 associated with the Chi-Chi earthquake, which cannot be detected by PCA. Consideration is also given for reported ground level geomagnetic field activity that occurred between mid-August and late October, leading up to and including the Chi-Chi and Chia-Yi earthquakes, which are associated with the same series of faults. It is possible that Aug. 14 is representative of an earthquake-associated TEC anomaly. This is an interesting result given how much earlier than the earthquake it occurred.

Gas sensor array is a useful device to enhance the selectivity of gas detectors and to identify the components of gas mixture. The key step for processing signal from a gas sensor array is to extract the signal feature and make pre-classification for tested gases. Conventional Principal Component Analysis (PCA) is widely used for this purpose. However, conventional PCA is a linear and variance-covariance matrix based technique and it is therefore not strictly applicable for processing the gas sensor array signals that exhibit significant non-linear behavior. Thus, in this paper, non-linear PCA (NPCA) algorithm is introduced to process the gas sensor array signals to adapt to the non-linear characteristics. The signals we processed are the responses of a micro-hotplate (MHP) based integrated gas sensor array to a CO and NO2 binary gas mixture. The gas sensor array, consisted of four SnO2 thin-film sensing elements, was fabricated with integrated circuit (IC) technology and micromachining on silicon substrate. The recognition results of NPCA and conventional PCA are compared in this paper.

The design of the step-size is crucial to the convergence rate of the Nonlinear Principle Component Analysis(NPCA) algorithm.However,the commonly used fixed step-size algorithm can hardly satisfy the convergence speed and estimation precision requirements simultaneously.To address this issue,the gradient-based adaptive step-size NPCA algorithm and optimal step-size NPCA algorithm were proposed to speed up the convergence rate and improve tracking ability.In particular,the optimal step-size NPCA algorithm linearly approximated the contrast function and figured out the optimal step-size currently.The optimal step-size NPCA algorithm utilized an adaptive step-size whose value was adjusted in sympathy with the value of the contrast function and free from any manual parameters.The simulation results show that the proposed adaptive step-size NPCA algorithms have faster convergence rate or better tracking ability in comparison with the fixed step-size NPCA algorithm when the estimation precisions are same.The convergence performance of the optimal step-size NPCA algorithm is superior to that of the gradient-based adaptive NPCA algorithm.