Eigenvectors from Eigenvalues Sparse Principal Component Analysis (EESPCA)

Previous versions of sparse principal component analysis (PCA) have presumed that the eigen-basis (a p × k matrix) is approximately sparse. We propose a method that presumes the p × k matrix becomes approximately sparse after a k × k rotation. The simplest version of the algorithm initializes with the leading k principal components. Then, the principal components are rotated with an k × k orthogonal rotation to make them approximately sparse. Finally, soft-thresholding is applied to the rotated principal components. This approach differs from prior approaches because it uses an orthogonal rotation to approximate a sparse basis. One consequence is that a sparse component need not to be a leading eigenvector, but rather a mixture of them. In this way, we propose a new (rotated) basis for sparse PCA. In addition, our approach avoids “deflation” and multiple tuning parameters required for that. Our sparse PCA framework is versatile; for example, it extends naturally to a two-way analysis of a data matrix for simultaneous dimensionality reduction of rows and columns. We provide evidence showing that for the same level of sparsity, the proposed sparse PCA method is more stable and can explain more variance compared to alternative methods. Through three applications—sparse coding of images, analysis of transcriptome sequencing data, and large-scale clustering of social networks, we demonstrate the modern usefulness of sparse PCA in exploring multivariate data. An R package, epca, and the supplementary materials for this article are available online.

Most of the existing procedures for sparse principal component analysis (PCA) use a penalty function to obtain a sparse matrix of weights by which a data matrix is post-multiplied to produce PC scores. In this paper, we propose a new sparse PCA procedure which differs from the existing ones in two ways. First, the new procedure does not sparsify the weight matrix. Instead, the so-called loadings matrix is sparsified by which the score matrix is post-multiplied to approximate the data matrix. Second, the cardinality of the loading matrix i.e., the total number of nonzero loadings, is pre-specified to be an integer without using penalty functions. The procedure is called unpenalized sparse loading PCA (USLPCA). A desirable property of USLPCA is that the indices for the percentages of explained variances can be defined in the same form as in the standard PCA. We develop an alternate least squares algorithm for USLPCA which uses the fact that the PCA loss function can be decomposed as a sum of a term irrelevant to the loadings, and another one being easily minimized under cardinality constraints. A procedure is also presented for selecting the best cardinality using information criteria. The procedures are assessed in a simulation study and illustrated with real data examples.

Sparse Principal Component Analysis (SPCA) is a method that can get the sparse loadings of the principal components (PCs), and it may formulate PCA as a regression-type optimization problem by using the elastic net. But the selected features are different with each PC and generally independent. A new method named SPCA has been proposed for removing these detect, which replaces the elastic net with L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> -norm penalty. The results of the method on gene expression data are still unknown. Therefore, we will take a test to prove this point in this paper. Firstly, this method is applied to the simulated data for obtaining an optimal parameter. Secondly, the L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> SPCA method is applied to the gene expression data, that is the head and neck squamous carcinoma data (HNSC). Thirdly, the characteristic genes are selected according the PCs. The results consist of very lower P-value and very higher hit count, which shows the method of L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> SPCA can obtain higher recognition accuracy and higher relevancy to the genes. Finally, the experimental results demonstrate that the L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> SPCA works well and has good performances in the gene expression data.

Continuous glucose monitoring (CGM) time-series are often analyzed, retrospectively, to investigate glucose variability (GV), a risk factor for the development of complications in type 1 diabetes (T1D). In the literature, several tens of different indices for GV quantification have been proposed, but many of them carry very similar information. The aim of this article is to select a relatively small subset of GV indices from a wider pool of metrics, to obtain a parsimonious but still comprehensive description of GV in T1D datasets. A pool of 25 GV indices was evaluated on two CGM time-series datasets of 17 and 16 T1D subjects, respectively, collected during the European Union Seventh Framework Programme project "Diadvisor" (2008-2012) in two different clinical research centers using the Dexcom(®) (San Diego, CA) SEVEN(®) Plus. After the indices were centered and scaled, the Sparse Principal Component Analysis (SPCA) technique was used to determine a reduced set of metrics that allows preserving a high percentage of the variance of the whole original set. In order to assess whether or not the selected subset of GV indices is dataset-dependent, the analysis was applied to both datasets, as well as to the one obtained by merging them. SPCA revealed that a subset of up to 10 different GV indices can be sufficient to preserve more than the 60% of the variance originally explained by all the 25 variables. It is remarkable that four of these GV indices (i.e., Index of Glycemic Control, percentage of Glycemic Risk Assessment Diabetes Equation score due to euglycemia, percentage Coefficient of Variation, and Low Blood Glucose Index) were selected for all the considered T1D datasets. The SPCA methodology appears a suitable candidate to identify, among the large number of literature GV indices, subsets that allow obtaining a parsimonious, but still comprehensive, description of GV.

Dual Bounds of Sparse Principal Component Analysis Sparse principal component analysis (PCA) is a widely used dimensionality reduction tool in machine learning and statistics. Compared with PCA, sparse PCA enhances the interpretability by incorporating a sparsity constraint. However, unlike PCA, conventional heuristics for sparse PCA cannot guarantee the qualities of obtained primal feasible solutions via associated dual bounds in a tractable fashion without underlying statistical assumptions. In “Using L1-Relaxation and Integer Programming to Obtain Dual Bounds for Sparse PCA,” Santanu S. Dey, Rahul Mazumder, and Guanyi Wang present a convex integer programming (IP) framework of sparse PCA to derive dual bounds. They show the worst-case results on the quality of the dual bounds provided by the convex IP. Moreover, the authors empirically illustrate that the proposed convex IP framework outperforms existing sparse PCA methods of finding dual bounds.

Abnormal glucose variability (GV) is considered to be a risk factor for the development of diabetes complications. For its quantification from continuous glucose monitoring (CGM) data, tens of different indices have been proposed in the literature, but the information carried by them is highly redundant. In the present work, the Sparse Principal Component Analysis (SPCA) technique is used to select, from a wide pool of GV metrics, a smaller subset of indices that preserves the majority of the total original variance, providing a parsimonious but still comprehensive description of GV. In detail, SPCA is applied to a set of 25 literature GV indices evaluated on CGM time-series collected in 17 type 1 (T1D) and 13 type 2 (T2D) diabetic subjects. Results show that the 10 GV indices selected by SPCA preserve more than the 75% of the variance of the original set of 25 indices, both in T1D and T2D. Moreover, 6 indices of the parsimonious set are shared by T1D and T2D.

Abnormal glucose variability (GV) is a risk factor for diabetes complications, and tens of indices for its quantification from continuous glucose monitoring (CGM) time series have been proposed. However, the information carried by these indices is redundant, and a parsimonious description of GV can be obtained through sparse principal component analysis (SPCA). We have recently shown that a set of 10 metrics selected by SPCA is able to describe more than 60% of the variance of 25 GV indicators in type 1 diabetes (T1D). Here, we want to extend the application of SPCA to type 2 diabetes (T2D). A data set of CGM time series collected in 13 T2D subjects was considered. The 25 GV indices considered for T1D were evaluated. SPCA was used to select a subset of indices able to describe the majority of the original variance. A subset of 10 indicators was selected and allowed to describe 83% of the variance of the original pool of 25 indices. Four metrics sufficient to describe 67% of the original variance turned out to be shared by the parsimonious sets of indices in T1D and T2D. Starting from a pool of 25 indices assessed from CGM time series in T2D subjects, reduced subsets of metrics virtually providing the same information content can be determined by SPCA. The fact that these indices also appear in the parsimonious description of GV in T1D may indicate that they could be particularly informative of GV in diabetes, regardless of the specific type of disease.

Multicollinearity refers to the presence of collinearity between multiple variables and renders the results of statistical inference erroneous (Type II error). This is particularly important in environmental health research where multicollinearity can hinder inference. To address this, correlated variables are often excluded from the analysis, limiting the discovery of new associations. An alternative approach to address this problem is the use of principal component analysis. This method, combines and projects a group of correlated variables onto a new orthogonal space. While this resolves the multicollinearity problem, it poses another challenge in relation to interpretability of results. Standard hypothesis testing methods can be used to evaluate the association of projected predictors, called principal components, with the outcomes of interest, however, there is no established way to trace the significance of principal components back to individual variables. To address this problem, we investigated the use of sparse principal component analysis which enforces a parsimonious projection. We hypothesise that this parsimony could facilitate the interpretability of findings. To this end, we investigated the association of 20 environmental predictors with all-cause mortality adjusting for demographic, socioeconomic, physiological, and behavioural factors. The study was conducted in a cohort of 379,690 individuals in the UK. During an average follow-up of 8.05 years (3,055,166 total person-years), 14,996 deaths were observed. We used Cox regression models to estimate the hazard ratio (HR) and 95% confidence intervals (CI). The Cox models were fitted to the standardised environmental predictors (a) without any transformation (b) transformed with PCA, and (c) transformed with SPCA. The comparison of findings underlined the potential of SPCA for conducting inference in scenarios where multicollinearity can increase the risk of Type II error. Our analysis unravelled a significant association between average noise pollution and increased risk of all-cause mortality. Specifically, those in the upper deciles of noise exposure have between 5 and 10% increased risk of all-cause mortality compared to the lowest decile.

Sparse principal component analysis (SPCA) imposes extra constraints or penalty terms to the standard PCA to achieve sparsity. In this paper, we introduce an efficient algorithm for finding an effective sparse feature principal component (PC) of multiple physiological signals. The algorithm consists of two stages. In the first stage, it identifies an active index set with a desired cardinality corresponding to the nonzero entries of the PC. In the second one, it uses the power iteration method to find the best direction with respect to the active index set. Experiments on randomly generated data and multiple physiological signals datasets show that our algorithm is very fast, especially on large and sparse data sets, while the numerical quality of the solution is comparable to the state-of-art algorithm.

The extension of internet facility profoundly eases the dissemination of all digital data such as audio, images and videos to the general public. This paper proposes a blind scheme for digital video watermarking. In this paper, a discrete wavelet domain watermarking is adopted for hiding a large amount of data with high security, good invisibility and no loss to the secret message. At first, dual tree complex wavelet transform (DTCWT) is applied on each frame decomposing it into a number of sub-bands. Then, holo entropy of each sub-band is calculated and the maximum entropy blocks are selected. The selected blocks are transformed using sparse principal component analysis (SPCA). The maximum coefficient of the SPCA blocks of each sub-band is quantised using quantisation index modulation (QIM). The proposed video watermarking scheme is analysed through various constraints such as the normalised correlation (NC) and the peak signal to noise ratio (PSNR).

ABSTRACTThe implementation of conventional sparse principal component analysis (SPCA) on high-dimensional data sets has become a time consuming work. In this paper, a series of subspace projections are constructed efficiently by using Householder QR factorization. With the aid of these subspace projections, a fast deflation method, called SPCA-SP, is developed for SPCA. This method keeps a good tradeoff between various criteria, including sparsity, orthogonality, explained variance, balance of sparsity, and computational cost. Comparative experiments on the benchmark data sets confirm the effectiveness of the proposed method.

Hyperspectral imaging (HSI) is increasingly being used in a wide range of practical applications. High correlations between and within classes, the curse of dimensionality, and overfitting have all piqued the interest of researchers. Because of the excellent feature extraction and good overall performance, deep learning-based algorithms with dimensionality reduction methods have been recommended as a viable alternative for hyperspectral image analysis. Principal component analysis (PCA) and Sparse PCA (SPCA) are commonly used in dimensionality reduction. Because each principal component is a linear mixture of the original variables, principal component analysis (PCA) is difficult to interpret, and SPCA cannot choose the most relevant features. To overcome the issue, a mutual information-based greedy feature selection approach called minimum redundancy maximum relevancy (mRMR) is combined with SPCA (SPCA-mRMR). Many deep learning combination techniques rely on the 2D and 3D Convolutional Neural Network (CNN) methodologies. A fused 3D-2D CNN model is less difficult than a 3D-CNN alone, yet it can also operate effectively in noisy settings with few training examples. Therefore, SPCA-mRMR and fused 3D-2D CNN are used to create a hybrid model called HMC-NET for HSI classification. Several optimization strategies are used to address overfitting and improve classification accuracy. To assess the efficacy of this proposed technique, performance results were compared to highly regarded models with fewer variables.

As the goals of ensuring process safety and energy efficiency become ever more challenging, engineers increasingly rely on data collected from such processes for informed decision making. During recent decades, extracting and interpreting valuable process information from large historical data sets have been an active area of research. Among the methods used, principal component analysis (PCA) is a well-established technique that allows for dimensionality reduction for large data sets by finding new uncorrelated variables, namely principal components (PCs). However, it is difficult to interpret the derived PCs, as each PC is a linear combination of all of the original variables and the loadings are typically nonzero. Sparse principal component analysis (SPCA) is a relatively recent technique proposed for producing PCs with sparse loadings via the variance–sparsity trade-off. We propose a forward SPCA approach that helps uncover the underlying process knowledge regarding variable relations. This approach s...

Regularized variants of Principal Components Analysis, especially Sparse PCA\nand Functional PCA, are among the most useful tools for the analysis of complex\nhigh-dimensional data. Many examples of massive data, have both sparse and\nfunctional (smooth) aspects and may benefit from a regularization scheme that\ncan capture both forms of structure. For example, in neuro-imaging data, the\nbrain's response to a stimulus may be restricted to a discrete region of\nactivation (spatial sparsity), while exhibiting a smooth response within that\nregion. We propose a unified approach to regularized PCA which can induce both\nsparsity and smoothness in both the row and column principal components. Our\nframework generalizes much of the previous literature, with sparse, functional,\ntwo-way sparse, and two-way functional PCA all being special cases of our\napproach. Our method permits flexible combinations of sparsity and smoothness\nthat lead to improvements in feature selection and signal recovery, as well as\nmore interpretable PCA factors. We demonstrate the efficacy of our method on\nsimulated data and a neuroimaging example on EEG data.\n

The reliability of Industrial Control Systems (ICS) is crucial given their substantial importance in society and business. These systems are susceptible to physical and cyber-attacks, which can seriously affect human life and the economy. Given the growing prevalence of Internet of Things (IoT) technologies and the possibility of cyber warfare, it is essential to prioritize safeguarding Industrial Control Systems (ICS) from cyberattacks. Previous studies revealed an imbalance in the ICS datasets. As a result, models didn't do well in minority classes but did well in majority classes, which made the intrusion detection system (IDS) less useful. The key objective is to provide insights into the normal functioning of the system and the disruptions produced by cyberattacks. In this study, we introduced an interdisciplinary framework that aims to enhance network intrusion detection systems (NIDSs). In this framework, we introduced an IDS via feature selection and feature reduction technique(s) with the attention-driven lightweight deep neural networks: Deep Recurrent Neural Networks (RNN), Deep Long Short-Term Memory (LSTM), and Deep Bi-directional Long Short-Term Memory (Bi-LSTM). Several feature selection techniques exclude features that fail to match the specified criteria. We employed Sparse Principal Component Analysis (SPCA) to extract higher-order features. We conducted experiments on the three datasets: the Secure Water Treatment System (SWaT), the Water Distribution (WADI), and the Gas Heating Loop (GHL).Among the models used in the framework, the attention-driven Deep LSTM model produced better results than the others, revealing lower training and testing times for the three datasets. In terms of precision, recall, F1-score, computational speed, and ability to work with larger datasets and different ICSs, the proposed framework is better than previous methods in detecting cyberattacks. This highlights how useful it is in the real world.