A Primal-Dual Algorithm for Distributed Sparse Principal Component Analysis

Most of the existing procedures for sparse principal component analysis (PCA) use a penalty function to obtain a sparse matrix of weights by which a data matrix is post-multiplied to produce PC scores. In this paper, we propose a new sparse PCA procedure which differs from the existing ones in two ways. First, the new procedure does not sparsify the weight matrix. Instead, the so-called loadings matrix is sparsified by which the score matrix is post-multiplied to approximate the data matrix. Second, the cardinality of the loading matrix i.e., the total number of nonzero loadings, is pre-specified to be an integer without using penalty functions. The procedure is called unpenalized sparse loading PCA (USLPCA). A desirable property of USLPCA is that the indices for the percentages of explained variances can be defined in the same form as in the standard PCA. We develop an alternate least squares algorithm for USLPCA which uses the fact that the PCA loss function can be decomposed as a sum of a term irrelevant to the loadings, and another one being easily minimized under cardinality constraints. A procedure is also presented for selecting the best cardinality using information criteria. The procedures are assessed in a simulation study and illustrated with real data examples.

Previous versions of sparse principal component analysis (PCA) have presumed that the eigen-basis (a p × k matrix) is approximately sparse. We propose a method that presumes the p × k matrix becomes approximately sparse after a k × k rotation. The simplest version of the algorithm initializes with the leading k principal components. Then, the principal components are rotated with an k × k orthogonal rotation to make them approximately sparse. Finally, soft-thresholding is applied to the rotated principal components. This approach differs from prior approaches because it uses an orthogonal rotation to approximate a sparse basis. One consequence is that a sparse component need not to be a leading eigenvector, but rather a mixture of them. In this way, we propose a new (rotated) basis for sparse PCA. In addition, our approach avoids “deflation” and multiple tuning parameters required for that. Our sparse PCA framework is versatile; for example, it extends naturally to a two-way analysis of a data matrix for simultaneous dimensionality reduction of rows and columns. We provide evidence showing that for the same level of sparsity, the proposed sparse PCA method is more stable and can explain more variance compared to alternative methods. Through three applications—sparse coding of images, analysis of transcriptome sequencing data, and large-scale clustering of social networks, we demonstrate the modern usefulness of sparse PCA in exploring multivariate data. An R package, epca, and the supplementary materials for this article are available online.

Dual Bounds of Sparse Principal Component Analysis Sparse principal component analysis (PCA) is a widely used dimensionality reduction tool in machine learning and statistics. Compared with PCA, sparse PCA enhances the interpretability by incorporating a sparsity constraint. However, unlike PCA, conventional heuristics for sparse PCA cannot guarantee the qualities of obtained primal feasible solutions via associated dual bounds in a tractable fashion without underlying statistical assumptions. In “Using L1-Relaxation and Integer Programming to Obtain Dual Bounds for Sparse PCA,” Santanu S. Dey, Rahul Mazumder, and Guanyi Wang present a convex integer programming (IP) framework of sparse PCA to derive dual bounds. They show the worst-case results on the quality of the dual bounds provided by the convex IP. Moreover, the authors empirically illustrate that the proposed convex IP framework outperforms existing sparse PCA methods of finding dual bounds.

Principal Component Analysis (PCA) is a convenient tool used in aggregating the indicators of sustainable development and providing indices where different weights are assigned to the various indicators. There are, however, problems in interpreting of indices, especially if time series data are used. This study explores the feasibility of applying recent developments in PCA of time series using Philippine data. We present the comparative advantages of SPCA (Sparse Principal Component Analysis) relative to averaging of an adequacy/inadequacy index and PCA in index construction from various indicators of sustainable development in the Philippines in terms of usefulness and validity of indices being developed. SPCA can attain sparse and non-overlapping loadings without losing a large amount of explained variance compared to PCA. Because of the non-overlapping contribution of variables in SPCA components, indices can have clear and mutually exclusive meanings, facilitating interpretation. Even with a more complicated algorithm, reduced dimensions and simpler interpretation of indices justify the advantages of SPCA over PCA in index construction. The indices are interpreted in terms of the milestone of sustainability in the Philippines. The resulting indices provide an adequate summary of the sustainable indicators and evidence of the importance of leadership and political will in sustainable development.

Sparse Principal Component Analysis (SPCA) is a powerful technique for dimensionality reduction and feature extraction in high-dimensional data, with applications spanning various fields such as computer vision, pattern recognition, and data mining. However, the computational intensity of SPCA presents a significant challenge, necessitating the development of efficient and robust algorithms. In this paper, we shed light on the SPCA problem and uncover intriguing structures that enable us to design an efficient algorithm, which we have named SPCA_ACC. Firstly, we identify a separable structure in this problem, which prompts us to draw on the Variable Projection (VP) strategy and generalize it to separable nonlinear problem in Stiefel manifold. This strategy projects out part of the parameters to obtain a reduced problems, allowing the SPCA_ACC algorithm to optimize in a lower-dimensional parameter space. Secondly, we resolve the coupling between different parameters of the SPCA problem in the optimization process on a fixed coordinate-sparsity manifold, which opens the way to the use of second-order Riemannian accelerated VP strategy. Moreover, we systematically analyze the advantages of using VP to solve the SPCA problem from a theoretical perspective, and confirm the local quadratic convergence of our algorithm. Numerical experiments on datasets of different sizes and types demonstrate that our method achieves rapid convergence and significantly reduces computational costs.

Abnormal glucose variability (GV) is a risk factor for diabetes complications, and tens of indices for its quantification from continuous glucose monitoring (CGM) time series have been proposed. However, the information carried by these indices is redundant, and a parsimonious description of GV can be obtained through sparse principal component analysis (SPCA). We have recently shown that a set of 10 metrics selected by SPCA is able to describe more than 60% of the variance of 25 GV indicators in type 1 diabetes (T1D). Here, we want to extend the application of SPCA to type 2 diabetes (T2D). A data set of CGM time series collected in 13 T2D subjects was considered. The 25 GV indices considered for T1D were evaluated. SPCA was used to select a subset of indices able to describe the majority of the original variance. A subset of 10 indicators was selected and allowed to describe 83% of the variance of the original pool of 25 indices. Four metrics sufficient to describe 67% of the original variance turned out to be shared by the parsimonious sets of indices in T1D and T2D. Starting from a pool of 25 indices assessed from CGM time series in T2D subjects, reduced subsets of metrics virtually providing the same information content can be determined by SPCA. The fact that these indices also appear in the parsimonious description of GV in T1D may indicate that they could be particularly informative of GV in diabetes, regardless of the specific type of disease.

Hyperspectral imaging (HSI) is increasingly being used in a wide range of practical applications. High correlations between and within classes, the curse of dimensionality, and overfitting have all piqued the interest of researchers. Because of the excellent feature extraction and good overall performance, deep learning-based algorithms with dimensionality reduction methods have been recommended as a viable alternative for hyperspectral image analysis. Principal component analysis (PCA) and Sparse PCA (SPCA) are commonly used in dimensionality reduction. Because each principal component is a linear mixture of the original variables, principal component analysis (PCA) is difficult to interpret, and SPCA cannot choose the most relevant features. To overcome the issue, a mutual information-based greedy feature selection approach called minimum redundancy maximum relevancy (mRMR) is combined with SPCA (SPCA-mRMR). Many deep learning combination techniques rely on the 2D and 3D Convolutional Neural Network (CNN) methodologies. A fused 3D-2D CNN model is less difficult than a 3D-CNN alone, yet it can also operate effectively in noisy settings with few training examples. Therefore, SPCA-mRMR and fused 3D-2D CNN are used to create a hybrid model called HMC-NET for HSI classification. Several optimization strategies are used to address overfitting and improve classification accuracy. To assess the efficacy of this proposed technique, performance results were compared to highly regarded models with fewer variables.

Sparse Principal Component Analysis (SPCA) is a method that can get the sparse loadings of the principal components (PCs), and it may formulate PCA as a regression-type optimization problem by using the elastic net. But the selected features are different with each PC and generally independent. A new method named SPCA has been proposed for removing these detect, which replaces the elastic net with L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> -norm penalty. The results of the method on gene expression data are still unknown. Therefore, we will take a test to prove this point in this paper. Firstly, this method is applied to the simulated data for obtaining an optimal parameter. Secondly, the L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> SPCA method is applied to the gene expression data, that is the head and neck squamous carcinoma data (HNSC). Thirdly, the characteristic genes are selected according the PCs. The results consist of very lower P-value and very higher hit count, which shows the method of L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> SPCA can obtain higher recognition accuracy and higher relevancy to the genes. Finally, the experimental results demonstrate that the L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> SPCA works well and has good performances in the gene expression data.

We consider the sparse principal component analysis for high‐dimensional stationary processes. The standard principal component analysis performs poorly when the dimension of the process is large. We establish oracle inequalities for penalized principal component estimators for the large class of processes including heavy‐tailed time series. The rate of convergence of the estimators is established. We also elucidate the theoretical rate for choosing the tuning parameter in penalized estimators. The performance of the sparse principal component analysis is demonstrated by numerical simulations. The utility of the sparse principal component analysis for time series data is exemplified by the application to average temperature data.

In this paper, we study the application of sparse principal component analysis (PCA) to clustering and feature selection problems. Sparse PCA seeks sparse factors, or linear combinations of the data variables, explaining a maximum amount of variance in the data while having only a limited number of nonzero coefficients. PCA is often used as a simple clustering technique and sparse factors allow us here to interpret the clusters in terms of a reduced set of variables. We begin with a brief introduction and motivation on sparse PCA and detail our implementation of the algorithm in d’Aspremont et al. (SIAM Rev. 49(3):434–448, 2007). We then apply these results to some classic clustering and feature selection problems arising in biology.

ABSTRACTThe implementation of conventional sparse principal component analysis (SPCA) on high-dimensional data sets has become a time consuming work. In this paper, a series of subspace projections are constructed efficiently by using Householder QR factorization. With the aid of these subspace projections, a fast deflation method, called SPCA-SP, is developed for SPCA. This method keeps a good tradeoff between various criteria, including sparsity, orthogonality, explained variance, balance of sparsity, and computational cost. Comparative experiments on the benchmark data sets confirm the effectiveness of the proposed method.

Multicollinearity refers to the presence of collinearity between multiple variables and renders the results of statistical inference erroneous (Type II error). This is particularly important in environmental health research where multicollinearity can hinder inference. To address this, correlated variables are often excluded from the analysis, limiting the discovery of new associations. An alternative approach to address this problem is the use of principal component analysis. This method, combines and projects a group of correlated variables onto a new orthogonal space. While this resolves the multicollinearity problem, it poses another challenge in relation to interpretability of results. Standard hypothesis testing methods can be used to evaluate the association of projected predictors, called principal components, with the outcomes of interest, however, there is no established way to trace the significance of principal components back to individual variables. To address this problem, we investigated the use of sparse principal component analysis which enforces a parsimonious projection. We hypothesise that this parsimony could facilitate the interpretability of findings. To this end, we investigated the association of 20 environmental predictors with all-cause mortality adjusting for demographic, socioeconomic, physiological, and behavioural factors. The study was conducted in a cohort of 379,690 individuals in the UK. During an average follow-up of 8.05 years (3,055,166 total person-years), 14,996 deaths were observed. We used Cox regression models to estimate the hazard ratio (HR) and 95% confidence intervals (CI). The Cox models were fitted to the standardised environmental predictors (a) without any transformation (b) transformed with PCA, and (c) transformed with SPCA. The comparison of findings underlined the potential of SPCA for conducting inference in scenarios where multicollinearity can increase the risk of Type II error. Our analysis unravelled a significant association between average noise pollution and increased risk of all-cause mortality. Specifically, those in the upper deciles of noise exposure have between 5 and 10% increased risk of all-cause mortality compared to the lowest decile.

As the goals of ensuring process safety and energy efficiency become ever more challenging, engineers increasingly rely on data collected from such processes for informed decision making. During recent decades, extracting and interpreting valuable process information from large historical data sets have been an active area of research. Among the methods used, principal component analysis (PCA) is a well-established technique that allows for dimensionality reduction for large data sets by finding new uncorrelated variables, namely principal components (PCs). However, it is difficult to interpret the derived PCs, as each PC is a linear combination of all of the original variables and the loadings are typically nonzero. Sparse principal component analysis (SPCA) is a relatively recent technique proposed for producing PCs with sparse loadings via the variance–sparsity trade-off. We propose a forward SPCA approach that helps uncover the underlying process knowledge regarding variable relations. This approach s...

The extension of internet facility profoundly eases the dissemination of all digital data such as audio, images and videos to the general public. This paper proposes a blind scheme for digital video watermarking. In this paper, a discrete wavelet domain watermarking is adopted for hiding a large amount of data with high security, good invisibility and no loss to the secret message. At first, dual tree complex wavelet transform (DTCWT) is applied on each frame decomposing it into a number of sub-bands. Then, holo entropy of each sub-band is calculated and the maximum entropy blocks are selected. The selected blocks are transformed using sparse principal component analysis (SPCA). The maximum coefficient of the SPCA blocks of each sub-band is quantised using quantisation index modulation (QIM). The proposed video watermarking scheme is analysed through various constraints such as the normalised correlation (NC) and the peak signal to noise ratio (PSNR).

To solve the multicollinearity issue and unequal contribution of vascular parameters for the quantification of angiogenesis, we developed a quantification evaluation method of vascular parameters for angiogenesis based on in vivo micro-CT imaging of hindlimb ischemic model mice. Taking vascular volume as the ground truth parameter, nine vascular parameters were first assembled into sparse principal components (PCs) to reduce the multicolinearity issue. Aggregated boosted trees (ABTs) were then employed to analyze the importance of vascular parameters for the quantification of angiogenesis via the loadings of sparse PCs. The results demonstrated that vascular volume was mainly characterized by vascular area, vascular junction, connectivity density, segment number and vascular length, which indicated they were the key vascular parameters for the quantification of angiogenesis. The proposed quantitative evaluation method was compared with both the ABTs directly using the nine vascular parameters and Pearson correlation, which were consistent. In contrast to the ABTs directly using the vascular parameters, the proposed method can select all the key vascular parameters simultaneously, because all the key vascular parameters were assembled into the sparse PCs with the highest relative importance.