Correlation tests for high-dimensional data under the strongly spiked eigenvalue model

Due to the intricate relationship between different dimensions of high-dimensional data, subspace analysis is often conducted to decompose dimensions and give prominence to certain subsets of dimensions, i.e. subspaces. Exploring and comparing subspaces are important to reveal the underlying features of subspaces, as well as to portray the characteristics of individual dimensions. To date, most of the existing high-dimensional data exploration and analysis approaches rely on dimensionality reduction algorithms (e.g. principal component analysis and multi-dimensional scaling) to project high-dimensional data, or their subspaces, to two-dimensional space and employ scatterplots for visualization. However, the dimensionality reduction algorithms are sometimes difficult to fine-tune and scatterplots are not effective for comparative visualization, making subspace comparison hard to perform. In this article, we aggregate high-dimensional data or their subspaces by computing pair-wise distances between all data items and showing the distances with matrix visualizations to present the original high-dimensional data or subspaces. Our approach enables effective visual comparisons among subspaces, which allows users to further investigate the characteristics of individual dimensions by studying their behaviors in similar subspaces. Through subspace comparisons, we identify dominant, similar, and conforming dimensions in different subspace contexts of synthetic and real-world high-dimensional data sets. Additionally, we present a prototype that integrates parallel coordinates plot and matrix visualization for high-dimensional data exploration and incremental dimensionality analysis, which also allows users to further validate the dimension characterization results derived from the subspace comparisons.

Contemporary single-cell technologies produce data with a vast number of variables at a rapid pace, making large volumes of high-dimensional data available. The exploratory analysis of such high dimensional data can be aided by intuitive low dimensional visualizations. In this work, we investigate how both discrete and continuous structures in single cell data can be captured using the recently proposed dimensionality reduction method SONG, and compare the results with commonly used methods UMAP and PHATE. Using simulated and real-world datasets, we observed that SONG preserves a variety of patterns including discrete clusters, continuums, and branching structures. More importantly, SONG produced more/equally insightful visualizations compared to UMAP and PHATE in all considered datasets. We also quantitatively validate the high-dimensional pairwise distance preservation ability of these visualization methods in the low dimensional space for the generated visualizations.

Applications of medical image analysis are often faced with the challenge of modelling high-dimensional data with relatively few samples. In many settings, normal or healthy samples are prevalent while pathological samples are rarer, highly diverse, and/or difficult to model. In such cases, a robust model of the normal population in the high-dimensional space can be useful for characterizing pathologies. In this context, there is utility in hybrid models, such as probabilistic PCA, which learns a low-dimensional model, commensurates with the available data, and combines it with a generic, isotropic noise model for the remaining dimensions. However, the isotropic noise model ignores the inherent correlations that are evident in so many high-dimensional data sets associated with images and shapes in medicine. This paper describes a method for estimating a Gaussian model for collections of images or shapes that exhibit underlying correlations, e.g., in the form of smoothness. The proposed method incorporates a Gaussian-process noise model within a generative formulation. For optimization, we derive a novel expectation maximization (EM) algorithm. We demonstrate the efficacy of the method on synthetic examples and on anatomical shape data.

IntroutionThe population multiple correlation coefficient (PMCC) measures the correlation between a given one-dimensional random variable X and multidimensional random vector X .This measure is the maximum correlation between X and any linear combination of components X .Testing the null hypothesis of zero PMCC is the most favourable approach to investigate the existence or non-existence of PMCC between X and X .The classical procedures for testing this hypothesis in high-dimensional data settings are invalid since the sample covariance matrix inverse or sample precision matrix is undefined.To cope with this problem, a simple test is constructed for testing zero PMCC in high-dimensional normal data.A small simulation study was carried out to evaluate the performance of the proposed test in both high-dimensional and low-dimensional normal data sets.Finally, the proposed test is applied to mice tumour volumes data. ateria an ethosThe proposed test includes the following two steps: In the first step, using a plug-in estimator of the sample precision matrix, a simple test statistic, called plug-in test statistic, was derived which uses the EQUALs method (Wang and Jiang, ) to estimate the precision matrix.In the second step, a permutation test is constructed using the proposed plug-in statistic to test the null hypothesis of zero PMCC.Theoretical investigations demonstrate that the proposed testing procedure is a level test. . . . . .

Numerous variable selection techniques have been developed for complete high-dimensional data but very few of them for censored data. The techniques for complete data must be modified if censoring is present. In this paper, we consider the variable selection technique for accelerated failure time (AFT) models by extending the ranking-based variable selection (RBVS) algorithm and its iterative procedure as proposed in the work of Baranowski et al. through the Stute’s weighted least square technique. Simulation studies are conducted to demonstrate the performance of the proposed methods. We further illustrate the performance of this method with a mantle cell lymphoma microarray example. When there is no correlation among the covariates, the proposed method outperforms the iterative sure independence screening and stability selection methods in terms of overall performance for high-dimensional data. Real data analysis also suggests that the proposed method can be chosen for high-dimensional censored data analysis in parallel to other methods in the literature.

Network propagation algorithms have proved useful for the analysis of high-dimensional genomic data. One limitation is that the current formulation only allows network propagation on positively weighted graphs. In this paper, we explore two signed network propagation algorithms and general optimization frameworks for detecting differential gene expressions and DNA copy number variations (CNV). The proposed algorithms consider both positive and negative relations in graphs to model gene up/down-regulation or amplification/deletion CNV events. The first algorithm (Signed-NP) integrates gene co-expressions and differential expressions for consistent and robust gene selection from microarray datasets by propagation on gene correlation graphs. The second algorithm (Signed-NPBi) identifies gene or CNV markers by propagation on sample-feature bipartite graphs to capture bi-clusters between samples and genomic features. Large scale experiments on several microarray gene expression datasets and CNV datasets validate that Signed-NP and Signed-NPBi perform better classification of gene expression and CNV data than standard network propagation. The experiments also demonstrate that Signed-NP is capable of selecting genes that are more biologically interpretable and consistent across multiple datasets, and Signed-NPBi can detect hidden CNV patterns in bi-clusters by smoothing on correlations between adjacent probes.

This chapter covers the state-of-the-art multivariate statistical methods designed for high dimensional multiset omics data analysis. Recent biotechnological developments have enabled large-scale measurement of various biomolecular data, such as genotypic and phenotypic data, dispersed over various omics domains. An emergent research direction is to analyze these data sources using an integrated approach to better model and understand the underlying biology of complex disease conditions. However, comprehensive analysis techniques that can handle both the size and complexity, and at the same time can account for the hierarchical structure of such data, are lacking. An overview of some of the developments in multivariate techniques for high-dimensional omics data analysis, highlighting two well-known multivariate methods, canonical correlation analysis (CCA) and redundancy analysis (RDA), is provided in this chapter. Penalized versions of CCA are widespread in the omics data analysis field, and there is recent work on multiset penalized RDA that is applicable to multiset omics data. How these methods meet the statistical challenges that come with high-dimensional multiset omics data analysis and help to further our understanding of the human condition in terms of health and disease are presented. Additionally, the current challenges to be resolved in the field of omics data analysis are discussed.

Uncovering driver genes is crucial for understanding heterogeneity in cancer. L1-type regularization approaches have been widely used for uncovering cancer driver genes based on genome-scale data. Although the existing methods have been widely applied in the field of bioinformatics, they possess several drawbacks: subset size limitations, erroneous estimation results, multicollinearity, and heavy time consumption. We introduce a novel statistical strategy, called a Recursive Random Lasso (RRLasso), for high dimensional genomic data analysis and investigation of driver genes. For time-effective analysis, we consider a recursive bootstrap procedure in line with the random lasso. Furthermore, we introduce a parametric statistical test for driver gene selection based on bootstrap regression modeling results. The proposed RRLasso is not only rapid but performs well for high dimensional genomic data analysis. Monte Carlo simulations and analysis of the “Sanger Genomics of Drug Sensitivity in Cancer dataset from the Cancer Genome Project” show that the proposed RRLasso is an effective tool for high dimensional genomic data analysis. The proposed methods provide reliable and biologically relevant results for cancer driver gene selection.

Instrument endogeneity and identification-robust tests: Some analytical results

Popular clustering algorithms based on usual distance functions (e.g., the Euclidean distance) often suffer in high dimension, low sample size (HDLSS) situations, where concentration of pairwise distances and violation of neighborhood structure have adverse effects on their performance. In this article, we use a new data-driven dissimilarity measure, called MADD, which takes care of these problems. MADD uses the distance concentration phenomenon to its advantage, and as a result, clustering algorithms based on MADD usually perform well for high dimensional data. We establish it using theoretical as well as numerical studies. We also address the problem of estimating the number of clusters. This is a challenging problem in cluster analysis, and several algorithms are available for it. We show that many of these existing algorithms have superior performance in high dimensions when they are constructed using MADD. We also construct a new estimator based on a penalized version of the Dunn index and prove its consistency in the HDLSS asymptotic regime. Several simulated and real data sets are analyzed to demonstrate the usefulness of MADD for cluster analysis of high dimensional data.

<ns3:p>Background Visualization and analysis of high-dimensional data are essential in biomedical research. There is a need for secure, scalable, and reproducible tools to facilitate data exploration and interpretation. Results We introduce DataMap, a browser-based application designed for the visualization of high-dimensional data using heatmaps and dimensionality reduction plots. DataMap operates directly in the web browser, ensuring data privacy without the requirement of installations or server setups. The application features an intuitive user interface for data transformation, annotation, and generation of reproducible R code. Conclusions Freely accessible as a GitHub page (https://gexijin.github.io/datamap), DataMap is a secure, user-friendly, and reproducible solution for visualizing high-dimensional data.</ns3:p>

This article is concerned with testing for noninvertible time series models. For a stationary but noninvertible autoregressive moving average model, I construct a derived process that is nonstationary but invertible with a nonstationary factor identical to the noninvertible factor of the original time series. I then propose a test procedure for testing noninvertibility using various unit-root test statistics available in the literature. The limiting distributions of the test statistics employed depend on the mean as well as the initial innovations of the original series. I also compare the performance of the proposed test procedure with that of other noninvertible tests available in the literature. For illustration, I apply the proposed test procedure to detect trend stationarity of two U.S. economic time series.

With the explosive growth of social media data, the need for effective high-dimensional data analysis techniques has become increasingly pressing. Current research in this field faces challenges such as data sparsity, noise, and scalability issues. To address these challenges, this paper proposes a novel approach utilizing Principal Component Analysis (PCA) for high-dimensional data analysis in social media. By applying PCA to reduce the dimensionality of the data while preserving essential information, our method aims to enhance the efficiency and accuracy of social media data analysis. Through comprehensive experimentation and evaluation, this paper demonstrates the effectiveness and potential of the proposed approach in improving the analysis of high-dimensional social media data, contributing valuable insights to the field of social media analytics.

BackgroundFor gene expression or gene association studies with a large number of hypotheses the number of measurements per marker in a conventional single-stage design is often low due to limited resources. Two-stage designs have been proposed where in a first stage promising hypotheses are identified and further investigated in the second stage with larger sample sizes. For two types of two-stage designs proposed in the literature we derive multiple testing procedures controlling the False Discovery Rate (FDR) demonstrating FDR control by simulations: designs where a fixed number of top-ranked hypotheses are selected and designs where the selection in the interim analysis is based on an FDR threshold. In contrast to earlier approaches which use only the second-stage data in the hypothesis tests (pilot approach), the proposed testing procedures are based on the pooled data from both stages (integrated approach).ResultsFor both selection rules the multiple testing procedures control the FDR in the considered simulation scenarios. This holds for the case of independent observations across hypotheses as well as for certain correlation structures. Additionally, we show that in scenarios with small effect sizes the testing procedures based on the pooled data from both stages can give a considerable improvement in power compared to tests based on the second-stage data only.ConclusionThe proposed hypothesis tests provide a tool for FDR control for the considered two-stage designs. Comparing the integrated approaches for both selection rules with the corresponding pilot approaches showed an advantage of the integrated approach in many simulation scenarios.

Partial least squares (PLS) regression is a very useful tool for the analysis of high dimensional data (HDD). Choosing the ideal number of PLS components is a vital step in developing the best model. The accuracy of the model will be affected if there are too many or too few PLS components being selected. Numerous classical methods, such as the leave-one-out cross-validation (LOOCV) and K-fold cross-validation (K-FoldCV) are developed to determine the optimal number of PLS components. Nonetheless, they are easily affected by high leverage points (HLPs). Thus, robust cross validation techniques, denoted as RMD- MRCD-PCA-LOOCV and RMD-MRCD-PCA-K-FoldCV are proposed to remedy this problem. The results of the simulation study and real data set indicate that the proposed methods successfully select the appropriate number of PLS components.