Sufficient dimension reduction in regressions through cumulative Hessian directions

With the advent of high throughput genome-wide assays it has become possible to simultaneously measure multiple types of genomic data. Several projects like TCGA, ICGC, NCI-60 has generated comprehensive, multi-dimensional maps of the key genomic changes like MiRNA, MRNA, proteomics etc. from cancer samples[2,4]. These genomic data can be used for classifying tumour types[5]. Integrative analysis of these data from multiple sources can potentially provide additional biological insights, but methods to do any such analysis are lacking. One of the widely used solutions to handle high dimension data is by removing redundant information in the integrated sample. Most of the expressed genes are overlapped and can be projected onto lower dimension, and then be used to classify different tumor types, without the loss of any/much information. Sufficient dimension reduction (SDR) [1], a supervised dimension reduction approach, can be ideal to achieve such a goal. In this paper, we propose a novel integrative SDR method that can reduce dimensions of multiple data types simultaneously while sharing common latent structures to improve prediction and interpretation. In particular, we extend the sliced inverse regression (SIR) technique, a major SDR method, to integrate multiple omits data for simultaneous dimension reduction. SIR is a supervised dimension reduction method that assumes that the outcome variable Y depends on the predictor variable X through d unknown linear combinations of the predictor[3]. The predictor variable is replaced by its projection into a lower dimension subspace of the predictor space without the loss of information. The aim is to find the intersection of all the subspaces δ called the central susbspace (CS) of the predictor space satisfying the property Y ╨ X| Pδ X. To integrate multiple types of data, we propose and implement a new integrative sufficient dimension reduction method extending SIR[3], called integrative SIR. The main idea is that we take into account all the multi-omics data information simultaneously while finding a basis matrix for each data type with some sharing latent structures. Finally, we get d dimension data which is much smaller than the original data dimension. The reduced dimension d was achieved by cross validation. To demonstrate the integrated analysis of multi-omics data, we applied and compared conventional SIR and integrative SIR to analyze MRNA, MiRNA and proteomics expression profile of a subset of cell lines from the NCI-60 panel. The data used is taken from [6]. The outcomes we have to classify are CNS, Leukemia and Melanoma tumor types. We pre-screened 400 variables from each data type with the criteria of high variance. To find classification error, we performed random forest classification after we applied to each method with leave-one-out cross-validation. As a result, we found out that integrative SIR leads to less classification error as compared to conventional SIR. To summarize, we proposed a new integrative SIR method, a supervised dimension reduction technique for integrative analysis of multi-omics data types. Unlike conventional SDR methods, the new approach can reduce the dimensions of multiple omics data simultaneously while sharing common latent structures across data types without losing any information in prediction. By efficiently capturing the common information, our numerical study shows that integrative SIR classifies tumor types more accurately as compared to conventional SDR methods.

In this article, we review, consolidate and extend a theory for sufficient dimension reduction in regression settings. This theory provides a powerful context for the construction, characterization and interpretation of low-dimensional displays of the data, and allows us to turn graphics into a consistent and theoretically motivated methodological body. In this spirit, we propose an iterative graphical procedure for estimating the meta-parameter which lies at the core of sufficient dimension reduction; namely, the central dimension-reduction subspace.

Sufficient dimension reduction (SDR) continues to be an active field of research. When estimating the central subspace (CS), inverse regression based SDR methods involve solving a generalized eigenvalue problem, which can be problematic under the large-p-small-n situation. In recent years, new techniques have emerged in numerical linear algebra, called randomized algorithms or random sketching, for high-dimensional and large scale problems. To overcome the large-p-small-n SDR problem, we combine the idea of statistical inference with random sketching to propose a new SDR method, called integrated random-partition SDR (iRP-SDR). Our method consists of the following three steps: (i)Randomly partition the covariates into subsets to construct an envelope subspace with low dimension. (ii)Obtain a sketch of the CS by applying a conventional SDR method within the constructed envelope subspace. (iii)Repeat the above two steps many times and integrate these multiple sketches to form the final estimate of the CS. After describing the details of these steps, the asymptotic properties of iRP-SDR are established. Unlike existing methods, iRP-SDR does not involve the determination of the structural dimension until the last stage, which makes it more adaptive to a high-dimensional setting. The advantageous performance of iRP-SDR is demonstrated via simulation studies and a practical example analyzing EEG data.

Robust sufficient dimension reduction via ball covariance

Sliced average third‐moment estimation (SATME) is a typical method for sufficient dimension reduction (SDR) based on high‐order conditional moment. It is useful, particularly in the scenarios of regression mixtures. However, as SATME uses the third‐order conditional moment of the predictors given the response, it may not as robust as some other SDR methods that use lower order moments, say, sliced inverse regression (SIR) and slice average variance estimation (SAVE). Based on the space displacement function, a local influence analysis framework of SATME is constructed including a statistic of influence assessment for the observations. Furthermore, a data‐trimming strategy is suggested based on the above influence assessment. The proposed methodologies solve a typical issue that also exists in some other SDR methods. A real‐data analysis and simulations are presented.

Sufficient dimension reduction (SDR) is a popular framework for supervised dimension reduction, aiming at reducing the dimensionality of input data while information on output data is maximally maintained. On the other hand, in many recent supervised classification learning tasks, it is conceivable that the balance of samples in each class varies between the training and testing phases. Such a phenomenon, referred to as class-prior change, causes existing SDR methods to perform undesirably particularly when the training data is highly imbalanced. In this paper, we extend the state-of-the-art SDR method called leastsquares gradients for dimension reduction (LSGDR) to be able to cope with such class-prior change under the semi-supervised learning setup where unlabeled test data are available in addition to labeled training data. Through experiments, we demonstrate the usefulness of our proposed method.

The idea of dimension reduction without loss of information can be quite helpful for guiding the construction of summary plots in regression without requiring a prespecified model. Focusing on the central subspace, we investigate such "sufficient" dimension reduction in regressions with a binary response. Three existing methods—sliced inverse regression, principal Hessian direction, and sliced average variance estimation—and one new method—difference of covariances—are studied for their ability to estimate the central subspace and produce sufficient summary plots. Combining these numerical methods with the graphical methods proposed earlier by Cook leads to a novel paradigm for the analysis of binary response regressions. Key Words: Dimension-reduction subspaceLinear combinations of chi-squared variablesRegression graphicsSliced inverse regressionPrincipal Hessian directionSliced average variance estimation.

SummarySliced inverse regression is one of the widely used dimension reduction methods. Chiaromonte and co-workers extended this method to regressions with qualitative predictors and developed a method, partial sliced inverse regression, under the assumption that the covariance matrices of the continuous predictors are constant across the levels of the qualitative predictor. We extend partial sliced inverse regression by removing the restrictive homogeneous covariance condition. This extension, which significantly expands the applicability of the previous methodology, is based on a new estimation method that makes use of a non-linear least squares objective function.

On testing common indices for two multi-index models: A link-free approach

On model-free conditional coordinate tests for regressions

In this work, we address the longstanding puzzle that Sliced Inverse Regression (SIR) often performs poorly for sufficient dimension reduction when the structural dimension d (the dimension of the central space) exceeds 4. We first show that in the multiple index model Y=f(PX)+ϵ where X is a p-standard normal vector, ϵ is an independent noise, and P is a projection operator from Rp to Rd, if the link function f follows the law of a Gaussian process. Then with high probability, the dth eigenvalue λd of Cov[E(X|Y)] satisfies λd≤Ce−θd for some positive constants C and θ. We then focus on the low signal regime where λd can be arbitrarily small and not larger than d−8.1, and prove that the minimax risk of estimating the central space is lower bounded by dp nλd. Combining these two results, we provide a convincing explanation for the poor performance of SIR when d is large, a phenomenon that has perplexed researchers for nearly three decades. The technical tools developed here may be of independent interest for studying other sufficient dimension reduction methods.

Because sliced inverse regression (SIR) using the conditional mean of the inverse regression fails to recover the central subspace when the inverse regression mean degenerates, sliced average variance estimation (SAVE) using the conditional variance was proposed in the sufficient dimension reduction literature. However, the efficacy of SAVE depends heavily upon the number of slices. In the present article, we introduce a class of weighted variance estimation (WVE), which, similar to SAVE and simple contour regression (SCR), uses the conditional variance of the inverse regression to recover the central subspace. The strong consistency and the asymptotic normality of the kernel estimation of WVE are established under mild regularity conditions. Finite sample studies are carried out for comparison with existing methods and an application to a real data is presented for illustration.

Sufficient dimension reduction (SDR) techniques have become essential tools in high‐dimensional data analysis. However, many SDR methods rely on iterative procedures, which impose substantial computational costs when applied to large‐scale datasets. To address these challenges, we develop optimal subsampling strategies tailored to two representative SDR approaches: The refined outer product of gradients based on the conditional density functions (rdOPG) and the minimum average (conditional) variance estimation based on the conditional density functions (dMAVE). Our methods are designed to minimize the trace of the asymptotic variance–covariance matrix under an inverse probability weighting framework, thereby achieving an effective trade‐off between computational efficiency and statistical accuracy. We derive explicit forms for the optimal subsampling probabilities and establish the consistency and asymptotic normality of the resulting estimators. Extensive simulation studies and a real‐data application demonstrate that our methods substantially reduce computational burden while achieving improved estimation accuracy compared to uniform subsampling approaches, highlighting their practical value for large‐scale SDR problems.

Sliced inverse regression (SIR, Li 1991) is a pioneering work and the most recognized method in sufficient dimension reduction. While promising progress has been made in theory and methods of high-dimensional SIR, two remaining challenges are still nagging high-dimensional multivariate applications. First, choosing the number of slices in SIR is a difficult problem, and it depends on the sample size, the distribution of variables, and other practical considerations. Second, the extension of SIR from univariate response to multivariate is not trivial. Targeting at the same dimension reduction subspace as SIR, we propose a new slicing-free method that provides a unified solution to sufficient dimension reduction with high-dimensional covariates and univariate or multivariate response. We achieve this by adopting the recently developed martingale difference divergence matrix (MDDM, Lee & Shao 2018) and penalized eigen-decomposition algorithms. To establish the consistency of our method with a high-dimensional predictor and a multivariate response, we develop a new concentration inequality for sample MDDM around its population counterpart using theories for U-statistics, which may be of independent interest. Simulations and real data analysis demonstrate the favorable finite sample performance of the proposed method.

A new sufficient dimension reduction method via rank divergence