Sparse Convoluted Rank Regression in High Dimensions

ABSTRACTThe presence of outliers would inevitably lead to distorted analysis and inappropriate prediction, especially for multiple outliers in high-dimensional regression, where the high dimensionality of the data might amplify the chance of an observation or multiple observations being outlying. Noting that the detection of outliers is not only necessary but also important in high-dimensional regression analysis, we, in this paper, propose a feasible outlier detection approach in sparse high-dimensional linear regression model. Firstly, we search a clean subset by use of the sure independence screening method and the least trimmed square regression estimates. Then, we define a high-dimensional outlier detection measure and propose a multiple outliers detection approach through multiple testing procedures. In addition, to enhance efficiency, we refine the outlier detection rule after obtaining a relatively reliable non-outlier subset based on the initial detection approach. By comparison studies based on Monte Carlo simulation, it is shown that the proposed method performs well for detecting multiple outliers in sparse high-dimensional linear regression model. We further illustrate the application of the proposed method by empirical analysis of a real-life protein and gene expression data.

While functional magnetic resonance imaging (fMRI) is important for healthcare/neuroscience applications, it is challenging to classify or interpret due to its multi-dimensional structure, high dimensionality, and small number of samples available. Recent sparse multilinear regression methods based on tensor are emerging as promising solutions for fMRI, yet existing works rely on unfolding/folding operations and a tensor rank relaxation with limited tightness. The newly proposed tensor singular value decomposition (t-SVD) sheds light on new directions. In this work, we study t-SVD for sparse multilinear regression and propose a Sparse tubal-regularized multilinear regression (Sturm) method for fMRI. Specifically, the Sturm model performs multilinear regression with two regularization terms: a tubal tensor nuclear norm based on t-SVD and a standard L1 norm. We further derive the algorithm under the alternating direction method of multipliers framework. We perform experiments on four classification problems, including both resting-state fMRI for disease diagnosis and task-based fMRI for neural decoding. The results show the superior performance of Sturm in classifying fMRI using just a small number of voxels.

Accurate classification of Alzheimer’s Disease (AD) and its prodromal stage, i.e., mild cognitive impairment (MCI) are critical for the effective treatment of AD. However, compared with AD classification tasks, predicting the conversion of MCI to AD is relatively difficult. as there are only minor differences among MCI groups. What’s more, in brain imaging analysis, the high dimensionality and relatively small number of subjects brings challenges to computer-aided diagnosis of AD and MCI. Many previous researches focused on the identification of imaging biomarkers for AD diagnosis. In this paper, we introduce sparse logistic regression for the early diagnosis of AD. Sparse logistic regression (SLR) uses L1/2 regularization to impose a sparsity constraint on logistic regression. The L1/2 regularization is considered a representative of Lq regularization, where fewer but informative key brain regions are applied for the classification of AD/MCI. We evaluated the SLR on 197 subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. Experimental results showed that the SLR improves the classification performance of AD/MCI compared other classical methods.

Sparse reduced-rank regression is a widespread tool to reveal the association between multiple responses and predictors, and it has been widely applied to many data-driven applications. Although much of the literature has studied related theoretical properties and numerical algorithms, due to high nonconvexity, the computation burden for large-scale data sets remains a great challenge in practice. Also, the gap between the statistical consistency and the algorithmic convergence needs more research. To address these two issues, we formulate a sparse reduced-rank regression as a set of parallel cosparse unit-rank estimation problems and propose a new algorithm to estimate these subproblems in parallel. Under mild conditions, the iteration complexity of the proposed algorithm is polynomial with high-dimensional responses and predictors. We show a statistical consistency for the numerical solution, thereby bridging the gap between statistical consistency and numerical computation from nonconvex optimization. Moreover, the main calculation of the algorithm is restricted to a small active set, so it exhibits fast computation even in high dimensions. Extensive numerical studies and an application in genetics demonstrate the effectiveness and scalability of our approach. History: Accepted by Antonio Frangioni, Area Editor for Design & Analysis of Algorithms–Continuous. Funding: This work was supported by the National Key R&D Program of China [Grant 2024YFA1012200], the National Natural Science Foundation of China [Grants 12171449 and 72401266], the Fundamental Research Funds for the Central Universities [Grant WK2040000079], the USTC Research Funds of the Double First-Class Initiative [Grant YD2040002019], and the China Postdoctoral Science Foundation [Grant 2023M733402]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2024.0691 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2024.0691 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

In this paper, we propose a novel and robust procedure for model identification in semiparametric additive models based on rank regression and spline approximation. Under some mild conditions, we establish the theoretical properties of the identified nonparametric functions and the linear parameters. Furthermore, we demonstrate that the proposed rank estimate has a great efficiency gain across a wide spectrum of non-normal error distributions and almost not lose any efficiency for the normal error compared with that of least square estimate. Even in the worst case scenarios, the asymptotic relative efficiency of the proposed rank estimate versus least squares estimate, which is show to have an expression closely related to that of the signed-rank Wilcoxon test in comparison with the t-test, has a lower bound equal to 0.864. Finally, an efficient algorithm is presented for computation and the selections of tuning parameters are discussed. Some simulation studies and a real data analysis are conducted to illustrate the finite sample performance of the proposed method.

This work focuses on the construction of functional representations in high-dimensional spaces.Attention is focused on the modeling of ignition phenomena using detailed kinetics, and on the ignition delay time as the primary quantity of interest (QoI). An iso-octane air mixture is first considered, using a detailed chemical mechanism with 3,811 elementary reactions. Uncertainty in all reaction rates is directly accounted for using associated uncertainty factors, assuming independent log-uniform priors. A Latin hypercube sample (LHS) of the ignition delay times was first generated, and the resulting database was then exploited to assess the possibility of constructing polynomial chaos (PC) representations in terms of the canonical random variables parametrizing the uncertain rates. We explored two avenues, namely sparse regression (SR) using LASSO, and a coordinate transform (CT) approach. Preconditioned variants of both approaches were also considered, namely using the logarithm of the ignition delay time as QoI. A global sensitivity analysis is performed using the representations constructed by SR and CT. Next, the tangent linear approximation is developed to estimate the sensitivity of the ignition delay time with respect to individual rate parameters in a detailed chemical mechanism. Attention is focused on a gas mixture reacting under adiabatic, constant-volume conditions. The approach is based on integrating the linearized system of equations governing the evolution of the partial derivatives of the state vector with respect to individual random variables, and a linearized approximation is developed to relate the ignition delay sensitivity to the scaled partial derivatives of temperature. In particular, the computations indicate that for detailed reaction mechanisms the TLA leads to robust local sensitivity predictions at a computational cost that is order-of-magnitude smaller than that incurred by finite-difference approaches based on one-at-a-time rate parameters perturbations. In the last part, we explore the potential of utilizing TLA-based sensitivities to identify active subspace and to construct suitable representations. Performance is assessed based contrasting experiences with CT-based machinery developed earlier.

Recent technological advances have expanded the availability of high-throughput biological datasets, opening the way to the reliable design of digital twins of biomedical systems or patients. Such computational tools represent key chemical reaction networks driving perturbation or drug response and can profoundly guide drug discovery and personalized therapeutics. Yet, their development still depends on laborious data integration by the human modeler, so that automated approaches are critically needed. The successes of data-driven system discovery in Physics, rooted in clean datasets and well-defined governing laws, have fueled interest in applying similar techniques in Biology, which presents unique challenges. Here, we reviewed 177 methodologies for automatically inferring digital twins from biological time series, which mostly involved symbolic or sparse regression, and recapitulated them in a Shiny app. We evaluated algorithms according to eight biological and methodological challenges, associated with integrating noisy/incomplete data, multiple conditions, prior knowledge, latent variables, or dealing with high dimensionality, unobserved variable derivatives, candidate library design, and uncertainty quantification. Upon these criteria, sparse regression generally outperformed symbolic regression, particularly when using Bayesian frameworks. Next, deep learning and large language models further emerge as innovative tools to integrate prior knowledge, although their reliability and consistency need to be improved. While no single method addresses all challenges, we argue that progress in learning digital twins will come from hybrid and modular frameworks combining chemical reaction network-based mechanistic grounding, Bayesian uncertainty quantification, and the generative and knowledge integration capacities of deep learning. To support their development, we further highlight key components required for future benchmark development to evaluate methods across all challenges.

Discovering nonlinear PDEs with sparse identification of nonlinear dynamical systems (SINDy) is hindered by high dimensionality, noise, and expensive data acquisition. We propose the greedy sampling neural network for sparse identification of nonlinear PDEs (GN-SINDy), a three–stage framework that integrates strategic sampling, differentiable surrogate modelling, and sparse equation discovery. First, a two–way Q-DEIM–based greedy strategy selects maximally informative space–time samples from snapshot data, drastically reducing data requirements. Second, a deep neural network (DNN) is trained as a differentiable surrogate of the solution field, enabling noise–robust analytic derivatives via automatic differentiation. Third, sparse regression with sparsity–promoting estimators [Brunton, S. L., Proctor, J. L., & Kutz, J. N. (2016a). Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15), 3932–3937; Rudy, S. H., Brunton, S. L., Proctor, J. L., & Kutz, J. N. (2017). Data-driven discovery of partial differential equations. Science Advances, 3(4), e1602614.] is applied to recover the governing PDE. Building on the DeepMoD paradigm, GN-SINDy embeds greedy sampling into data acquisition and stabilises coefficient estimation through neural–enhanced differentiation. We analyze noise robustness, structural stability, and QR–based sampling strategies to guide sampler and hyperparameter selection. Experiments on Burgers', Allen–Cahn, and Korteweg–de Vries equations show that GN-SINDy reliably recovers governing PDEs using under 1 % of the data, outperforming DeepMoD in efficiency, support recovery, and robustness to noise.

High-dimensional data sets are often available in genome-enabled predictions. Such data sets include nonlinear relationships with complex dependence structures. For such situations, vine copula-based (quantile) regression is an important tool. However, the current vine copula-based regression approaches do not scale up to high and ultra-high dimensions. To perform high-dimensional sparse vine copula-based regression, we propose 2 methods. First, we show their superiority regarding computational complexity over the existing methods. Second, we define relevant, irrelevant, and redundant explanatory variables for quantile regression. Then, we show our method's power in selecting relevant variables and prediction accuracy in high-dimensional sparse data sets via simulation studies. Next, we apply the proposed methods to the high-dimensional real data, aiming at the genomic prediction of maize traits. Some data processing and feature extraction steps for the real data are further discussed. Finally, we show the advantage of our methods over linear models and quantile regression forests in simulation studies and real data applications.

The training procedure of the minimal learning machine (MLM) requires the selection of two sets of patterns from the training dataset. These sets are called input reference points (IRP) and output reference points (ORP), which are used to build a mapping between the input geometric configurations and their corresponding outputs. In the original MLM, the number of input reference points is the hyper-parameter and the patterns are chosen at random. Therefore, the conventional proposal does not consider which patterns will belong to each reference point group, since the model does not implement an appropriate way of selecting the most suitable patterns as reference points. Such an approach can impact on the decision function in terms of smoothness, resulting in high complexity models. This paper introduces a new approach to select IRP for MLM applied to classification tasks. The optimally selected minimal learning machine (OS-MLM) relies on the multiresponse sparse regression (MRSR) ranking method and the leave-one-out (LOO) criterion to sort the patterns in terms of relevance and select an appropriate number of input reference points, respectively. The experimental assessment conducted on UCI datasets reports the proposal was able to produce sparser models and achieve competitive performance when compared to the regular strategy of selecting MLM input RPs.

Inferring gene regulatory networks (GRNs) from microarray expression data are an important but challenging issue in systems biology. In this study, the authors propose a Bayesian information criterion (BIC)-guided sparse regression approach for GRN reconstruction. This approach can adaptively model GRNs by optimising the l1-norm regularisation of sparse regression based on a modified version of BIC. The use of the regularisation strategy ensures the inferred GRNs to be as sparse as natural, while the modified BIC allows incorporating prior knowledge on expression regulation and thus avoids the overestimation of expression regulators as usual. Especially, the proposed method provides a clear interpretation of combinatorial regulations of gene expression by optimally extracting regulation coordination for a given target gene. Experimental results on both simulation data and real-world microarray data demonstrate the competent performance of discovering regulatory relationships in GRN reconstruction.

This tutorial provides an exposition of a flexible geometric framework for high-dimensional estimation problems with constraints. The tutorial develops geometric intuition about high-dimensional sets, justifies it with some results of asymptotic convex geometry, and demonstrates connections between geometric results and estimation problems. The theory is illustrated with applications to sparse recovery, matrix completion, quantization, linear and logistic regression, and generalized linear models.KeywordsAsymptotic Convex GeometryHigh-dimensional Estimation ProblemSparse RecoveryExact Recovery ConditionGeneral Feasible SetsThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This study focuses on the key challenge of achieving high-precision guidance for spacecraft: real-time trajectory optimization. The highly efficient sequential convex programming (SCP) is a potential method to address this issue, but its convergence is difficult to guarantee. To enhance the convergence performance of SCP, this paper proposes a high-dimensional merit-function-based SCP method. The main contribution of this paper is threefold. Firstly, the traditional one-dimensional penalty function used to ensure the convergence of SCP is extended to a high-dimensional merit function, which avoids using the penalty parameter that is difficult to determine in advance. Secondly, to match the high-dimensional merit function, a feasibility-then-optimality convergence strategy is proposed, which is essentially an application of the iterative direction descent property in higher dimensions. Finally, based on the high-dimensional merit function and the feasibility-then-optimality strategy, an iterative algorithm is developed, and its convergence analysis is provided under certain assumptions. In numerical simulations, the effectiveness and robustness of the proposed algorithm are validated using the reentry trajectory optimization problem of a lunar capsule.

Multivariate regression is known as a multivariate extension of multiple regression, which explain/predict the variations in multiple dependent variables by multiple independent variables. Recently, various procedures for Sparse Multivariate Regression (SMR) have been proposed, in which a sparse regression coefficient matrix (having a number of zero elements) is obtained aiming to facilitate its interpretation. The procedures for SMR can be classified into the following two types; penalized and cardinality-constrained procedures. In them, the resulting number of zeros in the regression coefficient matrix is controlled/constrained by a prespecified penalty parameter or cardinality value. In this research, we propose another approach for SMR, referred to as Layered Multivariate Regression (LMR). In LMR, the regression coefficient matrix is assumed to be a sum of several sparse matrices, which is called layer. Therefore, the sparseness of the resulting coefficient matrix is controlled by how many layers are used. In LMR, k-th layer can be viewed as the coefficient matrix in the regression of a partial residual (i.e., the residual for all but k-th layer) on independent variables, and thus the variance explained by LMR gets closer to that for the unconstrained regression as the number of layers increases. We present an alternating least squares algorithm for LMR and a procedure for determining how many layers should be used. LMR is assessed in a simulation study and illustrated with a real data example. As an application of LMR, procedures for sparse estimation in some multivariate analysis techniques (e.g., principal component analysis) are also presented.

Supervised sparse manifold regression for head pose estimation in 3D space