On the convergence of gradient descent for robust functional linear regression

Checking the adequacy of functional linear quantile regression model

Point process data have become increasingly popular these days. For example, many of the data captured in electronic health records (EHR) are in the format of point process data. It is of great interest to study the association between a point process predictor and a scalar response using generalized functional linear regression models. Various generalized functional linear regression models have been developed under different settings in the past decades. However, existing methods can only deal with functional or longitudinal predictors, not point process predictors. In this article, we propose a novel generalized functional linear regression model for a point process predictor. Our proposed model is based on the joint modeling framework, where we adopt a log-Gaussian Cox process model for the point process predictor and a generalized linear regression model for the outcome. We also develop a new algorithm for fast model estimation based on the Gaussian variational approximation method. We conduct extensive simulation studies to evaluate the performance of our proposed method and compare it to competing methods. The performance of our proposed method is further demonstrated on an EHR dataset of patients admitted into the intensive care units of the Beth Israel Deaconess Medical Center between 2001 and 2008.

Functional linear regression model with randomly censored data: Predicting conversion time to Alzheimer ’s disease

Inference in functional linear quantile regression

This paper considers a robust piecewise linear regression model with an unknown number of change points. Our estimation framework mainly contains two steps: First, we combine the linearization technique with rank-based estimators to estimate the regression coefficients and the location of thresholds simultaneously, given a large number of change points. The associated inferences for all the parameters are easily derived. Second, we use the LARS algorithm via generalized BIC to refine the candidate threshold estimates and obtain the ultimate estimators. The rank-based regression guarantees that our estimators are less sensitive to outliers and heavy-tailed data, and therefore achieves robustness. Simulation studies and an empirical example on BMI and age relationship illustrate the proposed method.

Large annotated datasets inevitably contain noisy labels, which poses a major challenge for training deep neural networks as they easily memorize the labels. Noise-robust loss functions have emerged as a notable strategy to counteract this issue, but it remains challenging to create a robust loss function which is not susceptible to underfitting. Through a quantitative approach, this paper explores the limited overlap between the network output at initialization and regions of non-vanishing gradients of bounded loss functions in the initial learning phase. Using these insights, we address underfitting of several noise robust losses with a novel method denoted as logit bias, which adds a real number epsilon to the logit at the position of the correct class. The logit bias enables these losses to achieve state-of-the-art results, even on datasets like WebVision, consisting of over a million images from 1000 classes. In addition, we demonstrate that our method can be used to determine optimal parameters for several loss functions – without having to train networks. Remarkably, our method determines the hyperparameters based on the number of classes, resulting in loss functions which require zero dataset or noise-dependent parameters.

This study examines the optimal selections of bandwidth and semi-metric for a functional partial linear model. Our proposed method begins by estimating the unknown error density using a kernel density estimator of residuals, where the regression function, consisting of parametric and nonparametric components, can be estimated by functional principal component and functional Nadayara-Watson estimators. The estimation accuracy of the regression function and error density crucially depends on the optimal estimations of bandwidth and semi-metric. A Bayesian method is utilized to simultaneously estimate the bandwidths in the regression function and kernel error density by minimizing the Kullback-Leibler divergence. For estimating the regression function and error density, a series of simulation studies demonstrate that the functional partial linear model gives improved estimation and forecast accuracies compared with the functional principal component regression and functional nonparametric regression. Using a spectroscopy dataset, the functional partial linear model yields better forecast accuracy than some commonly used functional regression models. As a by-product of the Bayesian method, a pointwise prediction interval can be obtained, and marginal likelihood can be used to select the optimal semi-metric.

Sparse wavelet regression with multiple predictive curves

Our work focuses on the functional linear model given by \(Y=\langle\theta,X\rangle+\epsilon,\) where Y and e are real random variables, X is a zero-mean random variable valued in a Hilbert space \((\mathcal{H},\langle\cdot,\cdot\rangle)\), and \(\theta\in\mathcal{H}\) is the fixed model parameter. Using an initial sample \(\{(X_i,Y_i)\}_{i=1}^n\), a bootstrap resampling \(Y_i^{*}=\langle\hat{\theta},X_i\rangle+\hat{\epsilon}_i^{*}\), \(i=1,\ldots,n\), is proposed, where \(\hat{\theta}\) is a general pilot estimator, and \(\hat{\epsilon}_i^{*}\) is a naive or wild bootstrap error. The obtained consistency of bootstrap allows us to calibrate distributions as \(P_X\{\sqrt{n}(\langle\hat{\theta},x\rangle-\langle\theta,x\rangle)\leq y\}\) for a fixed x, where P X is the probability conditionally on \(\{X_i\}_{i=1}^n\). Different applications illustrate the usefulness of bootstrap for testing different hypotheses related with θ, and a brief simulation study is also presented.

Functional linear regression models have been widely used in the gene association analysis of complex traits. These models retain all the genetic information in the data and take full advantage of spatial information in genetic variation data, which leads to brilliant detection power. However, the significant association signals identified by the high-power methods are not all the real causal SNPs, because it is easy to regard noise information as significant association signals, leading to a false association. In this paper, a method based on the sparse functional data association test (SFDAT) of gene region association analysis is developed based on a functional linear regression model with local sparse estimation. The evaluation indicators CSR and DL are defined to evaluate the feasibility and performance of the proposed method with other indicators. Simulation studies show that: (1) SFDAT performs well under both linkage equilibrium and linkage disequilibrium simulation; (2) SFDAT performs successfully for gene regions (including common variants, low-frequency variants, rare variants and mix variants); (3) With power and type I error rates comparable to OLS and Smooth, SFDAT has a better ability to handle the zero regions. The Oryza sativa data set is analyzed by SFDAT. It is shown that SFDAT can better perform gene association analysis and eliminate the false positive of gene localization. This study showed that SFDAT can lower the interference caused by noise while maintaining high power. SFDAT provides a new method for the association analysis between gene regions and phenotypic quantitative traits.

This paper develops a point impact linear regression model in which the trajectory of a continuous stochastic process, when evaluated at a "sensitive time point", is associated with a scalar response. The proposed model complements and is more interpretable than the functional linear regression approach that has become popular in recent years. The trajectories are assumed to have fractal (self-similar) properties in common with a fractional Brownian motion with an unknown Hurst exponent. Bootstrap confidence intervals based on the least-squares estimator of the sensitive time point are developed. Misspecification of the point impact model by a functional linear model is also investigated. Non-Gaussian limit distributions and rates of convergence determined by the Hurst exponent play an important role.

In this paper, we consider rank estimation for partial functional linear regression models based on functional principal component analysis. The proposed rank-based method is robust to outliers in the errors and highly efficient under a wide range of error distributions. The asymptotic properties of the resulting estimators are established under some regularity conditions. A simulation study conducted to investigate the finite sample performance of the proposed estimators shows that the proposed rank method performs well. Furthermore, the proposed methodology is illustrated by means of an analysis of the Berkeley growth data.

Model checks for functional linear regression models based on projected empirical processes

Machine learning models are often learned by minimising a loss function on the training data using a gradient descent algorithm. These models often suffer from overfitting, leading to a decline in predictive performance on unseen data. A standard solution is early stopping using a hold-out validation set, which halts the minimisation when the validation loss stops decreasing. However, this hold-out set reduces the data available for training. This paper presents GRADSTOP, a novel stochastic early stopping method that only uses information in the gradients, which are produced by the gradient descent algorithm “for free.” Our main contributions are that we estimate the Bayesian posterior by the gradient information, define the early stopping problem as drawing sample from this posterior, and use the approximated posterior to obtain a stopping criterion. Our empirical evaluation shows that GRADSTOP achieves a small loss on test data and compares favourably to a validation-set-based stopping criterion. By leveraging the entire dataset for training, our method is particularly advantageous in data-limited settings, such as transfer learning. It can be incorporated as an optional feature in gradient descent libraries with only a small computational overhead. The source code is available at https://github.com/edahelsinki/gradstop.

We study smooth backfitting when there are errors-in-variables, which is motivated by functional additive models for a functional regression model with a scalar response and multiple functional predictors that are additive in the functional principal components of the predictor processes. The development of a new smooth backfitting technique for the estimation of the additive component functions in functional additive models with multiple functional predictors requires to address the difficulty that the eigenfunctions and therefore the functional principal components of the predictor processes, which are the arguments of the proposed additive model, are unknown and need to be estimated from the data. The available estimated functional principal components contain an error that is small for large samples but nevertheless affects the estimation of the additive component functions. This error-in-variables situation requires to develop new asymptotic theory for smooth backfitting. Our analysis also pertains to general situations where one encounters errors in the predictors for an additive model, when the errors become smaller asymptotically. We also study the finite sample properties of the proposed method for the application in functional additive regression through a simulation study and a real data example.