Nonparametric Estimation of Regression Level Sets

Optimal bandwidths for local polynomial regression usually involve functionals of the derivatives of the unknown regression function. In the multivariate case, estimates of these functionals are not readily available, primarily because estimating multivariate derivatives is complicated. In this paper, an estimator of multivariate second derivative is obtained via local quadratic regression with cross terms left out. This estimator has the optimal rate of convergence but is simpler and uses a lot less computing time than the full local quadratic estimator. Using this as a pilot estimator, an estimator of the integrated squared Laplacian of a multivariate regression function is obtained which leads to a plug-in formula of the optimal bandwidth for multivariate local linear regression. This bandwidth has good theoretical properties as well as satisfactory performance in our simulation study. It is also recommended for variable selection methods.

Let $X_1, \dots, X_n$ be independent identically distributed observations from an unknown probability density $f(\cdot)$. Consider the problem of estimating the level set $G = G_f(\lambda) = {x \epsilon\mathbb{R}^2: f(x) \geq \lambda}$ from the sample $X_1, \dots, X_n$, under the assumption that the boundary of G has a certain smoothness. We propose piecewise-polynomial estimators of G based on the maximization of local empirical excess masses. We show that the estimators have optimal rates of convergence in the asymptotically minimax sense within the studied classes of densities. We find also the optimal convergence rates for estimation of convex level sets. A generalization to the N-dimensional case, where $N > 2$, is given.

We consider the wavelet-based estimators of mean regression function with long memory moving average errors and investigate their asymptotic rates of convergence based on thresholding of empirical wavelet coefficients. We show that these estimators achieve nearly optimal minimax convergence rates within a logarithmic term over a large range of Besov function classes \(B^{s}_{p,q}\). Therefore, in the presence of long memory non-Gaussian moving average noise, wavelet estimators still achieve nearly optimal convergence rates and provide explicitly the extraordinary local adaptability. The theory is illustrated with some numerical examples.

We propose a general class of randomized gradient estimates to be employed in a recursive search for the minimum of an unknown multivariate regression function. Here only two observations per iteration step are used. Special cases include random direction stochastic approximation (Kushner and Clark), simultaneous perturbation stochastic approximation (Spall) and a special kernel based stochastic approximation method (Polyak and Tsybakov). If the unknown regression is p-smooth ($p\\ge 2$) at the point of minimum, these methods achieve the optimal rate of convergence $O(n^{-(p-1)/(2p)})$. For both the classical stochastic approximation scheme (Kiefer and Wolfowitz) and the averaging scheme (Ruppert and Polyak) the related asymptotic distributions are computed.

Click to increase image sizeClick to decrease image sizeKeywords: Stochastic Approximationconvergence in quadratic meanerror bounds

Nonasymptotic risk bounds are provided for maximum likelihood-type isotonic estimators of an unknown nondecreasing regression function, with general average loss at design points. These bounds are optimal up to scale constants, and they imply uniform $n^{-1/3}$-consistency of the $\ell_p$ risk for unknown regression functions of uniformly bounded variation, under mild assumptions on the joint probability distribution of the data, with possibly dependent observations.

In this paper, we apply semi-Lagrangian discontinuous Galerkin (SLDG) methods for linear hyperbolic equations in one space dimension and analyze the error between the numerical and exact solutions under the L 2 L^2 -norm. In all the previous works, the theoretical analysis of the SLDG method would suggest a suboptimal convergence rate due to the error accumulation over time steps. However, numerical experiments demonstrate an optimal convergence rate and, if the terminal time is large, a superconvergence rate. In this paper, we will prove optimal convergence and optimal superconvergence rates. There are three main difficulties: 1. The error analysis on overlapping meshes. Due to the nature of the semi-Lagrangian time discretization, we need to introduce the background Eulerian mesh and the shifted mesh. The two meshes are staggered, and it is not easy to construct local projections and to handle the error accumulation during time evolution. 2. The superconvergence of time-dependent terms under the L 2 L^2 -norm. The error of the numerical and exact solutions can be divided into two parts, the projection error and the time-dependent superconvergence term. The projection strongly depends on the superconvergence rates. Therefore, we need to construct a sequence of projections and gradually improve the superconvergence rates. 3. The stopping criterion of the sequence of projections. The sequence of projections are basically of the same form. We need to show that the projections exist up to some certain order since the superconvergence rate cannot be infinity. Hence, we will seek some “hidden” condition for the existence of the projections. In this paper, we will solve all the three difficulties and construct several local projections to prove the optimal convergence and superconvergence rates. Numerical experiments verify the theoretical findings.

Max-affine regression refers to a model where the unknown regression function is modeled as a maximum of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> unknown affine functions for a fixed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k \geq 1$ </tex-math></inline-formula> . This generalizes linear regression and (real) phase retrieval, and is closely related to convex regression. We study this problem in the high-dimensional setting assuming that <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> is a fixed constant, and focus on the estimation of the unknown coefficients of the affine functions underlying the model. We analyze a natural alternating minimization (AM) algorithm for the non-convex least squares objective when the design is Gaussian. We show that the AM algorithm, when initialized suitably, converges with high probability and at a geometric rate to a small ball around the optimal coefficients. In order to initialize the algorithm, we propose and analyze a combination of a spectral method and a search algorithm in a low-dimensional space, which may be of independent interest. The final rate that we obtain is near-parametric and minimax optimal (up to a polylogarithmic factor) as a function of the dimension, sample size, and noise variance. In that sense, our approach should be viewed as a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">direct</i> and implementable method of enforcing regularization to alleviate the curse of dimensionality in problems of the convex regression type. Numerical experiments illustrate the sharpness of our bounds in the various problem parameters.

Consider the following nonparametric transformation model $\Lambda (Y)=m(X)+\varepsilon $, where $X$ is a $d$-dimensional covariate, $Y$ is a continuous univariate dependent variable and $\varepsilon $ is an error term with zero mean and which is independent of $X$. We assume that the unknown transformation $\Lambda $ is strictly increasing and that $m$ is an unknown regression function. Our goal is to develop two new nonparametric estimators of the transformation $\Lambda $, the first one based on the least squares loss and the second one based on the least absolute deviation loss, and to compare their performance with that of the estimators developed by Chiappori, Komunjer and Kristensen (J. Econometrics 188 (2015) 22–39). Our proposed estimators are based on an estimator of the conditional distribution of $U$ given $X$, where $U$ is an appropriate transformation of $Y$ that is uniformly distributed. The main motivation for working with $U$ instead of $Y$ is that, in transformation models, the response $Y$ is often skewed with very long tails, and so kernel smoothing based on $Y$ does not work well. Hence, we expect to obtain better estimators if we pre-transform $Y$ before applying kernel smoothing. We establish the asymptotic normality of the two proposed estimators. We also carry out a simulation study to illustrate the performance of our estimators, to compare these new estimators with the ones of Chiappori, Komunjer and Kristensen (J. Econometrics 188 (2015) 22–39) and to see under which model conditions which estimators behave the best.

This work takes advantage of semiparametric modelling which improves significantly in many situations the estimation accuracy of the purely nonparametric approach. Herein for semiparametric estimations of probability mass function (pmf) of count data, and an unknown count regression function (crf), the kernel used is a binomial one and the bandiwdth selection is investigated by developing Bayesian approaches. About the latter, Bayes local and global bandwidth approaches are used to establish data-driven selection procedures in semiparametric framework. From conjugate beta prior distributions of the smoothing parameter and under the squared errors loss function, Bayes estimate for pmf is obtained in closed form. This is not available for the crf which is computed by the Markov Chain Monte Carlo technique. Simulation studies demonstrate that both proposed methods perform better than the classical cross-validation procedures, in particular the smoothing quality and execution times are optimized. All applications are made on real data sets.

The convergence of stochastic interacting particle systems in the mean-field limit to solutions of conservative stochastic partial differential equations is established, with optimal rate of convergence. As a second main result, a quantitative central limit theorem for such SPDEs is derived, again, with optimal rate of convergence. The results apply, in particular, to the convergence in the mean-field scaling of stochastic gradient descent dynamics in overparametrized, shallow neural networks to solutions of SPDEs. It is shown that the inclusion of fluctuations in the limiting SPDE improves the rate of convergence, and retains information about the fluctuations of stochastic gradient descent in the continuum limit.

Hypersonic similarity for steady compressible full Euler flows over two-dimensional Lipschitz wedges

Kirchhoff–Love shell formulation based on triangular isogeometric analysis

Models built with deep neural network (DNN) can handle complicated real-world data extremely well, seemingly without suffering from the curse of dimensionality or the non-convex optimization. To contribute to the theoretical understanding of deep learning, this work studies the nonparametric perspective of DNNs by considering the following questions: (1) What is the underlying estimation problem and what are the most appropriate data assumptions? (2) What is the corresponding optimal convergence rate and does the curse of dimensionality occur? (3) Is the optimal rate achievable for DNN estimators and is there any optimization guarantee? These questions are investigated on two of the most fundamental problems --- regression and classification. Specifically, statistical optimality of DNN estimators is established under various settings with special focuses on the curse of dimensionality and optimization guarantee.In the classic binary classification problem, statistical optimal convergence rates that suffer less from the curse of dimensionality are established under two settings:(1) Under the smooth boundary assumption, I show that DNN classifiers with proper architectures can benefit from the compositional smoothness structure underlying the high dimensional data in the sense that the optimal convergence rates only depend on some effective dimension d*, potentially much smaller than the data dimension d. (2) Under a novel teacher-student framework that assumes the Bayes classifier to be expressed as ReLU neural networks, I obtain a dimension-free rate of convergence O(n^{-2/3}) for DNN classifiers, which is also proven optimal. The optimization of DNN is highly complicated with generally no algorithmic guarantee on finding the global minimizer. To this end, I turn to the recently proposed neural tangent kernel (NTK) literature where the similarity between overparametrized DNN trained by gradient descent (GD) and kernel methods are established. Specifically, through a comprehensive analysis of L2-regularized GD trajectories, I prove that for overparametrized one-hidden-layer ReLU neural networks with L2 regularization, the output from GD is close to that from the kernel ridge regression with the corresponding NTK and optimal rate of L2 estimation error can be achieved.

How well can a regression function be estimated if the distribution of the (random) design is concentrated on a finite set?