Nonparametric Estimation and Symmetry Tests for Conditional Density Functions

A kernel-based parametric method for conditional density estimation

Methods of density estimation on the Grassmann manifold

The Fisher information matrix (FIM) is a widely used measure for applications including statistical inference, information geometry, experiment design, and the study of criticality in biological systems. The FIM is defined for a parametric family of probability distributions and its estimation from data follows one of two paths: either the distribution is assumed to be known and the parameters are estimated from the data or the parameters are known and the distribution is estimated from the data. We consider the latter case which is applicable, for example, to experiments where the parameters are controlled by the experimenter and a complicated relation exists between the input parameters and the resulting distribution of the data. Since we assume that the distribution is unknown, we use a nonparametric density estimation on the data and then compute the FIM directly from that estimate using a finite-difference approximation to estimate the derivatives in its definition. The accuracy of the estimate depends on both the method of nonparametric estimation and the difference Δθ between the densities used in the finite-difference formula. We develop an approach for choosing the optimal parameter difference Δθ based on large deviations theory and compare two nonparametric density estimation methods, the Gaussian kernel density estimator and a novel density estimation using field theory method. We also compare these two methods to a recently published approach that circumvents the need for density estimation by estimating a nonparametric f divergence and using it to approximate the FIM. We use the Fisher information of the normal distribution to validate our method and as a more involved example we compute the temperature component of the FIM in the two-dimensional Ising model and show that it obeys the expected relation to the heat capacity and therefore peaks at the phase transition at the correct critical temperature.

In the linear model y = x′β + u where x is a regressor vector and E(u | x)= 0, we estimate β in E(y | x)= x′β However, the assumption of the linear model, or any nonlinear model for that matter, is a strong one. In nonparametric regression, we try to estimate E(y | x) without specifying the functional form. Since $$E(y{\kern 1pt} |{\kern 1pt} x) = \smallint y \cdot f(y|x)dy = \smallint y \cdot \{ f(y,x)/f(x)\} dy $$ (1.1) if we can estimate f(y, x) and f(x), we can also estimate E(y | x). In this chapter, we study nonparametric density estimation for x as a prelude to nonparametric regression in the next chapter. We will assume that x has a continuous density function f (x) If x is discrete, one can estimate P(x = x 0) either with the same estimation method used for the continuous case or with the number of observations with x i = x 0. There are several nonparametric density estimation methods available. The most popular is “kernel density estimation method” which we explore mainly. Other methods will be examined briefly in the last section. See Prakasa Rao (1983), Silverman (1986), Izenman (1991), Rosenblatt (1991) and Scott (1992) for more on nonparametric density estimation in general.

This work reviews the literature on spline local basis methods for non-parametric density estimation. Particular attention is paid to B-spline density estimators which have experienced recent advances in both theory and methodology. These estimators occupy a very interesting space in statistics, which lies aptly at the cross-section of numerous statistical frameworks. New insights, experiments, and analyses are presented to cast the various estimation concepts in a unified context, while parallels and contrasts are drawn to the more familiar contexts of kernel density estimation. Unlike kernel density estimation, the study of local basis estimation is not yet fully mature, and this work also aims to highlight the gaps in existing literature which merit further investigation.

For eight years (2007–2014), Greeneridge Sciences deployed, collected, and analyzed an immense passive acoustic dataset to study the response of bowhead whales (Balaena mysticetus) to oil exploration activities along a 280 km swath of the Alaskan Arctic continental shelf. Each year, up to 40 Directional Autonomous Seafloor Acoustic Recorders (DASARs) detected bowhead calls during the whales’ annual fall migration, resulting in over 2.4 million localized calls over the study period. These enormous sample sizes, combined with the unique localization capability of the DASARs, have enabled the investigation and cross-validation of several acoustic density estimation (DE) methods. Here, we compared results among three density estimation methods: (i) direct census, (ii) distance sampling, and (iii) spatially explicit capture recapture (SECR). In the course of our investigation, we encountered and addressed two challenges: (i) how to apply point transect theory to distributed array systems, and (ii) how to implement SECR in situations where detections between sensors are not statistically independent or where false positives exist on single sensors. These density estimates were then used to estimate relative abundance of the Western Arctic bowhead whale population over multiple years. [Work sponsored by the Joint Industry Programme.]

We address the problem of nonparametric estimation of characteristics for stationary and ergodic time series. We consider finite-alphabet time series and the real-valued ones and the following problems: estimation of the (limiting) probability P(u0 hellip us) for every s and each sequence u0 hellip us of letters from the process alphabet (or estimation of the density p(x0, hellip , xs) for real-valued time series), so-called on-line prediction, where the conditional probability P(xt+1/x1x2 hellip xt) (or the conditional density p(xt+1/x1x2 hellip xt)) should be estimated (in the case where x1x2 hellip xt is known), regression and classification (or so-called problems with side information). We show that any universal code (or a universal data compressor) can be used as a basis for constructing asymptotically optimal methods for the above problems.

We introduce a recursive method for estimating a probability density subject to constraints of unimodality or monotonicity. It uses an empirical estimate of the probability transform to construct a sequence of maps of a known template, which satisfies the constraints. The algorithm may be employed without a smoothing step, in which case it produces step-function approximations to the sampling density. More satisfactorily, a certain amount of smoothing may be interleaved between each recursion, in which case the estimate is smooth. The amount of smoothing may be chosen using a standard cross-validation algorithm. Unlike other methods for density estimation, however, the recursive approach is robust against variation of the amount of smoothing, and so choice of bandwidth is not critical.

Nonparametric density estimation is a fundamental problem of statistics and data mining. Even though kernel density estimation is the most widely used method, its performance highly depends on the choice of the kernel bandwidth, and it can become computationally expensive for large data sets. We present an adaptive sparse-grid-based density estimation method which discretizes the estimated density function on basis functions centered at grid points rather than on kernels centered at the data points. Thus, the costs of evaluating the estimated density function are independent from the number of data points. We give details on how to estimate density functions on sparse grids and develop a cross validation technique for the parameter selection. We show numerical results to confirm that our sparse-grid-based method is well-suited for large data sets, and, finally, employ our method for the classification of astronomical objects to demonstrate that it is competitive to current kernel-based density estimation approaches with respect to classification accuracy and runtime.

Kernel density estimation (KDE) is one of the most widely used nonparametric density estimation methods. The fact that it is a memory-based method, i.e., it uses the entire training data set for prediction, makes it unsuitable for most current big data applications. Several strategies, such as tree-based or hashing-based estimators, have been proposed to improve the efficiency of the kernel density estimation method. The novel density kernel density estimation method (DMKDE) uses density matrices, a quantum mechanical formalism, and random Fourier features, an explicit kernel approximation, to produce density estimates. This method has its roots in the KDE and can be considered as an approximation method, without its memory-based restriction. In this paper, we systematically evaluate the novel DMKDE algorithm and compare it with other state-of-the-art fast procedures for approximating the kernel density estimation method on different synthetic data sets. Our experimental results show that DMKDE is on par with its competitors for computing density estimates and advantages are shown when performed on high-dimensional data. We have made all the code available as an open source software repository.

Regression is a fundamental problem in statistical data analysis, which aims at estimating the conditional mean of output given input. However, regression is not informative enough if the conditional probability density is multi-modal, asymmetric, and heteroscedastic. To overcome this limitation, various estimators of conditional densities themselves have been developed, and a kernel-based approach called least-squares conditional density estimation (LS-CDE) was demonstrated to be promising. However, LS-CDE still suffers from large estimation error if input contains many irrelevant features. In this paper, we therefore propose an extension of LS-CDE called sparse additive CDE (SA-CDE), which allows automatic feature selection in CDE. SA-CDE applies kernel LS-CDE to each input feature in an additive manner and penalizes the whole solution by a group-sparse regularizer. We also give a subgradient-based optimization method for SA-CDE training that scales well to high-dimensional large data sets. Through experiments with benchmark and humanoid robot transition datasets, we demonstrate the usefulness of SA-CDE in noisy CDE problems.

Nonparametric density estimation is an unsupervised learning problem. In this work we propose a two-step procedure that casts the density estimation problem in the first step into a supervised regression problem. The advantage is that we can afterwards apply supervised learning methods. Compared to the standard nonparametric regression setting, the proposed procedure creates, however, dependence among the training samples. To derive statistical risk bounds, one can therefore not rely on the well-developed theory for i.i.d. data. To overcome this, we prove an oracle inequality for this specific form of data dependence. As an application, it is shown that under a compositional structure assumption on the underlying density, the proposed two-step method achieves convergence rates that are faster than the standard nonparametric rates. A simulation study illustrates the finite sample performance.

In many data stream mining applications, traditional density estimation methods such as kernel density estimation, reduced set density estimation can not be applied to the density estimation of data streams because of their high computational burden, processing time and intensive memory allocation requirement. In order to reduce the time and space complexity, a novel density estimation method Dm-KDE over data streams based on the proposed algorithm m-KDE which can be used to design a KDE estimator with the fixed number of kernel components for a dataset is proposed. In this method, Dm-KDE sequence entries are created by algorithm m-KDE instead of all kernels obtained from other density estimation methods. In order to further reduce the storage space, Dm-KDE sequence entries can be merged by calculating their KL divergences. Finally, the probability density functions over arbitrary time or entire time can be estimated through the obtained estimation model. In contrast to the state-of-the-art algorithm SOMKE, the distinctive advantage of the proposed algorithm Dm-KDE exists in that it can achieve the same accuracy with much less fixed number of kernel components such that it is suitable for the scenarios where higher on-line computation about the kernel density estimation over data streams is required.We compare Dm-KDE with SOMKE and M-kernel in terms of density estimation accuracy and running time for various stationary datasets. We also apply Dm-KDE to evolving data streams. Experimental results illustrate the effectiveness of the proposed method.

We propose a method for nonparametric density estimation that exhibits robustness to contamination of the training sample. This method achieves robustness by combining a traditional kernel density estimator (KDE) with ideas from classical M-estimation. We interpret the KDE based on a positive semi-definite kernel as a sample mean in the associated reproducing kernel Hilbert space. Since the sample mean is sensitive to outliers, we estimate it robustly via M-estimation, yielding a robust kernel density estimator (RKDE). An RKDE can be computed efficiently via a kernelized iteratively re-weighted least squares (IRWLS) algorithm. Necessary and sufficient conditions are given for kernelized IRWLS to converge to the global minimizer of the M-estimator objective function. The robustness of the RKDE is demonstrated with a representer theorem, the influence function, and experimental results for density estimation and anomaly detection.

A conditional beta probability density function is employed to model time-delay measurement errors associated with sensors used in triangulation ranging applications. This type of density function is less restrictive than the beta density used in previous analyses in that end points can be extended out beyond the imposed range constraints and data editing is allowed before range calculations. The resulting conditional range density function is computed along with the mean and variance. Comparisons are made of the pertinent range statistics associated with the beta and conditional beta density functions. Finally, the conditional beta density function range statistics are compared with those obtained from a gated Gaussian density function for equivalent values of time-delay measurement error standard deviation. The Gaussian statistics are determined via Monte-Carlo simulation.