A supervised deep learning method for nonparametric density estimation

The Fisher information matrix (FIM) is a widely used measure for applications including statistical inference, information geometry, experiment design, and the study of criticality in biological systems. The FIM is defined for a parametric family of probability distributions and its estimation from data follows one of two paths: either the distribution is assumed to be known and the parameters are estimated from the data or the parameters are known and the distribution is estimated from the data. We consider the latter case which is applicable, for example, to experiments where the parameters are controlled by the experimenter and a complicated relation exists between the input parameters and the resulting distribution of the data. Since we assume that the distribution is unknown, we use a nonparametric density estimation on the data and then compute the FIM directly from that estimate using a finite-difference approximation to estimate the derivatives in its definition. The accuracy of the estimate depends on both the method of nonparametric estimation and the difference Δθ between the densities used in the finite-difference formula. We develop an approach for choosing the optimal parameter difference Δθ based on large deviations theory and compare two nonparametric density estimation methods, the Gaussian kernel density estimator and a novel density estimation using field theory method. We also compare these two methods to a recently published approach that circumvents the need for density estimation by estimating a nonparametric f divergence and using it to approximate the FIM. We use the Fisher information of the normal distribution to validate our method and as a more involved example we compute the temperature component of the FIM in the two-dimensional Ising model and show that it obeys the expected relation to the heat capacity and therefore peaks at the phase transition at the correct critical temperature.

In the linear model y = x′β + u where x is a regressor vector and E(u | x)= 0, we estimate β in E(y | x)= x′β However, the assumption of the linear model, or any nonlinear model for that matter, is a strong one. In nonparametric regression, we try to estimate E(y | x) without specifying the functional form. Since $$E(y{\kern 1pt} |{\kern 1pt} x) = \smallint y \cdot f(y|x)dy = \smallint y \cdot \{ f(y,x)/f(x)\} dy $$ (1.1) if we can estimate f(y, x) and f(x), we can also estimate E(y | x). In this chapter, we study nonparametric density estimation for x as a prelude to nonparametric regression in the next chapter. We will assume that x has a continuous density function f (x) If x is discrete, one can estimate P(x = x 0) either with the same estimation method used for the continuous case or with the number of observations with x i = x 0. There are several nonparametric density estimation methods available. The most popular is “kernel density estimation method” which we explore mainly. Other methods will be examined briefly in the last section. See Prakasa Rao (1983), Silverman (1986), Izenman (1991), Rosenblatt (1991) and Scott (1992) for more on nonparametric density estimation in general.

With the increasing proportion of wind power access in southern China, the impact of the randomness and volatility of wind power generation on power system operation cannot be ignored. Therefore, it is necessary to study the characteristics and random distribution of wind power generation in southern China. Firstly, the influence of time and geographical factors on wind power generation characteristics is analyzed. Secondly, an improved non-parametric kernel density estimation method considering the optimal bandwidth selection of kernel function is proposed to establish the random distribution models of wind power generation in five southern provinces under multiple time scales. The evaluation index system of model fitting effect is also established. Thirdly, the effectiveness of the improved non-parametric kernel density estimation method proposed is verified by comparing with two parameter estimation methods based on typical empirical distribution function. Finally, the differences of probability characteristics of wind power generation in different provinces of southern China are analyzed by daily, quarterly and annual time scales, which can play an auxiliary role in decision-making of power system dispatching, power planning, site selection and capacity determination of wind farms.

The article investigates the accuracy of nonparametric univariate density estimation methods applied to various Gaussian mixture models. A comprehensive comparative analysis is performed for four popular estimation approaches: adaptive kernel density estimation, projection pursuit, log-spline estimation, and wavelet-based estimation. The study is extended with modified versions of these methods, where the sample is first clustered using the EM algorithm based on Gaussian mixture components prior to density estimation. Estimation accuracy is quantitatively evaluated using MAE and MAPE criteria, with simulation experiments conducted over 100,000 replications for various sample sizes. The results show that estimation accuracy strongly depends on the density structure, sample size, and degree of component overlap. Clustering before density estimation significantly improves accuracy for multimodal and asymmetric densities. Although no formal statistical tests are conducted, the performance improvement is validated through non-overlapping confidence intervals obtained from 100,000 simulation replications. In addition, several decision-making systems are compared for automatically selecting the most appropriate estimation method based on the sample’s statistical features. Among the tested systems, kernel discriminant analysis yielded the lowest error rates, while neural networks and hybrid methods showed competitive but more variable performance depending on the evaluation criterion. The findings highlight the importance of using structurally adaptive estimators and automation of method selection in nonparametric statistics. The article concludes with recommendations for method selection based on sample characteristics and outlines future research directions, including extensions to multivariate settings and real-time decision-making systems.

We introduce a recursive method for estimating a probability density subject to constraints of unimodality or monotonicity. It uses an empirical estimate of the probability transform to construct a sequence of maps of a known template, which satisfies the constraints. The algorithm may be employed without a smoothing step, in which case it produces step-function approximations to the sampling density. More satisfactorily, a certain amount of smoothing may be interleaved between each recursion, in which case the estimate is smooth. The amount of smoothing may be chosen using a standard cross-validation algorithm. Unlike other methods for density estimation, however, the recursive approach is robust against variation of the amount of smoothing, and so choice of bandwidth is not critical.

Nonparametric density estimation is a fundamental problem of statistics and data mining. Even though kernel density estimation is the most widely used method, its performance highly depends on the choice of the kernel bandwidth, and it can become computationally expensive for large data sets. We present an adaptive sparse-grid-based density estimation method which discretizes the estimated density function on basis functions centered at grid points rather than on kernels centered at the data points. Thus, the costs of evaluating the estimated density function are independent from the number of data points. We give details on how to estimate density functions on sparse grids and develop a cross validation technique for the parameter selection. We show numerical results to confirm that our sparse-grid-based method is well-suited for large data sets, and, finally, employ our method for the classification of astronomical objects to demonstrate that it is competitive to current kernel-based density estimation approaches with respect to classification accuracy and runtime.

We propose a method for nonparametric density estimation that exhibits robustness to contamination of the training sample. This method achieves robustness by combining a traditional kernel density estimator (KDE) with ideas from classical M-estimation. We interpret the KDE based on a positive semi-definite kernel as a sample mean in the associated reproducing kernel Hilbert space. Since the sample mean is sensitive to outliers, we estimate it robustly via M-estimation, yielding a robust kernel density estimator (RKDE). An RKDE can be computed efficiently via a kernelized iteratively re-weighted least squares (IRWLS) algorithm. Necessary and sufficient conditions are given for kernelized IRWLS to converge to the global minimizer of the M-estimator objective function. The robustness of the RKDE is demonstrated with a representer theorem, the influence function, and experimental results for density estimation and anomaly detection.

Non-parametric density estimation methods are more flexible than parametric methods, due to the fact that they do not assume any specific shape or structure for the data. Most non-parametric methods, like Kernel estimation, require tuning of parameters to achieve good data smoothing, a non-trivial task, even in low dimensions. In higher dimensions, sparsity of data in local neighborhoods becomes a challenge even for non-parametric methods. In this paper, we use the copula transform and two efficient non-parametric methods to develop a new method for improved non-parametric density estimation in multivariate domain. After separation of marginal and joint densities using copula transform, a diffusion-based kernel estimator is employed to estimate the marginals. Next, Bayesian sequential partitioning (BSP) is used in the joint density estimation.

Nonparametric Density Estimation and Tests of Continuous Time Interest Rate Models

This work reviews the literature on spline local basis methods for non-parametric density estimation. Particular attention is paid to B-spline density estimators which have experienced recent advances in both theory and methodology. These estimators occupy a very interesting space in statistics, which lies aptly at the cross-section of numerous statistical frameworks. New insights, experiments, and analyses are presented to cast the various estimation concepts in a unified context, while parallels and contrasts are drawn to the more familiar contexts of kernel density estimation. Unlike kernel density estimation, the study of local basis estimation is not yet fully mature, and this work also aims to highlight the gaps in existing literature which merit further investigation.

Density estimation plays a fundamental role in many areas of statistics and machine learning. Parametric, nonparametric, and semiparametric density estimation methods have been proposed in the literature. Semiparametric density models are flexible in incorporating domain knowledge and uncertainty regarding the shape of the density function. Existing literature on semiparametric density models is scattered and lacks a systematic framework. In this article, we consider a unified framework based on reproducing kernel Hilbert space for modeling, estimation, computation, and theory. We propose general semiparametric density models for both a single sample and multiple samples which include many existing semiparametric density models as special cases. We develop penalized likelihood based estimation methods and computational methods under different situations. We establish joint consistency and derive convergence rates of the proposed estimators for both finite dimensional Euclidean parameters and an infinite-dimensional functional parameter. We validate our estimation methods empirically through simulations and an application. Supplementary materials for this article are available online.

Kernel density estimation (KDE) is one of the most widely used nonparametric density estimation methods. The fact that it is a memory-based method, i.e., it uses the entire training data set for prediction, makes it unsuitable for most current big data applications. Several strategies, such as tree-based or hashing-based estimators, have been proposed to improve the efficiency of the kernel density estimation method. The novel density kernel density estimation method (DMKDE) uses density matrices, a quantum mechanical formalism, and random Fourier features, an explicit kernel approximation, to produce density estimates. This method has its roots in the KDE and can be considered as an approximation method, without its memory-based restriction. In this paper, we systematically evaluate the novel DMKDE algorithm and compare it with other state-of-the-art fast procedures for approximating the kernel density estimation method on different synthetic data sets. Our experimental results show that DMKDE is on par with its competitors for computing density estimates and advantages are shown when performed on high-dimensional data. We have made all the code available as an open source software repository.

We consider the problem of choosing between two density estimates, a non-parametric estimate with the the standard properties of nonparametric estimates (universal consistency, robustness, but not extremely good rate of convergence) and a special estimate designed to perform well on a given set T of densities. The special estimate can often be thought of as a parametric estimate. The selection we propose is based upon the L1 distance oetween both estimates. Among otner things, we show how one should proceed to insure that the selected estimate matches the special estimate's rate on T, and that it matches the nonparametric estimate's rate off T

We examine how Bayesian network (BN) learning and analysis methods can help to meet several methodological challenges that arise in interpreting significant regression coefficients in exposure–response regression modeling. As a motivating example, we consider the challenge of interpreting positive regression coefficients for blood lead level (BLL) as a predictor of mortality risk for nonsmoking men. We first note that practices such as dichotomizing or categorizing continuous confounders (e.g. income), omitting potentially important socioeconomic confounders (e.g. education), and assuming specific parametric regression model forms leave unclear to what extent a positive regression coefficient reflects these modeling choices, rather than a direct dependence of mortality risk on exposure. Therefore, significant exposure–response coefficients in parametric regression models do not necessarily reveal the extent to which reducing exposure-related variables (e.g. BLL) alone, while leaving fixed other correlates of exposure and mortality risks (e.g. education, income, etc.) would reduce adverse outcome risks (e.g. mortality risks). We then consider how BN structure-learning and inference algorithms and nonparametric estimation methods (partial dependence plots) can be used to clarify dependencies between variables, variable selection, confounding, and quantification of joint effects of multiple factors on risk, including possible high-order interactions and nonlinearities. We conclude that these details must be carefully modeled to determine whether a data set provides evidence that exposure itself directly affects risks; and that BN and nonparametric effect estimation and uncertainty quantification methods can complement regression modeling and help to improve the scientific basis for risk management decisions and policy-making by addressing these issues.

This chapter examines how the Bayesian network (BN) learning and analysis methods discussed I Chap. 9 can help to meet several methodological challenges that arise in interpreting significant regression coefficients in exposure-response regression modeling. As a motivating example, consider the challenge of interpreting positive regression coefficients for blood lead level (BLL) as a predictor of mortality risk for nonsmoking men. Practices such as dichotomizing or categorizing continuous confounders (e.g., income), omitting potentially important socioeconomic confounders (e.g., education), and assuming specific parametric regression model forms leave unclear to what extent a positive regression coefficient reflects these modeling choices, rather than a direct dependence of mortality risk on exposure. Therefore, significant exposure-response coefficients in parametric regression models do not necessarily reveal the extent to which reducing exposure-related variables (e.g., BLL) alone, while leaving fixed other correlates of exposure and mortality risks (e.g., education, income etc.) would reduce adverse outcome risks (e.g., mortality risks). This chapter illustrates how BN structure-learning and inference algorithms and nonparametric estimation methods (partial dependence plots) from Chap. 9 can be used to clarify dependencies between variables, variable selection, confounding, and quantification of joint effects of multiple factors on risk, including possible high-order interactions and nonlinearities. It concludes that these details must be carefully modeled to determine whether a data set provides evidence that exposure itself directly affects risks; and that BN and nonparametric effect estimation and uncertainty quantification methods can complement regression modeling and help to improve the scientific basis for risk management decisions and policy-making by addressing these issues.