Numerical performance of penalized comparison to overfitting for multivariate kernel density estimation

A Bayesian approach to bandwidth selection for multivariate kernel density estimation

Density estimation is an important aspect of statistics. Statistical inference often requires the knowledge of observed data density. A common method of density estimation is the kernel density estimation (KDE). It is a nonparametric estimation approach which requires a kernel function and a window size (smoothing parameter H). It aids density estimation and pattern recognition. So, this work focuses on the use of a modified intersection of confidence intervals (MICIH) approach in estimating density. The Nigerian crime rate data reported to the Police as reported by the National Bureau of Statistics was used to demonstrate this new approach. This approach in the multivariate kernel density estimation is based on the data. The main way to improve density estimation is to obtain a reduced mean squared error (MSE), the errors for this approach was evaluated. Some improvements were seen. The aim is to achieve adaptive kernel density estimation. This was achieved under a sufficiently smoothing technique. This adaptive approach was based on the bandwidths selection. The quality of the estimates obtained of the MICIH approach when applied, showed some improvements over the existing methods. The MICIH approach has reduced mean squared error and relative faster rate of convergence compared to some other approaches. The approach of MICIH has reduced points of discontinuities in the graphical densities the datasets. This will help to correct points of discontinuities and display adaptive density. Keywords: approach, bandwidth, estimate, error, kernel density

In practical classification problems, imbalanced class distribution is very common. Traditional classification methods usually aim at optimizing the overall accuracy, which leads to classification decisions biased towards categories with a large number of samples. Similar to most traditional classifiers, Naive Bayes (NB) is sensitive to the class prior distribution and has poor recognition performance for minority class samples, which restricts its application in imbalanced classification. Therefore, we propose a Bayesian classifier based on distance-corrected multivariate kernel density estimation (BC-DMKDE). BC-DMKDE relaxes the assumptions regarding conditional independence and the normality of continuous variables in NB classifiers, by incorporating the multivariate kernel density estimation (MKDE). We also incorporate a distance-correction mechanism utilizing kernel density estimation, specifically to alleviate the impact of class imbalance and overlapping data samples. Across 18 real-world benchmarks, BC-DMKDE exhibits superior classification performance and robustness, outperforming existing methods at both the algorithm and data levels.

Abstract: Home‐range estimators are commonly tested with simulated animal locational data in the laboratory before the estimators are used in practice. Although kernel density estimation (KDE) has performed well as a home‐range estimator for simulated data, several recent studies have reported its poor performance when used with data collected in the field. This difference may be because KDE and other home‐range estimators are generally tested with simulated point locations that follow known statistical distributions, such as bivariate normal mixtures, which may not represent well the space‐use patterns of all wildlife species. We used simulated animal locational data of 5 point pattern shapes that represent a range of wildlife utilization distributions to test 4 methods of home‐range estimation: 1) KDE with reference bandwidths, 2) KDE with least‐squares cross‐validation, 3) KDE with plug‐in bandwidths, and 4) minimum convex polygon (MCP). For the point patterns we simulated, MCP tended to produce more accurate area estimates than KDE methods. However, MCP estimates were markedly unstable, with bias varying widely with both sample size and point pattern shape. The KDE methods performed best for concave distributions, which are similar to bivariate normal mixtures, but still overestimated home ranges by about 40–50% even in the best cases. For convex, linear, perforated, and disjoint point patterns, KDE methods overestimated home‐range sizes by 50–300%, depending on sample size and method of bandwidth selection. These results indicate that KDE does not produce home‐range estimates that are as accurate as the literature suggests, and we recommend exploring other techniques of home‐range estimation.

This paper presents fabrication of complex surficial micro-features employing a cross-innovative hybrid process inspired from lithography and Jet-ECM. The process is referred here as mask electrolyte jet machining (MEJM). MEJM is a non-contact machining process which combines high resolution of lithography and greater flexibility of Jet-ECM. It is a non-contact process which can fabricate variety of microstructures on difficult-to-machine materials without need of expensive tooling. The presented work demonstrates the process performance of this technology by statistical analysis and multivariate kernel density estimation (KDE) based on probabilistic density function. Micro-letters are fabricated as an example of complex surficial structure comprising of multiple intersecting, straight and curved grooves. The processing response is characterized in terms of geometrical size, similarity ratio, and cumulative shape deviation. Experimental results demonstrated that micro letters with good repeatability (minimum SD of shape error ratio 0.297%) and shape accuracy (minimum shape error of 0.039%) can be fabricated with this technology. The results suggest MEJM could be a promising technology for batch manufacturing of surface microstructures with high productivity.

A novel maximum a posteriori (MAP) method for dynamic single-photon emission computed tomography image reconstruction is proposed. The prior probability is modeled as a multivariate kernel density estimator (KDE), effectively modeling the prior probability non-parametrically, with the aim of reducing the effects of artifacts arising from inconsistencies in projection measurements in low-count regimes where projections are dominated by noise. The proposed prior spatially and temporally limits the variation of time-activity functions (TAFs) and attracts similar TAFs together. The similarity between TAFs is determined by the spatial and range scaling parameters of the KDE-like prior. The resulting iterative image reconstruction method is evaluated using two simulated phantoms, namely the extended cardiac-torso (XCAT) heart phantom and a simulated Mini-Deluxe Phantom. The phantoms were chosen to observe the effects of the proposed prior on the TAFs based on the vicinity and abutments of regions with different activities. Our results show the effectiveness of the proposed iterative reconstruction method, especially in low-count regimes, which provides better uniformity within each region of activity, significant reduction of spatiotemporal variations caused by noise, and sharper separation between different regions of activity than expectation maximization and an MAP method employing a more traditional Gibbs prior.

A bandwidth selection method that combines the concept of least-squares cross-validation and the plug-in approach is being introduced in connection with kernel density estimation. A simulation study reveals that this hybrid methodology outperforms some commonly used bandwidth selection rules. It is shown that the proposed approach can also be readily employed in the context of variable kernel density estimation. We conclude with two illustrative examples.

Since conventional cross-validation bandwidth selection methods don't work for the case where the data considered are dependent time series, alternative bandwidth selection methods are needed. In recent years, Bayesian based global bandwidth selection methods have been proposed. Our experience shows that the use of a global bandwidth is however less suitable than using a localized bandwidth in kernel density estimation in the case where the data are dependent time series as discussed in an empirical application of this paper. Nonetheless, a difficult issue is how we can consistently estimate a localized bandwidth. In this paper, we propose a semiparametric estimation method, for which we establish an asymptotic theory for the proposed semiparametric estimator. A by-product of this bandwidth estimate is a new sampling-based likelihood approach to hyperparameter estimation. Monte Carlo simulation studies show that the proposed hyperparameter estimation method works very well, and that the proposed bandwidth estimator outperforms its competitors. Applications of the new bandwidth estimator to the kernel density estimation of Eurodollar deposit rate, as well as the S&P 500 daily return under conditional heteroscedasticity, demonstrate the effectiveness and competitiveness of the proposed semiparametric localized bandwidth.

Exploratory data analysis (EDA) can reveal important features of underlying distributions, and these features often have an impact on inferences and conclusions drawn from data. Graphical analysis is central to EDA, and graphical representations of distributions often benefit from smoothing. A viable method of estimating and graphing the underlying density in EDA is kernel density estimation (KDE). This article provides an introduction to KDE and examines alternative methods for specifying the smoothing bandwidth in terms of their ability to recover the true density. We also illustrate the comparison and use of KDE methods with 2 empirical examples. Simulations were carried out in which we compared 8 bandwidth selection methods (Sheather-Jones plug-in [SJDP], normal rule of thumb, Silverman's rule of thumb, least squares cross-validation, biased cross-validation, and 3 adaptive kernel estimators) using 5 true density shapes (standard normal, positively skewed, bimodal, skewed bimodal, and standard lognormal) and 9 sample sizes (15, 25, 50, 75, 100, 250, 500, 1,000, 2,000). Results indicate that, overall, SJDP outperformed all methods. However, for smaller sample sizes (25 to 100) either biased cross-validation or Silverman's rule of thumb was recommended, and for larger sample sizes the adaptive kernel estimator with SJDP was recommended. Information is provided about implementing the recommendations in the R computing language.

Nonparametric estimation for a probability density function that describes multivariate data has typically been addressed by kernel density estimation (KDE). A novel density estimator recently developed by Farmer and Jacobs offers an alternative high-throughput automated approach to univariate nonparametric density estimation based on maximum entropy and order statistics, improving accuracy over univariate KDE. This article presents an extension of the single variable case to multiple variables. The univariate estimator is used to recursively calculate a product array of one-dimensional conditional probabilities. In combination with interpolation methods, a complete joint probability density estimate is generated for multiple variables. Good accuracy and speed performance in synthetic data are demonstrated by a numerical study using known distributions over a range of sample sizes from 100 to 106 for two to six variables. Performance in terms of speed and accuracy is compared to KDE. The multivariate density estimate developed here tends to perform better as the number of samples and/or variables increases. As an example application, measurements are analyzed over five filters of photometric data from the Sloan Digital Sky Survey Data Release 17. The multivariate estimation is used to form the basis for a binary classifier that distinguishes quasars from galaxies and stars with up to 94% accuracy.

Kernel density estimation is a widely used statistical tool and bandwidth selection is critically important. The Sheather and Jones’ (SJ) selector [A reliable data-based bandwidth selection method for kernel density estimation, J. R. Stat. Soc. Ser. B 53 (1991), pp. 683–690] remains the best available data-driven bandwidth selector. It can, however, perform poorly if the true density deviates too much in shape from normal. This paper first develops an alternative selector by following ideas in Park and Marron [On the use of pilot estimators in bandwidth selection, Nonparametr. Stat. 1 (1992), pp. 231–240] to reduce the impact of the normal reference density. The selector can bring drastic improvement to less smooth densities that the SJ selector has difficulty with, but may be slightly worse off otherwise. We then propose to combine the alternative selector and SJ selector by using the smaller of the two bandwidths, which has the effect of automatically picking the better one for a particular density. In our extensive simulation, study using the 15 benchmark densities in Marron and Wand [Exact mean integrated squared error, Ann. Statist. 20 (1992), pp. 712–736], the combined selector significantly improves the SJ selector for 5 difficult densities and retains the superior performance of the SJ selector for the other 10. A ready-to-use R function is provided.

A variable bandwidth selector in multivariate kernel density estimation

Kernel density estimation is an important data smoothing technique. It has been applied most successfully for univariate data whilst for multivariate data its development and implementation have been relatively limited. The performance of kernel density estimators depends crucially on the bandwidth selection. Bandwidth selection in the univariate case involves selecting a scalar parameter which controls the amount of smoothing. In the multivariate case, the bandwidth matrix controls both the degree and direction of smoothing so its selection is more difficult. So far most of the research effort has been expended on automatic, data-driven selectors for univariate data. There is, on the other hand, a relative paucity of multivariate counterparts. Most of these multivariate bandwidth selectors are focused on the restricted case of diagonal matrices. In this thesis practical algorithms are constructed, with supporting theoretical justifications, for unconstrained bandwidth matrices. The two main classes of univariate bandwidth selectors are plug-in and cross validation. These unidimensional selectors are generalised to the multidimensional case. The univariate framework for theoretically analysing kernel density estimators is extended to a general multivariate version. This framework has at its core the quantification of the relative rates of convergence which provide a guide to the asymptotic behaviour of bandwidth selectors. Simulation studies and real data analysis are employed to illustrate their finite sample behaviour. It is found that unconstrained selectors possess good asymptotic and finite sample properties in a wide range of situations. Buoyed by this success, two extensions are embarked upon. The first is variable bandwidth selection, generalising the above case where the bandwidth is fixed throughout the sample space. The variation of the bandwidths is controlled by the local properties of the data. The novel contribution is to use non-parametric clustering to summarise these local properties, along with unconstrained bandwidth matrices. The second is in kernel discriminant analysis where unconstrained bandwidth matrices are shown to produce more accurate discrimination.

Context Despite the diversity of available home range estimators, no single method performs equally well in all circumstances. It is therefore important to understand how different estimators perform for data collected under diverse conditions. Kernel density estimation is a popular approach for home range estimation. While many studies have evaluated different kernel bandwidth selectors, few studies have compared different formulations of the bandwidth matrix using wildlife telemetry data. Additionally, few studies have compared the performance of kernel bandwidth selectors using VHF radio-telemetry data from small-bodied taxa. Aims In this study, we used eight different combinations of bandwidth selectors and matrices to evaluate their ability to meet several criteria that could be potentially used to select a home range estimator. Methods We used handheld VHF telemetry data from two species of snake displaying non-migratory and migratory movement patterns. We used subsampling to estimate each estimator’s sensitivity to sampling duration and fix rate and compared home range size, the number of disjunct volume contours and the proportion of telemetry fixes not included in those contours among estimators. Key Results We found marked differences among bandwidth selectors with regards to our criteria but comparatively little difference among bandwidth matrices for a given bandwidth selector. Least-squares cross-validation bandwidths exhibited near-universal convergence failure whereas likelihood cross-validation bandwidths showed high sensitivity to sampling duration and fix rate. The reference, plug-in and smoothed cross-validation bandwidths were more robust to variation in sampling intensity, with the former consistently producing the largest estimates of home range size. Conclusions Our study illustrates the performance of multiple kernel bandwidth estimators for estimating home ranges with datasets typical of many small-bodied taxa. The reference and plug-in bandwidths with an unconstrained bandwidth matrix generally had the best performance. However, our study concurs with earlier studies indicating that no single home range estimator performs equally well in all circumstances. Implications Although we did not find strong differences between bandwidth matrices, we encourage the use of unconstrained matrices because of their greater flexibility in smoothing data not parallel to the coordinate axes. We also encourage researchers to select an estimator suited to their study objectives and the life history of their study organism.

K-means is the best known clustering algorithm, because of its usage simplicity, fast speed and efficiency.However, resultant clusters are influenced by the randomly selected initial centroids.Therefore, many techniques have been implemented to solve the mentioned issue. In this paper, a new version of the k-means clustering algorithm named as ImpKmeans shortly (An Improved Version of K-Means Algorithm by Determining Optimum InitialCentroids Based on Multivariate Kernel Density Estimation and Kd-tree) that uses kernel density estimation, to find the optimum initial centroids, is proposed.Kernel density estimation is used, because it is a nonparametric distribution estimation method, that can identify density regions.To understand the efficiency of the ImpKmeans, we compared it with some state-of-the-art algorithms.According to the experimental studies, the proposed algorithm was better than the compared versions of k-means.While ImpKmeans was the most successful algorithm in 46 tests of 60, the second-best algorithm, was the best on 34 tests.Moreover, experimental results indicated that the ImpKmeans is fast, compared to the selected k-means versions.