A Comparative Study of Bandwidth Choice in Kernel Density Estimation for Naive Bayesian Classification

Opinion mining has become a major part in today's economy. People would want to know more about a product and the customers opinion before buying it. Companies would also want to know the opinions of the customers. Therefore, analyzing the customer’s opinion is important. A new customer would consider a product as good by analyzing the opinions of other customers. The opinions are collected from various areas, which include blogs, web forums, and product review sites. Classifying these large set of opinions requires a good classifier. In view of this, a comparative study of three classification techniques - Naive Bayes classifier with Kernel Density Estimation (KDE), Support Vector Machine (SVM), Decision Tree and KNN was made. To evaluate the classifier accuracy, precision, recall and F-measure techniques are used. Experimental results show that the Naive Bayes with Kernel Density Estimation (KDE) classifier achieved higher accuracy among others.

Mining data over streams in one pass and using constant memory is a challenging task. Decision trees are one of the most popular classifiers for both batch and incremental learning due to their high degree of interpretability, ease of construction and good accuracy. The most popular decision tree for stream classification is Hoeffding Tree based on Hoeffding bound. Literature shows a few variants of decision trees based on different bounds. The default class prediction method adopted in decision tree is “majority class” approach. Later, the accuracy of prediction was scaled up by a hybrid decision tree where Naive Bayes classifier was used for prediction. Kernel Density Estimation (KDE) is employed in Flexible Naive Bayes for classification. However, it is suitable for modeling static data set. This paper proposes an Incremental Flexible Naive Bayes (IFNB) based hybrid decision tree paradigm that uses KDE to model continuous attributes at leaf nodes of the tree for improving the class prediction accuracy. Experimental results on both synthetic and real dataset show that the proposed IFNB based leaf classifiers achieves improvement over the class prediction methods adopted in existing decision trees for data streams.

Abstract: Home‐range estimators are commonly tested with simulated animal locational data in the laboratory before the estimators are used in practice. Although kernel density estimation (KDE) has performed well as a home‐range estimator for simulated data, several recent studies have reported its poor performance when used with data collected in the field. This difference may be because KDE and other home‐range estimators are generally tested with simulated point locations that follow known statistical distributions, such as bivariate normal mixtures, which may not represent well the space‐use patterns of all wildlife species. We used simulated animal locational data of 5 point pattern shapes that represent a range of wildlife utilization distributions to test 4 methods of home‐range estimation: 1) KDE with reference bandwidths, 2) KDE with least‐squares cross‐validation, 3) KDE with plug‐in bandwidths, and 4) minimum convex polygon (MCP). For the point patterns we simulated, MCP tended to produce more accurate area estimates than KDE methods. However, MCP estimates were markedly unstable, with bias varying widely with both sample size and point pattern shape. The KDE methods performed best for concave distributions, which are similar to bivariate normal mixtures, but still overestimated home ranges by about 40–50% even in the best cases. For convex, linear, perforated, and disjoint point patterns, KDE methods overestimated home‐range sizes by 50–300%, depending on sample size and method of bandwidth selection. These results indicate that KDE does not produce home‐range estimates that are as accurate as the literature suggests, and we recommend exploring other techniques of home‐range estimation.

A density-based oversampling approach for class imbalance and data overlap

Multivariate online kernel density estimation with Gaussian kernels

The identification of malicious jamming pattern is significant for wireless communication system to adopt targeted anti-jamming approaches. Efficient identification of malicious jamming patterns requires sufficient sample data and computing resources for the complex electromagnetic environment. However, with the use of portable communication equipment and emergency communications, it is difficult to ensure adequate sample data and computing resources. Against this background, this letter focuses on the jamming identification based on a small sample data-driven Naive Bayes classifier. The main idea is to obtain approximate conditional feature probability via data augment and kernel density estimation (KDE) from a small sample labelled data and construct Naive Bayes classifier, so as to form a fast jamming recognition method. Simulation results show that compared with the methods using decision tree or simple neutral network, the proposed scheme achieves a better average accuracy between six jamming patterns, as well as low-complexity.

Naïve Bayes (NB) Classifier is considered one of the most widely used algorithms to solve classification problems due to its simplicity, high performance and fast computational process, which assumes that feature variables are independent and continuous variables follow gaussian distribution. Due to that most real-life applications may have feature variables that can’t be modeled by gaussian distribution, or may violate the independence assumption, Flexible Naïve Bayes (FNB) Classifier becomes more suitable, which estimates the densities using a non-parametric Kernel Density Estimator (KDE). In this paper, we introduce K-Nearest Neighbor Naïve Bayes (KNNB) Classifier, an improved flexible classifier which uses K-Nearest Neighbor Density Estimator (KNNDE) instead of KDE, such classifier would solve both NB problems, where the independence assumption is removed and the likelihood density is estimated for all feature variables together. Practical applications using R programming language on datasets from different fields were made for both FNB and KNNB classifiers with and without the independence assumption using 10-folds repeated cross validation, despite that FNB achieved greater accuracy than KNNB when dealing with the feature variables separately, the results showed that KNNB classifier performs better than FNB when the independence assumption is removed, meaning that KNNB is more efficient and suitable than FNB in real-life applications, where the independence assumption can be dropped.

In practical classification problems, imbalanced class distribution is very common. Traditional classification methods usually aim at optimizing the overall accuracy, which leads to classification decisions biased towards categories with a large number of samples. Similar to most traditional classifiers, Naive Bayes (NB) is sensitive to the class prior distribution and has poor recognition performance for minority class samples, which restricts its application in imbalanced classification. Therefore, we propose a Bayesian classifier based on distance-corrected multivariate kernel density estimation (BC-DMKDE). BC-DMKDE relaxes the assumptions regarding conditional independence and the normality of continuous variables in NB classifiers, by incorporating the multivariate kernel density estimation (MKDE). We also incorporate a distance-correction mechanism utilizing kernel density estimation, specifically to alleviate the impact of class imbalance and overlapping data samples. Across 18 real-world benchmarks, BC-DMKDE exhibits superior classification performance and robustness, outperforming existing methods at both the algorithm and data levels.

This paper proposes a wind power probabilistic model (WPPM) using the reflection method and multi-kernel function kernel density estimation (KDE). With the increasing penetration of renewable energy sources (RESs) into power systems, several probabilistic approaches have been introduced to assess the impact of RESs on the power system. A probabilistic approach requires a wind power scenario (WPS), and the WPS is generated from the WPPM. Previously, WPPM was generated using a parametric density estimation, and it had limitations in reflecting the characteristics of wind power data (WPD) due to a boundary bias problem. The paper proposes a WPPM generated using the KDE, which is a non-parametric method. Additionally, the paper proposes a reflection method correcting for the boundary bias problem caused by the double-bounded characteristic of the WPD and the multi-kernel function KDE minimizing the effect of tied values. Six bandwidth selectors are used to calculate the bandwidth for the KDE, and one is selected by analyzing the correlation between the normalized WPD and the calculated bandwidth. The results were validated by generating WPPMs with WPDs in six regions of the Republic of Korea, and it was confirmed that the accuracy and goodness-of-fit are improved when the proposed method is used.

Kernel density estimation (KDE) is a widely used method in geography to study concentration of point pattern data. Geographical networks are 1.5 dimensional spaces with specific characteristics, analyzing events occurring on networks (accidents on roads, leakages of pipes, species along rivers, etc.). In the last decade, they required the extension of spatial KDE. Several versions of Network KDE (NKDE) have been proposed, each with their particular advantages and disadvantages, and are now used on a regular basis. However, scant attention has been given to the temporal extension of NKDE (TNKDE). In practice, when the studied events happen at specific time points and are constrained on a network, the methodologies used by geographers tend to overlook either the network or the temporal dimension. Here we propose a TNKDE based on the recent development of NKDE and the product of kernels. We also adapt classical methods of KDE (Diggle's correction, Abramson's adaptive bandwidth and bandwidth selection by leave‐one‐out maximum likelihood). We also illustrate the method with Montreal road crashes involving a pedestrian between 2016 and 2019.

Robust parametric modeling of Alzheimer’s disease progression

Kernel density estimation is an important data smoothing technique. It has been applied most successfully for univariate data whilst for multivariate data its development and implementation have been relatively limited. The performance of kernel density estimators depends crucially on the bandwidth selection. Bandwidth selection in the univariate case involves selecting a scalar parameter which controls the amount of smoothing. In the multivariate case, the bandwidth matrix controls both the degree and direction of smoothing so its selection is more difficult. So far most of the research effort has been expended on automatic, data-driven selectors for univariate data. There is, on the other hand, a relative paucity of multivariate counterparts. Most of these multivariate bandwidth selectors are focused on the restricted case of diagonal matrices. In this thesis practical algorithms are constructed, with supporting theoretical justifications, for unconstrained bandwidth matrices. The two main classes of univariate bandwidth selectors are plug-in and cross validation. These unidimensional selectors are generalised to the multidimensional case. The univariate framework for theoretically analysing kernel density estimators is extended to a general multivariate version. This framework has at its core the quantification of the relative rates of convergence which provide a guide to the asymptotic behaviour of bandwidth selectors. Simulation studies and real data analysis are employed to illustrate their finite sample behaviour. It is found that unconstrained selectors possess good asymptotic and finite sample properties in a wide range of situations. Buoyed by this success, two extensions are embarked upon. The first is variable bandwidth selection, generalising the above case where the bandwidth is fixed throughout the sample space. The variation of the bandwidths is controlled by the local properties of the data. The novel contribution is to use non-parametric clustering to summarise these local properties, along with unconstrained bandwidth matrices. The second is in kernel discriminant analysis where unconstrained bandwidth matrices are shown to produce more accurate discrimination.

Kernel density estimation is a widely used statistical tool and bandwidth selection is critically important. The Sheather and Jones’ (SJ) selector [A reliable data-based bandwidth selection method for kernel density estimation, J. R. Stat. Soc. Ser. B 53 (1991), pp. 683–690] remains the best available data-driven bandwidth selector. It can, however, perform poorly if the true density deviates too much in shape from normal. This paper first develops an alternative selector by following ideas in Park and Marron [On the use of pilot estimators in bandwidth selection, Nonparametr. Stat. 1 (1992), pp. 231–240] to reduce the impact of the normal reference density. The selector can bring drastic improvement to less smooth densities that the SJ selector has difficulty with, but may be slightly worse off otherwise. We then propose to combine the alternative selector and SJ selector by using the smaller of the two bandwidths, which has the effect of automatically picking the better one for a particular density. In our extensive simulation, study using the 15 benchmark densities in Marron and Wand [Exact mean integrated squared error, Ann. Statist. 20 (1992), pp. 712–736], the combined selector significantly improves the SJ selector for 5 difficult densities and retains the superior performance of the SJ selector for the other 10. A ready-to-use R function is provided.

As the proportion of renewable energy sources that are intermittent and variability increases, system uncertainty is increasing, and a probabilistic analysis that can reflect these characteristics is required. In this paper, we propose a method for generating a probablistic wind power generation model using kernel density estimation. The boundary bias problem caused by using kernel density estimation for wind power generation data is eliminated through the boundary correction method, and a bandwidth suitable for the data can be selected through the bandwidth selector. Scenarios were created from probabilistic models made in various ways, and the accuracy of the models was validated by comparing them with real data. As a result, the accuracy was improved when the scenario was created by generating the model directly from the wind power generation data by estimating the kernel density estimation.

In this paper, we present a Markov chain Monte Carlo (MCMC) simulation algorithm for estimating parameters in the kernel density estimation of bivariate insurance claim data via transformations. Our data set consists of two types of auto insurance claim costs and exhibits a high-level of skewness in the marginal empirical distributions. Therefore, the kernel density estimator based on original data does not perform well. However, the density of the original data can be estimated through estimating the density of the transformed data using kernels. It is well known that the performance of a kernel density estimator is mainly determined by the bandwidth, and only in a minor way by the kernel. In the current literature, there have been some developments in the area of estimating densities based on transformed data, where bandwidth selection usually depends on pre-determined transformation parameters. Moreover, in the bivariate situation, the transformation parameters were estimated for each dimension individually. We use a Bayesian sampling algorithm and present a Metropolis-Hastings sampling procedure to sample the bandwidth and transformation parameters from their posterior density. Our contribution is to estimate the bandwidths and transformation parameters simultaneously within a Metropolis-Hastings sampling procedure. Moreover, we demonstrate that the correlation between the two dimensions is better captured through the bivariate density estimator based on transformed data.