Improving importance estimation in covariate shift for providing accurate prediction error

Research on output distribution modeling of photovoltaic modules based on kernel density estimation method and its application in anomaly identification

Kernel density estimation (KDE) is one of the most widely used nonparametric density estimation methods. The fact that it is a memory-based method, i.e., it uses the entire training data set for prediction, makes it unsuitable for most current big data applications. Several strategies, such as tree-based or hashing-based estimators, have been proposed to improve the efficiency of the kernel density estimation method. The novel density kernel density estimation method (DMKDE) uses density matrices, a quantum mechanical formalism, and random Fourier features, an explicit kernel approximation, to produce density estimates. This method has its roots in the KDE and can be considered as an approximation method, without its memory-based restriction. In this paper, we systematically evaluate the novel DMKDE algorithm and compare it with other state-of-the-art fast procedures for approximating the kernel density estimation method on different synthetic data sets. Our experimental results show that DMKDE is on par with its competitors for computing density estimates and advantages are shown when performed on high-dimensional data. We have made all the code available as an open source software repository.

This paper presents a new method for predictive crime hotspot analysis that further improves the kernel density estimation (KDE) method and the spatio-temporal kernel density estimation (STKDE) method by accounting for temporal crime cycles and is therefore termed the ‘cyclically adjusted STKDE (cSTKDE) method’. The case study on robbery incidents in Baton Rouge, Louisiana, shows a temporal cycle with a 6-month period of statistical significance from January 2010 to May 2018. This identified period is incorporated into the temporal kernel function of the new cSTKDE method. For validation, the Forecast Accuracy Index (FAI) and Forecast Precision Index (FPI) are used to evaluate the performance across 52 weeks in 2013. For 11 consecutive weeks since the beginning of 2013, the cSTKDE method outperforms the STKDE by 89% lower average abs(1-FAI) and 17% higher average FPI, and outperforms the KDE by 90% lower average abs(1-FAI) and 8% higher average FPI. Overall, the scenario with the best predictive accuracy by the cSTKDE is recommended over the traditional KDE or STKDE method as most feasible and effective in implementation of hotspot policing in practice.

Abstract: Home‐range estimators are commonly tested with simulated animal locational data in the laboratory before the estimators are used in practice. Although kernel density estimation (KDE) has performed well as a home‐range estimator for simulated data, several recent studies have reported its poor performance when used with data collected in the field. This difference may be because KDE and other home‐range estimators are generally tested with simulated point locations that follow known statistical distributions, such as bivariate normal mixtures, which may not represent well the space‐use patterns of all wildlife species. We used simulated animal locational data of 5 point pattern shapes that represent a range of wildlife utilization distributions to test 4 methods of home‐range estimation: 1) KDE with reference bandwidths, 2) KDE with least‐squares cross‐validation, 3) KDE with plug‐in bandwidths, and 4) minimum convex polygon (MCP). For the point patterns we simulated, MCP tended to produce more accurate area estimates than KDE methods. However, MCP estimates were markedly unstable, with bias varying widely with both sample size and point pattern shape. The KDE methods performed best for concave distributions, which are similar to bivariate normal mixtures, but still overestimated home ranges by about 40–50% even in the best cases. For convex, linear, perforated, and disjoint point patterns, KDE methods overestimated home‐range sizes by 50–300%, depending on sample size and method of bandwidth selection. These results indicate that KDE does not produce home‐range estimates that are as accurate as the literature suggests, and we recommend exploring other techniques of home‐range estimation.

Geological hazards caused by strong earthquakes have caused continuous social and economic losses and destruction of the ecological environment in the hazard area, and are mostly manifested in the areas with frequent occurrence of geological hazards or the clustering of geological hazards. Considering the long-term nature of earthquakes and geological disasters in this region, this paper takes ten earthquake-stricken areas in Wenchuan earthquake zone as examples to collect shallow landslide data in 2010, combined with the spatial location of landslides and other factors. Kernel density estimation (KDE) method is used to analyze the spatial characteristics of shallow landslide. Taking the space of shallow landslide as the characteristic variable and fully considering the regulating factors of earthquake-induced landslide: terrain complexity, distance to river, distance to fault, distance to road, lithology, normalized vegetation difference index (NDVI) and ground peak acceleration (PGA) as independent variables, based on KDE and polynomial logistic regression (MLR), A quantitative model of shallow landslide in the earthquake area is constructed. The results show that: (1) PGA has the greatest impact on landslide in the study area. (2) Compared with the two-category logistic regression (two-category LR) model, the susceptibility map of landslide prediction results based on the KDE-MLR landslide susceptibility prediction model is more consistent with the actual situation. (3) The prediction accuracy of the model validation set is 70.7%, indicating that the landslide susceptibility prediction model based on KDE-MLR can effectively highlight the spatial characteristics of shallow landslides in 10 extreme disaster areas. The research results can provide decision-making basis for shallow landslide warning and post-disaster reconstruction in earthquake-stricken areas.

In social science researches, there may be cases where a category of the dependent variable is seen hundred times less (more) than the other category. Events like wars, mass migrations or coups in social sciences; an event of interest in binary variable(s) may have very low prevalence, resulting in low or even zero cell counts in one or two cells in the 2X2 tables of two factors. In this case, independent variable predict the dependent variable perfectly or almost perfectly, and this leads to an issue called complete or quasi-complete separation problem in statistical modelling. This study aims to compare three methods suggested in the literature for the quasi-complete separation in a real small dataset; penalized maximum likelihood (Firth-type), exact logistic regression and bayesian logistic regression. Methods were compared via odds ratios, odds’ standard error estimates, confidence intervals and statistical significance. Parameter estimates were obtained under three different models with binary and continuous variables. Results show that all methods can provide convergence in the presence of quasi-complete separation. In conclusion, bayesian logistic regression estimates tend to be superior than the other methods in terms of estimation of standard errors.

The bootstrap: Some large sample theory and connections with robustness

Combined probability density model for medium term load forecasting based on quantile regression and kernel density estimation

Radio source counts provide a fundamental census of cosmic radio emission, yet their estimation is usually based on coarse histograms that suffer from bin-choice bias, boundary effects, and survey incompleteness. We apply and rigorously evaluate kernel density estimation (KDE) as a nonparametric alternative to the conventional binned method for estimating differential radio source counts. Using simulated flux-limited samples derived from an input luminosity function model, we compare the performance of standard KDE, adaptive KDE, and traditional binning methods. Our results show that KDE-based approaches yield more accurate and stable estimates, particularly in the high-flux regime where data are sparse and conventional methods struggle. We also apply the adaptive KDE method to real observational data from the LOFAR Two-Metre Sky Survey Deep Fields. Our analysis robustly confirms the pronounced “drop and bump” feature at sub-mJy flux densities, but also reveals that a secondary, modest bump seen in the binned data at ∼10 mJy is likely a binning artifact. We also demonstrate the flexibility of KDE in addressing observational incompleteness through weighted estimation, which applies weights continuously at the level of individual sources rather than averaging them in discrete bins. These strengths make KDE a powerful tool for source-count analyses in current and future radio surveys and, more broadly, in analogous studies at other wavelengths. All computations in this study are implemented with AstroKDE , a Python package we have developed for astronomical applications.

In addition to its vital significance in combustion and atmospheric chemistry, the reaction between H' and HO2 on the ground triplet state represents a prototype with multiple product channels, including H2 + O2, OH + OH, O + H2O, and H + H'O2. In this work, a full-dimensional accurate potential energy surface (PES) for the title reaction was developed to provide reliable descriptions for all dynamically relevant regions. Using this PES, we adopted the quasi-classical trajectory approach to study the corresponding reaction dynamics, including the reactivity of each product channel and the associated product branching ratio, the product energy distributions, product angular distributions, and associated microscopic mechanisms. For representing distributions of the product energies, such as product translational energy as well as product rotational and vibrational energies, both the traditional histogram and the kernel density estimation (KDE) methods were used and compared. It seems that the features of the resulting distributions in this work are very similar to each other among different methods. The KDE method is suggested for statistics, particularly for those populations with small oscillations in the histogram plot.

Sampling timestamps enhance the reliability of spatiotemporal trajectory density estimation. Existing kernel density estimation (KDE) methods leverage sampling time to modify kernel shapes or weight specifications. However, controlling two concurrent errors remains challenging: density over/underestimation due to temporal autocorrelation, and nonzero density assignments to unreachable locations. This study proposes a time-aware KDE (t-KDE) method for irregular sampling intervals, adopting a divide-and-conquer strategy that synergistically integrates time geography and temporal autocorrelation. By establishing two temporal mappings—from time to kernel functions and to weight coefficients—t-KDE retrospectively mitigates the above errors through KDE recalibration. The approach aims to maximize temporal information utilization to reduce estimation uncertainty, providing a theoretical basis for unbiased density modeling. Empirical analysis demonstrates that t-KDE outperforms state-of-the-art methods in accuracy and reliability.

PurposeIn a recent paper, Yoon and Lee (2019) (YL hereafter) propose a weighted Fama and MacBeth (FMB hereafter) two-step panel regression procedure and provide evidence that their weighted FMB procedure produces more efficient coefficient estimators than the usual unweighted FMB procedure. The purpose of this study is to supplement and improve their weighted FMB procedure, as they provide neither asymptotic results (i.e. consistency and asymptotic distribution) nor evidence on how close their standard error estimator is to the true standard error.Design/methodology/approachFirst, asymptotic results for the weighted FMB coefficient estimator are provided. Second, a finite-sample-adjusted standard error estimator is provided. Finally, the performance of the adjusted standard error estimator compared to the true standard error is assessed.FindingsIt is found that the standard error estimator proposed by Yoon and Lee (2019) is asymptotically consistent, although the finite-sample-adjusted standard error estimator proposed in this study works better and helps to reduce bias. The findings of Yoon and Lee (2019) are confirmed even when the average R2 over time is very small with about 1% or 0.1%.Originality/valueThe findings of this study strongly suggest that the weighted FMB regression procedure, in particular the finite-sample-adjusted procedure proposed here, is a computationally simple but more powerful alternative to the usual unweighted FMB procedure. In addition, to the best of the authors’ knowledge, this is the first study that presents a formal proof of the asymptotic distribution for the FMB coefficient estimator.

When estimating the mean value of a variable, or the total amount of a resource, within a specified region it is desirable to report an estimated standard error for the resulting estimate. If the sample sites are selected according to a probability sampling design, it usually is possible to construct an appropriate design-based standard error estimate. One exception is systematic sampling for which no such standard error estimator exists. However, a slight modification of systematic sampling, termed 2-step tessellation stratified (2TS) sampling, does permit the estimation of design-based standard errors. This paper develops a design-based standard error estimator for 2TS sampling. It is shown that the Taylor series approximation to the variance of the sample mean under 2TS sampling may be expressed in terms of either a deterministic variogram or a deterministic covariance function. Variance estimation then can be approached through the estimation of a variogram or a covariance function. The resulting standard error estimators are compared to some more traditional variance estimators through a simulation study. The simulation results show that estimators based on the new approach may perform better than traditional variance estimators.

K-nearest neighbors and a kernel density estimator for GEFCom2014 probabilistic wind power forecasting

Identification of high-risk regions of schistosomiasis is important for rational resource allocation and effective control strategies. We conducted the first study to apply the newly developed method of adaptive kernel density estimation (KDE)-based spatial relative risk function (sRRF) to detect the high-risk regions of schistosomiasis in the Guichi region of China and compared it with the fixed KDE-based sRRF. We found that the adaptive KDE-based sRRF had a better ability to depict the heterogeneity of risk regions, but was more sensitive to altering the user-defined smoothing parameters. Specifically, the impact of bandwidths on the estimated risk value and risk significance (P value) was higher for the adaptive KDE-based sRRF, but lower on the estimated risk variation standard error (s.e.) compared with the fixed KDE-based sRRF. Based on this application the adaptive and fixed KDE-based sRRF have their respective advantages and disadvantages and the joint application of the two approaches can warrant the best possible identification of high-risk subregions of diseases.