Adaptive log-density estimation

Penalized logspline density estimation using total variation penalty

We investigate the impact of Byzantine attacks in distributed detection under binary hypothesis testing. It is assumed that a fraction of the transmitted sensor measurements are compromised by the injected data from a Byzantine attacker, whose purpose is to confuse the decision maker at the fusion center. From the perspective of a Byzantine attacker, under the injection energy constraint, an optimization problem is formulated to maximize the asymptotic missed detection error probability, which is based on the Kullback-Leibler divergence. The properties of the optimal attack strategy are analyzed by convex optimization and parametric optimization methods. Based on the derived theoretic results, a coordinate descent algorithm is proposed to search the optimal attack solution. Simulation examples are provided to illustrate the effectiveness of the obtained attack strategy.

Outlier detection is imperative in biomedical data analysis to achieve reliable knowledge discovery. In this paper, a new outlier detection method based on Kullback-Leibler (KL) divergence is presented. The original concept of KL divergence was designed as a measure of distance between two distributions. Stemming from that, we extend it to biological sample outlier detection by forming sample sets composed of nearest neighbors. To handle the non-linearity during the KL divergence calculation and to tackle with the singularity problem due to small sample size, we map the original data into a higher feature space and apply kernel functions without resorting to a mapping function. A sample possessing the largest KL divergence is detected as an outlier. The proposed method is tested with one synthetic data, two public gene expression data sets, and our own mass spectrometry data generated for prostate cancer study.

This paper considers a Kullback-Leibler distance (KLD) which is asymptotically equivalent to the KLD by Goutis and Robert [1] when the reference model (in comparison to a competing fitted model) is correctly specified and that certain regularity conditions hold true (ref. Akaike [2]). We derive the asymptotic property of this Goutis-Robert-Akaike KLD under certain regularity conditions. We also examine the impact of this asymptotic property when the regularity conditions are partially satisfied. Furthermore, the connection between the Goutis-Robert-Akaike KLD and a weighted posterior predictive p-value (WPPP) is established. Finally, both the Goutis-Robert-Akaike KLD and WPPP are applied to compare models using various simulated examples as well as two cohort studies of diabetes.

The entropy rates of Markov chains (random walks) defined on connected undirected graphs are well studied in many surveys. We study the entropy rates related to the first-passage time probability distributions of fair random walks, their relative (Kullback–Leibler) entropies, and the entropy related to two biased random walks – with the random absorption of walkers and the shortest paths random walks. We show that uncertainty of first-passage times quantified by the entropy rates characterizes the connectedness of the graph. The relative entropy derived for the biased random walks estimates the level of uncertainty between connectivity and connectedness – the local and global properties of nodes in the graph.

The main issue in this work is to study the limit functions necessary for the reliability assessment of structural steel with the use of the relative entropy apparatus. This will be done using a few different mathematical theories relevant to this relative entropy, namely those proposed by Bhattacharyya, Kullback–Leibler, Jeffreys, and Hellinger. Probabilistic analysis in the presence of uncertainty in material characteristics will be delivered using three different numerical strategies—Monte Carlo simulation, the stochastic perturbation method, as well as the semi-analytical approach. All of these methods are based on the weighted least squares method approximations of the structural response functions versus the given uncertainty source, and they allow efficient determination of the first two probabilistic moments of the structural responses including stresses, displacements, and strains. The entire computational implementation will be delivered using the finite element method system ABAQUS and computer algebra program MAPLE, where relative entropies, as well as polynomial response functions, will be determined. This study demonstrates that the relative entropies may be efficiently used in reliability assessment close to the widely engaged first-order reliability method (FORM). The relative entropy concept enables us to study the probabilistic distance of any two distributions, so that structural resistance and extreme effort in elastoplastic behavior need not be restricted to Gaussian distributions.

For high-dimensional models with a focus on classification performance, the l1-penalized logistic regression is becoming important and popular. However, the Lasso estimates could be problematic when penalties of different coefficients are all the same and not related to the data. We propose two types of weighted Lasso estimates, depending upon covariates determined by the McDiarmid inequality. Given sample size n and a dimension of covariates p, the finite sample behavior of our proposed method with a diverging number of predictors is illustrated by non-asymptotic oracle inequalities such as the l1-estimation error and the squared prediction error of the unknown parameters. We compare the performance of our method with that of former weighted estimates on simulated data, then apply it to do real data analysis.

In this paper, we study optimal model averaging estimators of regression coefficients in a multinomial logit model, which is commonly used in many scientific fields. A Kullback–Leibler (KL) loss-based weight choice criterion is developed to determine averaging weights. Under some regularity conditions, we prove that the resulting model averaging estimators are asymptotically optimal. When the true model is one of the candidate models, the averaged estimators are consistent. Simulation studies suggest the superiority of the proposed method over commonly used model selection criterions, model averaging methods, as well as some other related methods in terms of the KL loss and mean squared forecast error. Finally, the website phishing data is used to illustrate the proposed method.

Model averaging has long been used as a powerful approach to reduce the risk of model misspecification. However, there is little work on model averaging methods under generalized semiparametric models. In this article, we study model averaging in generalized partially linear varying‐coefficient models and propose a semiparametric model averaging estimation (SMAE) method for the canonical parameters. Specifically, we take the partially linear varying‐coefficient model as candidate models in which there is only one varying‐coefficient component. The different covariates could enjoy the benefit of matching the optimal degrees of freedom and each candidate model can take into account the confounding effects among predictors as well. Moreover, the weight choice criterion based on the Kullback–Leibler divergence is adopted to determine the weights. We prove that the corresponding model averaging estimator is asymptotically optimal under certain regularity conditions. Finally, some simulation studies and a real data analysis are conducted to compare with existing methods, and the results show that the proposed method has better out‐of‐sample performance.

Distributed detection over a blockchain-aided Internet of Things (BIoT) network in the presence of attacks is considered, where the integrated blockchain is employed to secure data exchanges over the BIoT as well as data storage at the agents of the BIoT. We consider a general adversary model where attackers jointly exploit the vulnerability of IoT devices and that of the blockchain employed in the BIoT. The optimal attacking strategy which minimizes the Kullback-Leibler divergence is pursued. It can be shown that this optimization problem is nonconvex, and hence it is generally intractable to find the globally optimal solution to such a problem. To overcome this issue, we first propose a relaxation method that can convert the original nonconvex optimization problem into a convex optimization problem, and then the analytic expression for the optimal solution to the relaxed convex optimization problem is derived. The optimal value of the relaxed convex optimization problem provides a detection performance guarantee for the BIoT in the presence of attacks. In addition, we develop a coordinate descent algorithm which is based on a capped water-filling method to solve the relaxed convex optimization problem, and moreover, we show that the convergence of the proposed coordinate descent algorithm can be guaranteed.

Neuroscientists have recently shown that images that are difficult to find in visual search elicit similar patterns of firing across a population of recorded neurons. The $L^{1}$ distance between firing rate vectors associated with two images was strongly correlated with the inverse of decision time in behavior. But why should decision times be correlated with $L^{1}$ distance? What is the decision-theoretic basis? In our decision theoretic formulation, we model visual search as an active sequential hypothesis testing problem with switching costs. Our analysis suggests an appropriate neuronal dissimilarity index, which correlates equally strongly with the inverse of decision time as the $L^{1}$ distance. We also consider a number of other possibilities, such as the relative entropy (Kullback–Leibler divergence) and the Chernoff entropy of the firing rate distributions. A more stringent test of equality of means, which would have provided a strong backing for our modeling, fails for our proposed as well as the other already discussed dissimilarity indices. However, test statistics from the equality of means test, when used to rank the indices in terms of their ability to explain the observed results, places our proposed dissimilarity index at the top followed by relative entropy, Chernoff entropy, and the $L^{1}$ indices. Computations of the different indices require an estimate of the relative entropy between two Poisson point processes. An estimator is developed and is shown to have near unbiased performance for almost all operating regions.

We prove characterization theorems for relative entropy (also known as Kullback–Leibler divergence), q-logarithmic entropy (also known as Tsallis entropy), and q-logarithmic relative entropy. All three have been characterized axiomatically before, but we show that earlier proofs can be simplified considerably, at the same time relaxing some of the hypotheses.

Embedding a large vocabulary speech recognition system in mobile devices requires a reduced acoustic model obtained by eliminating redundant model parameters. In conventional optimization methods based on the minimum description length (MDL) criterion, a binary Gaussian tree is built at each state of a hidden Markov model by iteratively finding and merging similar mixture components. An optimal subset of the tree nodes is then selected to generate a downsized acoustic model. To obtain a better binary Gaussian tree by improving the process of finding the most similar Gaussian components, this paper proposes a new distance measure that exploits the difference in likelihood values for cases before and after two components are combined. The mixture weight of Gaussian components is also introduced in the component merging step. Experimental results show that the proposed method outperforms MDL-based optimization using either a Kullback-Leibler (KL) divergence or weighted KL divergence measure. The proposed method could also reduce the acoustic model size by 50% with less than a 1.5% increase in error rate compared to a baseline system.

A set of Wi-Fi RSSI (Received Signal Strength Indicator) measurements is one of basic sensory observation available for indoor localization. One major drawback of the RSSI based localization is maintenance of the RSSI fingerprint database, which should be periodically updated against measurement pattern changes caused by relocation, removal and malfunction of Wi-Fi APs (access points). To address this problem, a new change detection method is proposed in this paper. First, by machine learning techniques, the RSSI database is reconstructed to a probabilistic feature database by the implementations of PCA (Principal Component Analysis) and GP (Gaussian Process). Then, KL (Kullback-Leibler) divergence is used as a metric to measure the similarity of the existing database and a newly arrived test sets. The proposed method is evaluated by a real experiment at a multi-storey building. For experimental study, different cases that provoke changes of RSSI patterns are considered, and the positioning accuracy is examined by the k-NN (Nearest Neighbor) method. From the experimental results, it is found that the bigger the RSSI pattern changes, the large the KL divergences become. Also, when a modified change detection algorithm as the benchmark, which does not implement the PCA feature extraction, is compared, the proposed algorithm yields accurate and fast computing performances. In addition, the required number of survey points is empirically found associated with the threshold value to trigger the detection alarm.

This paper considers nonparametric estimations of a density function in a two-class mixture model. A linear wavelet estimator and an adaptive wavelet estimator are constructed. Upper bound estimations over $ L^{p}\; (1\leq p < +\infty) $ risk of those wavelet estimators are proved in Besov spaces. When $ \tilde{p}\geq p\geq1 $, the convergence rate of adaptive wavelet estimator is the same as the linear estimator up to a $ \ln n $ factor. The adaptive wavelet estimator can get better than the linear estimator in the case of $ 1\leq \tilde{p} < p $. Finally, some numerical experiments are presented to validate the theoretical results.