A Kullback-Leibler Divergence for Bayesian Model Diagnostics.

In this paper, we study optimal model averaging estimators of regression coefficients in a multinomial logit model, which is commonly used in many scientific fields. A Kullback–Leibler (KL) loss-based weight choice criterion is developed to determine averaging weights. Under some regularity conditions, we prove that the resulting model averaging estimators are asymptotically optimal. When the true model is one of the candidate models, the averaged estimators are consistent. Simulation studies suggest the superiority of the proposed method over commonly used model selection criterions, model averaging methods, as well as some other related methods in terms of the KL loss and mean squared forecast error. Finally, the website phishing data is used to illustrate the proposed method.

This study examines an adaptive log-density estimation method with an $$\ell _1$$ -type penalty. The proposed estimator is guaranteed to be a valid density in the sense that it is positive and integrates to one. The smoothness of the estimator is controlled in a data-adaptive way via $$\ell _1$$ penalization. The advantages of the penalized log-density estimator are discussed with an emphasis on wavelet estimators. Theoretical properties of the estimator are studied when the quality of fit is measured by the Kullback–Leibler divergence (relative entropy). A nonasymptotic oracle inequality is obtained assuming a near orthogonality condition on the given dictionary. Based on the oracle inequality, selection consistency and minimax adaptivity are proved under some regularity conditions. The proposed method is implemented with a coordinate descent algorithm. Numerical illustrations based on the periodized Meyer wavelets are performed to demonstrate the finite sample performance of the proposed estimator.

Model averaging has long been used as a powerful approach to reduce the risk of model misspecification. However, there is little work on model averaging methods under generalized semiparametric models. In this article, we study model averaging in generalized partially linear varying‐coefficient models and propose a semiparametric model averaging estimation (SMAE) method for the canonical parameters. Specifically, we take the partially linear varying‐coefficient model as candidate models in which there is only one varying‐coefficient component. The different covariates could enjoy the benefit of matching the optimal degrees of freedom and each candidate model can take into account the confounding effects among predictors as well. Moreover, the weight choice criterion based on the Kullback–Leibler divergence is adopted to determine the weights. We prove that the corresponding model averaging estimator is asymptotically optimal under certain regularity conditions. Finally, some simulation studies and a real data analysis are conducted to compare with existing methods, and the results show that the proposed method has better out‐of‐sample performance.

We describe and develop a close relationship between two problems that have customarily been regarded as distinct: that of maximizing entropy, and that of minimizing worst-case expected loss. Using a formulation grounded in the equilibrium theory of zero-sum games between Decision Maker and Nature, these two problems are shown to be dual to each other, the solution to each providing that to the other. Although Topsøe described this connection for the Shannon entropy over 20 years ago, it does not appear to be widely known even in that important special case. We here generalize this theory to apply to arbitrary decision problems and loss functions. We indicate how an appropriate generalized definition of entropy can be associated with such a problem, and we show that, subject to certain regularity conditions, the above-mentioned duality continues to apply in this extended context. This simultaneously provides a possible rationale for maximizing entropy and a tool for finding robust Bayes acts. We also describe the essential identity between the problem of maximizing entropy and that of minimizing a related discrepancy or divergence between distributions. This leads to an extension, to arbitrary discrepancies, of a well-known minimax theorem for the case of Kullback–Leibler divergence (the “redundancy-capacity theorem” of information theory). For the important case of families of distributions having certain mean values specified, we develop simple sufficient conditions and methods for identifying the desired solutions. We use this theory to introduce a new concept of “generalized exponential family” linked to the specific decision problem under consideration, and we demonstrate that this shares many of the properties of standard exponential families. Finally, we show that the existence of an equilibrium in our game can be rephrased in terms of a “Pythagorean property” of the related divergence, thus generalizing previously announced results for Kullback–Leibler and Bregman divergences.

The Fiction of Full BEKK

Semi-parametric estimation of American option prices

We investigate the impact of Byzantine attacks in distributed detection under binary hypothesis testing. It is assumed that a fraction of the transmitted sensor measurements are compromised by the injected data from a Byzantine attacker, whose purpose is to confuse the decision maker at the fusion center. From the perspective of a Byzantine attacker, under the injection energy constraint, an optimization problem is formulated to maximize the asymptotic missed detection error probability, which is based on the Kullback-Leibler divergence. The properties of the optimal attack strategy are analyzed by convex optimization and parametric optimization methods. Based on the derived theoretic results, a coordinate descent algorithm is proposed to search the optimal attack solution. Simulation examples are provided to illustrate the effectiveness of the obtained attack strategy.

This work investigates the attack strategy design problem of the false data injection attack against the cyber-physical system. Distinct from the relevant results which utilise the current intercepted data or additionally consider side information, a more universal attack model is proposed which combines historical data and side information with the current intercepted data to synergistically deteriorate the system estimation performance. In order to quantify the impact resulting from the proposed attack strategy, the optimisation objective is characterised by deriving the error covariance matrix under the attacks, which becomes more intricate since the proposed attack model introduces more decision variables and coupled terms. Take the stealthiness which is characterised by Kullback-Leibler divergence as the constraints, the problem investigated in this work is equivalently transformed into the constrained multi-variable non-convex optimisation problems, which are not able to be solved directly by the methods in the relevant results. By utilising the Lagrange multiplier method, the structural characteristic of the optimal mean and the optimal covariance which only related to the Lagrange multiplier are derived, such that the optimal distribution of the modified innovation is able to be obtained by a simple search procedure. Following that, the design of the optimal attack strategy is completed by using semi-definite programming to derive the optimal attack matrices. Finally, the simulation examples are given such that the validity of the results is verified.

Decentralized detection is studied for parallel-access sensor networks, where sensor statistics are not known completely and are assumed to follow distribution functions which belong to known uncertainty classes. It is shown that there exist no minimax robust tests over the deterministic decision rules for the uncertainty classes built with respect to the Kullback-Leibler (KL)-divergence. For the KL-divergence as well as for some other uncertainty classes, such as the $\alpha $ -divergences, the joint stochastic boundedness property, which is the fundamental rule to prove minimax robustness, fails to hold. This raises a natural question whether a solution to minimax robust decentralized detection problem can be given if the uncertainty classes do not own this property. An answer to this question has been shown to be positive, which leads to a generalization of an existing work. Moreover, it is shown that for Huber’s extended uncertainty classes quantization functions at the sensors are not required to be monotone in order to claim minimax robustness. A possible generalization of the theory to minimax- and Neyman-Pearson formulations, repeated observations, imperfect reporting channels and different network topologies have been discussed. Simulation examples are provided considering clipped- and censored likelihood ratio tests.

Optimum designs for parameter estimation in generalized regression models are usually based on the Fisher information matrix (cf. Atkinson et al. (J Stat Plan Inference 144:81–91, 2014) for a recent exposition). The corresponding optimality criteria are related to the asymptotic properties of maximum likelihood (ML) estimators in such models. However, in finite sample experiments there can be problems with identifiability, stability and uniqueness of the ML estimate, which are not reflected by information matrices. In Pazman and Pronzato (Ann Stat 42:1426–1451, 2014) and in Chap. 7 of Pronzato and Pazman (Design of Experiments in Nonlinear Models. Asymptotic Normality, Optimality Criteria and Small-Sample Properties. Springer, New York, 2013) is discussed how to solve some of these estimability issues at the design stage of an experiment in standard nonlinear regression. Here we want to extend this design methodology to more general models based on exponential families of distributions (binomial, Poisson, normal with parametrized variances, etc.). The main tool is the information (or Kullback-Leibler) divergence, which is closely related to ML estimation.

The purpose of the paper is to explore the relative biases in the estimation of the Full BEKK model as compared with the Diagonal BEKK model, which is used as a theoretical and empirical benchmark. Chang and McAleer show that univariate GARCH is not a special case of multivariate GARCH, specifically, the Full BEKK model, and demonstrate that Full BEKK which, in practice, is estimated almost exclusively, has no underlying stochastic process, regularity conditions, or asymptotic properties. Diagonal BEKK (DBEKK) does not suffer from these limitations, and hence provides a suitable benchmark. We use simulated financial returns series to contrast estimates of the conditional variances and covariances from DBEKK and BEKK. The results of non-parametric tests suggest evidence of considerable bias in the Full BEKK estimates. The results of quantile regression analysis show there is a systematic relationship between the two sets of estimates as we move across the quantiles. Estimates of conditional variances from Full BEKK, relative to those from DBEKK, are lower in the left tail and higher in the right tail.

In this paper we provide a method to test the existence of the change points in the nonparametric regression function of partially linear models with conditional heteroscedastic variance. We propose the test statistic and establish its asymptotic properties under some regular conditions. Some simulation studies are given to investigate the performance of the proposed method in finite samples. Finally, the proposed method is applied to a real data for illustration.

7 - Methods Based on Likelihood and Their Asymptotic properties

Additive hazards model with random effects is proposed for modelling the correlated failure time data when focus is on comparing the failure times within clusters and on estimating the correlation between failure times from the same cluster, as well as the marginal regression parameters. Our model features that, when marginalized over the random effect variable, it still enjoys the structure of the additive hazards model. We develop the estimating equations for inferring the regression parameters. The proposed estimators are shown to be consistent and asymptotically normal under appropriate regularity conditions. Furthermore, the estimator of the baseline hazards function is proposed and its asymptotic properties are also established. We propose a class of diagnostic methods to assess the overall fitting adequacy of the additive hazards model with random effects. We conduct simulation studies to evaluate the finite sample behaviors of the proposed estimators in various scenarios. Analysis of the Diabetic Retinopathy Study is provided as an illustration for the proposed method.

In the case of the random design nonparametric regression, locally weighted regression (LWR) in Cleveland is considered. Under some regularity conditions, we show that the regression function estimator derived from LWR has the same asymptotic variance quantity as the k-nearest neighbor (k-NN) smoother. But, it has better asymptotic bias quality than the k-NN smoother. By these results, this regression function estimator has two advantages. One is that it is better than the k-NN smoother in a minimax sense. The other is that it does not suffer from boundary effects when boundary points of the support of the design density are known in advance. The motivation that the regression function estimator derived from LWR has nice asymptotic bias quality and boundary properties comes from Fan and Gijbels and Fan Furthermore, estimators of derivatives of the regression function derived from LWR are considered.