Chapter 13 - Empirical Methods

As pointed out in the introduction (chapter 1) there exists a clear analogy between the annealing of solids and the problem of solving large combinatorial optimization problems. The physical annealing process can be successfully modelled by using computer simulation methods from condensed matter physics [BAR76], [WOOD68]. These methods in turn are based on the theory of statistical mechanics which can be viewed as the central discipline of condensed matter physics [BIN78], [TOD83]. Starting off with the first paper on simulated annealing by Kirkpatrick et al. [KIR82], a number of authors elaborate on the relation between combinatorial optimization and statistical mechanics, either because of a phenomenological interest in the analogy (Bernasconi [BER87], Bonomi and Lutton [BON84], Kirkpatrick et al. [KIR82], [KIR83], [KIR85] and Mezard and Parisi [MEZ85], [MEZ86]) or because of a possible framework to model the convergence and the corresponding control of the simulated annealing algorithm (Aarts and Van Laarhoven [AAR85a], [AAR88a], Grest et al. [GRES86], Kirkpatrick et al. [KIR82], [KIR83], Otten and Van Ginneken [OTT84], Randelman and Grest [RAN86] and White [WHI84]). In this chapter a number of quantities are discussed that are defined in the theory of statistical mechanics for the description of a physical many-particle system at equilibrium and it will be elucidated how these quantities are related to the problem of solving large combinatorial optimization problems.

To develop, maintain, and update orbital debris environmental and break-up models, such as the NASA orbital debris engineering model (ordem) and the NASA standard satellite breakup model (ssbm), the NASA orbital debris program office (odpo) relies on the most recent validated datasets from various measurement sources, both laboratory and environmental. One key project that will provide insight for break-up events using modern-day spacecraft mate-rials and construction techniques is the debrisat laboratory hypervelocity-impact test. Based on the mass of the target, the projectile, and the impact velocity, the expected number of fragments greater than 2 mm, generated using the NASA ssbm, was estimated to be close to 85,000. To date, the debrisat fragment database continues to grow, with over 200,000 fragments collected that will help inform updates to the ssbm. Additionally, the growing fragment ensemble will support key parameters for the next release of NASA’s environmental models, employing fragment shapes, densities, and size distributions.

Covariance estimation and selection for high-dimensional multivariate datasets is a fundamental problem in modern statistics. Gaussian directed acyclic graph (DAG) models are a popular class of models used for this purpose. Gaussian DAG models introduce sparsity in the Cholesky factor of the inverse covariance matrix, and the sparsity pattern in turn corresponds to specific conditional independence assumptions on the underlying variables. A variety of priors have been developed in recent years for Bayesian inference in DAG models, yet crucial convergence and sparsity selection properties for these models have not been thoroughly investigated. Most of these priors are adaptations/generalizations of the Wishart distribution in the DAG context. In this paper, we consider a flexible and general class of these “DAG-Wishart” priors with multiple shape parameters. Under mild regularity assumptions, we establish strong graph selection consistency and establish posterior convergence rates for estimation when the number of variables $p$ is allowed to grow at an appropriate subexponential rate with the sample size $n$.

Consider the problem of estimating the mean of a Gaussian random vector when the mean vector is assumed to be in a given convex set. The most natural solution is to take the Euclidean projection of the data vector on to this convex set; in other words, performing "least squares under a convex constraint." Many problems in modern statistics and statistical signal processing theory are special cases of this general situation. Examples include the lasso and other high-dimensional regression techniques, function estimation problems, matrix estimation and completion, shape-restricted regression, constrained denoising, linear inverse problems, etc. This paper presents three general results about this problem, namely, (a) an exact computation of the main term in the estimation error by relating it to expected maxima of Gaussian processes (existing results only give upper bounds), (b) a theorem showing that the least squares estimator is always admissible up to a universal constant in any problem of the above kind and (c) a counterexample showing that least squares estimator may not always be minimax rate-optimal. The result from part (a) is then used to compute the error of the least squares estimator in two examples of contemporary interest.

Bayesian inference for models with intractable likelihood functions represents a challenging suite of problems in modern statistics. In this work we analyse the Conway-Maxwell-Poisson (COM-Poisson) distribution, a two parameter generalisation of the Poisson distribution. COM-Poisson regression modelling allows the flexibility to model dispersed count data as part of a generalised linear model (GLM) with a COM-Poisson response, where exogenous covariates control the mean and dispersion level of the response. The major difficulty with COM-Poisson regression is that the likelihood function contains multiple intractable normalising constants and is not amenable to standard inference and Markov Chain Monte Carlo (MCMC) techniques. Recent work by Chanialidis et al. (2018) has seen the development of a sampler to draw random variates from the COM-Poisson likelihood using a rejection sampling algorithm. We provide a new rejection sampler for the COM-Poisson distribution which significantly reduces the central processing unit (CPU) time required to perform inference for COM-Poisson regression models. An extension of this work shows that for any intractable likelihood function with an associated rejection sampler it is possible to construct unbiased estimators of the intractable likelihood which proves useful for model selection or for use within pseudo-marginal MCMC algorithms (Andrieu and Roberts, 2009). We demonstrate all of these methods on a real-world dataset of takeover bids.

This paper reviews several statistical problems arising in Naval Engineering that the authors were involved with professionally or at NSWCCD over the past several years. The considered problems relate to statistical uncertainty, characterizing rare events, and ocean modeling, and naturally involve a stochastic component which needs to be accounted for through statistical methods. In statistical uncertainty, for example, one problem consists of constructing confidence intervals for measured quantities of interest (e.g. the variance of a ship motion) when temporal dependence in a signal needs to be taken into account. In characterizing rare events (e.g. ship capsizing or broaching to), a common problem is to estimate their frequency, which can be carried out under the umbrella of the statistical Extreme Value Theory. In ocean modeling, spatiotemporal statistical modeling of significant wave height has attracted much attention, especially in the context of modern treatments of “big data.” The focus throughout this work is on the theoretical underpinnings of these statistical problems, related work in the Statistics literature, and some open future directions.

One of the first steps in attempting to understand data is to visualize it. Visualization of data and information provides a wealth of tools that can be used in detecting features, in discovering relationships, and finally in retaining the knowledge gained. Graphical displays have always been an important part of statistical data analysis, but with the continuing developments in high-speed computers and high-resolution devices, the usefulness of graphics has greatly increased. Higher resolution makes for a more visually pleasing display, and occasionally it allows features to be seen that could not be distinguished otherwise. The most important effects of the computer on graphical methods in statistics, however, arise from the ease and speed with which graphical displays can be produced, rather than from the resolution. Rapid production of graphical displays has introduced motion and articulated projections and sections into statistical graphics. Such graphical methods are important tools of computational statistics. The multiple views are tools of discovery, not just ways of displaying a set of data that has already been analyzed. Although the power of graphical displays has greatly increased, some of the most useful graphs are the simple ones, as illustrated in Section 1.1, and they should not be ignored just because we can do more impressive things. Proper design of a graphical display depends on the context of the application and the purpose of the graphics, whether it is for the analyst to get a better understanding of the data or to present a picture that conveys a message. Our emphasis in the following discussion is on methods useful in exploratory graphics. One thing that is lagging in the statistical literature is the use of color in graphical displays. The simple mechanics used in producing popular magazines are yet to be incorporated in the production of learned journals. Journals available in electronic form do not have these production problems, and it is likely that the paper versions of many journals will be discontinued before the production of color is mastered.

Many statistical inference problems correspond to recovering the values of a set of hidden variables from sparse observations on them. For instance, in a planted constraint satisfaction problem such as planted 3-SAT, the clauses are sparse observations from which the hidden assignment is to be recovered. In the problem of community detection in a stochastic block model, the community labels are hidden variables that are to be recovered from the edges of the graph. Inspired by ideas from statistical physics, the presence of a stable fixed point for belief propogation has been widely conjectured to characterize the computational tractability of these problems. For community detection in stochastic block models, many of these predictions have been rigorously confirmed. In this work, we consider a general model of statistical inference problems that includes both community detection in stochastic block models, and all planted constraint satisfaction problems as special cases. We carry out the cavity method calculations from statistical physics to compute the regime of parameters where detection and recovery should be algorithmically tractable. At precisely the predicted tractable regime, we give: (i) a general polynomial-time algorithm for the problem of detection: distinguishing an input with a planted signal from one without; (ii) a general polynomial-time algorithm for the problem of recovery: outputting a vector that correlates with the hidden assignment significantly better than a random guess would. Analogous to the spectral algorithm for community detection [1], [2], the detection and recovery algorithms are based on the spectra of a matrix that arises as the derivatives of the belief propagation update rule. To devise a spectral algorithm in our general model, we obtain bounds on the spectral norms of certain families of random matrices with correlated and matrix valued entries. We then demonstrate how eigenvectors of various powers of the matrix can be used to partially recover the hidden variables.

Joint sparse subspace clustering via fast [formula omitted]-norm constrained optimization

The paper proposes an agent-based evolutionary ecological-economic model that captures the link between the economy and the ecosystem in a more inclusive way than standard economic optimization models do. We argue that an evolutionary approach is required to understand the integrated dynamics of both systems, i.e. micro–macro feedbacks. In the paper, we illustrate that claim by analyzing the non-triviality of finding a sustainability policy mix as a use case for such a coupled system. The model has three characteristics distinguishing it from traditional environmental and resource economic models: (1) it implements a multi-dimensional link between the economic and the ecological system, considering side effects of production, and thus combines the analyses of environmental and resource economics; (2) following literature from biology, it uses a discrete time approach for the biological resource allowing for the whole range of stability regimes instead of artificially stabilizing the system, and (3) it links this resource system to an evolving, agent-based economy (on the basis of a Nelson-Winter model) with bounded rational decision makers instead of the standard optimization model. The policy case illustrates the relevance of the proposed integrated assessment as it delivers some surprising results on the effects of combined and consecutively introduced policies that would go unnoticed in standard models.

The first Survey and Review article in this issue is “Decay Properties of Spectral Projectors with Applications to Electronic Structure,” by Michele Benzi, Paolo Boito, and Nader Razouk. The linear scaling methods that motivate this work use a reformulation of the conventional electronic structure calculation. Instead of iterating through linear eigenvalue problems, a sequence of spectral projectors, known as density matrices, is computed, from which physical quantities of interest follow directly. Large-scale computations then become feasible, provided that entries in the density matrices are localized; that is, they decay rapidly as we move away from the diagonal. The authors first introduce the basic principles of electronic structure theory and then survey current computational approaches, most of which have arisen in the physical sciences, and their underlying localization requirements. The overview highlights a distinction, often blurred outside the mathematics literature, between physical insight/intuition and mathematical rigor. The key contribution is then to formulate the localization problem mathematically, with transparent assumptions, and develop a unified analytical approach. The tools of matrix analysis and approximation theory are used to justify, where feasible, the localization “results” that are relied upon by today's algorithms. The theory is framed in an asymptotic regime where the system volume increases but the density of particles remains fixed. This type of thermodynamic limit will be familiar to readers who work on multiscale models in physics and chemistry, but may seem strange to numerical analysts who are used to “convergence” in the sense of fixed problem size and arbitrarily fine mesh. The upshot is a sequence of matrices of increasing dimension that look very different from those arising in the discretization of PDEs. After pushing the theory as far as possible, the authors comment on the practical implications of their bounds and raise a number of open questions. They also point out that these decay results may find useful application in other, unrelated, areas: quantum information theory, complex networks, and eigenvalue solvers for tridiagonal matrices. This article illustrates how applied analysis can justify some of the leaps of faith made in the physical sciences and also resolve controversies (section 8.7). It will be of particular interest to readers who work in matrix computation and approximation theory. The second article, “Conditional Gradient Algorithms for Rank-One Matrix Approximations with a Sparsity Constraint,” by Ronny Luss and Marc Teboulle, shares with the first article the theme of studying algorithms based on large-scale matrix computations for which our current analytical understanding fails to explain the performance seen in practice. Given a symmetric matrix, $A$, most applied mathematicians know that dimension reduction can be achieved by using the dominant eigenvectors (singular vectors) of $A$ to form optimal low-rank approximations. In the statistics literature, modulo a centering operation, the same idea is called principal component analysis (PCA). This type of least-squares approach tries to recover all rows and columns of $A$; equivalently, if we think of $A$ as representing pairwise interactions between nodes in a network, it aims to summarize the full set of interactions. It is, of course, possible to impose a postprocessing threshold to cut off the less important contributions, but it is also reasonable to shoot directly for the important rows/columns or the important network nodes. This alternative leads us to the class of problems discussed here---matrix optimization problems where there is a prescribed upper bound on the number of nonzeros in the required “eigenvectors.” The authors discuss a variety of computational approaches that have been proposed in the literature and introduce a unifying framework, which they call ConGradU, to characterize and analyze them. The basic iterative scheme, summarized in Algorithm 1, has an optimization substep with closed-form solutions in a number of important cases. The authors also include an informative computational example based on the textual content of State of the Union addresses from 1790--2011. For example, Table 6.2 shows the key word stems associated with the three principal factors for both thresholded PCA and the more direct sparsity-seeking alternative. It is clear that the sparse version is summarizing different information. This article will appeal to readers who are keen to keep up with developments in modern optimization and to learn about techniques that fall under the current “Big Data” banner.

N the aerospace industry, the stress analysis of complex axisymmetric structures of arbitrary shape subjected to thermal and mechanical loads is of considerable interest. Rocket nozzles and cases, solid-propellant grains, and spacecraft heat shields are practical examples of such structures. Although the governing differential equations for solids of revolution have been known for many years, closed form solutions have been obtained for only a limited number of structures; thus, the stress analyst must rely on experimental or numerical techniques. The finite difference method has been the most popular of the numerical techniques; however, for structures of composite materials and of arbitrary geometry, this procedure is difficult to apply. In the present investigation, the finite element idealization is used as the basic numerical procedure. This technique has been applied successfully in the stress analysis of many complex structures.13 In Ref. 4 impressive results were obtained in the analysis of axisymmetric shells approximated by a series of truncated cone elements. The approach, which is presented here, is similar in many respects to existing methods used in the analysis of two-dimensional stress problems.57 Recently, the finite element method was applied to the structural analysis of axisymmetric solids subjected to axisymmetric loads.8 In the present paper, the finite element method is used in the determination of stresses and displacements developed within elastic solids of revolution which are subjected to axisymmetric or nonaxisymmetr ic loads. Emphasis is placed on the application of the technique to complex aerospace structures.

Simulation of obstacles’ effect on industrial robots’ working space using genetic algorithm

N the aerospace industry, the stress analysis of complex axisymmetric structures of arbitrary shape subjected to thermal and mechanical loads is of considerable interest. Rocket nozzles and cases, solid-propellant grains, and spacecraft heat shields are practical examples of such structures. Although the governing differential equations for solids of revolution have been known for many years, closed form solutions have been obtained for only a limited number of structures; thus, the stress analyst must rely on experimental or numerical techniques. The finite difference method has been the most popular of the numerical techniques; however, for structures of composite materials and of arbitrary geometry, this procedure is difficult to apply. In the present investigation, the finite element idealization is used as the basic numerical procedure. This technique has been applied successfully in the stress analysis of many complex structures.13 In Ref. 4 impressive results were obtained in the analysis of axisymmetric shells approximated by a series of truncated cone elements. The approach, which is presented here, is similar in many respects to existing methods used in the analysis of two-dimensional stress problems.57 Recently, the finite element method was applied to the structural analysis of axisymmetric solids subjected to axisymmetric loads.8 In the present paper, the finite element method is used in the determination of stresses and displacements developed within elastic solids of revolution which are subjected to axisymmetric or nonaxisymmetr ic loads. Emphasis is placed on the application of the technique to complex aerospace structures.

Structural reliability methodology — application at the system level