Randomized algorithms for large-scale inverse problems with general Tikhonov regularizations

The Newsletter is a key element in further enhancing the value of the journal to the inverse problems community. So why not be a part of this exciting forum by sending to our Bristol office material suitable for inclusion under any of the categories mentioned above. Your contributions will be very welcome. Book review Introduction to Inverse Problems in Imaging M Bertero and P Boccacci 1998 Bristol: Institute of Physics 362 pp ISBN 0-7503-0435-9 (pbk) £25.00, $49.00 This book shows that several problems in imaging are in fact linear inverse problems. In general inverse problems are ill-posed in the sense that small perturbations in the (measured) data have a significant influence on the output. Consequently, numerical methods developed in a general setting of inverse problems which cure the ill-posedness (such methods are called regularization methods) can be applied to solve problems in imaging. This is an excellent textbook on the principle of linear inverse problems, methods of their numerical solution and practical applications in imaging. It introduces basic ideas and methods for the solution of inverse and ill-posed problems while avoiding the mathematics of functional treatment of operator equations. This goal is achieved by a bottom-up presentation: the authors focus on the two problems of image deconvolution and tomography (bottom) and develop regularization theory specifically for these two problems. Most other books which are concerned with the numerical solution of inverse problems present a top-down approach. That is, a general regularization theory (top) is developed which is then applied to particular problems. The distinguished presentation makes the book very useful to students in applied mathematics, practitioners in engineering science and image processing. This book has the best prerequisites to establish a close link between the mathematical fields of imaging and inverse problems. Mainly the book focuses on two topics: image deconvolution and linear imaging systems. 1. In image deconvolution the goal is to restore a space invariant blurred image. This part of the book contains mathematical tools in image deconvolution, examples of space invariant imaging systems, a comparison of image deconvolution techniques and low-pass filtering techniques, the analysis of the ill-posedness of the deconvolution problem, regularization methods for deconvolution such as constrained least squares regularization, Tikhonov regularization, iterative regularization and statistical methods. Since the authors focus on particular problems they are able to analyse various methods for deconvolution on a very concrete level. As an example I would like to mention that the authors analyse Fourier methods for deconvolution. Thus they provide an optimal starting point to the theory of inverse problems for people working in the area of signal and image processing. 2. The second part of the book considers linear imaging systems that are not of a convolution type such as tomography problems, diffraction problems and inverse scattering problems. Efficient numerical methods for the solution of this class of problems such as singular value decomposition, Tikhonov regularization methods and Fourier-based methods are presented. O Scherzer Universität Linz

We develop a framework for goal-oriented optimal design of experiments (GOODE) for large-scale Bayesian linear inverse problems governed by PDEs. This framework differs from classical Bayesian optimal design of experiments (ODE) in the following sense: we seek experimental designs that minimize the posterior uncertainty in the experiment end-goal, e.g. a quantity of interest (QoI), rather than the estimated parameter itself. This is suitable for scenarios in which the solution of an inverse problem is an intermediate step and the estimated parameter is then used to compute a QoI. In such problems, a GOODE approach has two benefits: the designs can avoid wastage of experimental resources by a targeted collection of data, and the resulting design criteria are computationally easier to evaluate due to the often low-dimensionality of the QoIs. We present two modified design criteria, A-GOODE and D-GOODE, which are natural analogues of classical Bayesian A- and D-optimal criteria. We analyze the connections to other ODE criteria, and provide interpretations for the GOODE criteria by using tools from information theory. Then, we develop an efficient gradient-based optimization framework for solving the GOODE optimization problems. Additionally, we present comprehensive numerical experiments testing the various aspects of the presented approach. The driving application is the optimal placement of sensors to identify the source of contaminants in a diffusion and transport problem. We enforce sparsity of the sensor placements using an -norm penalty approach, and propose a practical strategy for specifying the associated penalty parameter.

Many inverse problems include nuisance parameters which, while not of direct interest, are required to recover primary parameters. The structure of these problems allows efficient optimization strategies—a well-known example is variable projection, where nonlinear least-squares problems which are linear in some parameters can be very efficiently optimized. In this paper, we extend the idea of projecting out a subset over the variables to a broad class of maximum likelihood and maximum a posteriori likelihood problems with nuisance parameters, such as variance or degrees of freedom (d.o.f.). As a result, we are able to incorporate nuisance parameter estimation into large-scale constrained and unconstrained inverse problem formulations. We apply the approach to a variety of problems, including estimation of unknown variance parameters in the Gaussian model, d.o.f. parameter estimation in the context of robust inverse problems, and automatic calibration. Using numerical examples, we demonstrate improvement in recovery of primary parameters for several large-scale inverse problems. The proposed approach is compatible with a wide variety of algorithms and formulations, and its implementation requires only minor modifications to existing algorithms.

The identification of nonhomogeneous elastic property distributions has been traditionally achieved with well acknowledged optimization based inverse approaches, but when full-field displacement measurements are available, the virtual fields method (VFM) can be computationally more efficient by converting the large-scale optimization problem into multiple small-scale optimization problems. A possible downside of the VFM so far was not to take into account prior knowledge, which is often available and needed when there is a very large number of unknowns and the inverse problem is ill-posed. In this work, different approaches are proposed for introducing regularization into the VFM, aiming to penalize the local variations of identified stiffness properties in order to reduce the effects of uncertainty in the inverse problem resolution. The feasibility and accuracy of the regularized VFM are tested through several numerical and experimental datasets. It is shown that the main advantage of the novel VFM approaches is the low computational cost, as large-scale inverse problems with 10,000 unknown parameters can be solved within several seconds using a standard personal computer. Although the regularized VFM can successfully detect a stiff inclusion in a soft solid with high accuracy, regularization also introduces unexpected spurious effects in the results, blurring the interface between soft and stiff regions. We also observed that the regularization did not improve the smoothness significantly due to local effects of the small-scale optimization problem introduced in the proposed VFM method. Therefore, traditional regularization, which penalizes local variations of identified stiffness properties, can be combined with the VFM to solve inverse problems with a high computational efficiency, but supplemental regularization conditions will need to be adapted in the future to better delineate soft-stiff interfaces with this methodology.

The resolution of many large-scale inverse problems using MCMC methods\nrequires a step of drawing samples from a high dimensional Gaussian\ndistribution. While direct Gaussian sampling techniques, such as those based on\nCholesky factorization, induce an excessive numerical complexity and memory\nrequirement, sequential coordinate sampling methods present a low rate of\nconvergence. Based on the reversible jump Markov chain framework, this paper\nproposes an efficient Gaussian sampling algorithm having a reduced computation\ncost and memory usage. The main feature of the algorithm is to perform an\napproximate resolution of a linear system with a truncation level adjusted\nusing a self-tuning adaptive scheme allowing to achieve the minimal computation\ncost. The connection between this algorithm and some existing strategies is\ndiscussed and its efficiency is illustrated on a linear inverse problem of\nimage resolution enhancement.\n

Stochastic inverse modeling deals with the estimation of functions from sparse data, which is a problem with a nonunique solution, with the objective to evaluate best estimates, measures of uncertainty, and sets of solutions that are consistent with the data. As finer resolutions become desirable, the computational requirements increase dramatically when using conventional solvers. A method is developed in this paper to solve large-scale stochastic linear inverse problems, based on the hierarchical matrix (or ℋ 2 matrix) approach. The proposed approach can also exploit the sparsity of the underlying measurement operator, which relates observations to unknowns. Conventional direct algorithms for solving large-scale linear inverse problems, using stochastic linear inversion techniques, typically scale as 𝒪(n 2 m+nm 2), where n is the number of measurements and m is the number of unknowns. We typically have n ≪ m. In contrast, the algorithm presented here scales as 𝒪(n 2 m), i.e., it scales linearly with the larger problem dimension m. The algorithm also allows quantification of uncertainty in the solution at a computational cost that also grows only linearly in the number of unknowns. The speedup gained is significant since the number of unknowns m is often large. The effectiveness of the algorithm is demonstrated by solving a realistic crosswell tomography problem by formulating it as a stochastic linear inverse problem. In the case of the crosswell tomography problem, the sparsity of the measurement operator allows us to further reduce the cost of our proposed algorithm from 𝒪(n 2 m) to $\mathcal {O}(n^{2} \sqrt {m} + nm)$ . The computational speedup gained by using the new algorithm makes it easier, among other things, to optimize the location of sources and receivers, by minimizing the mean square error of the estimation. Without this fast algorithm, this optimization would be computationally impractical using conventional methods.

We study Bayesian inference methods for solving linear inverse problems, focusing on hierarchical formulations where the prior or the likelihood function depend on unspecified hyperparameters. In practice, these hyperparameters are often determined via an empirical Bayesian method that maximizes the marginal likelihood function, i.e. the probability density of the data conditional on the hyperparameters. Evaluating the marginal likelihood, however, is computationally challenging for large-scale problems. In this work, we present a method to approximately evaluate marginal likelihood functions, based on a low-rank approximation of the update from the prior covariance to the posterior covariance. We show that this approximation is optimal in a minimax sense. Moreover, we provide an efficient algorithm to implement the proposed method, based on a combination of the randomized SVD and a spectral approximation method to compute square roots of the prior covariance matrix. Several numerical examples demonstrate good performance of the proposed method.

Seismic reflectivity inversion problem can be formulated using a basis-pursuit method, aiming to generate a sparse reflectivity series of the subsurface media. In the basis-pursuit method, the reflectivity series is composed by large amounts of even and odd dipoles, thus the size of the seismic response matrix is huge and the matrix operations involved in seismic inversion are very time-consuming. In order to accelerate the matrix computation, a basis-pursuit method-based seismic inversion algorithm is implemented on Graphics Processing Unit (GPU). In the basis-persuit inversion algorithm, the problem is imposed with a L1-norm model constraint for sparsity, and this L1-norm basis-pursuit inversion problem is reformulated using a linear programming method. The core problems in the inversion are large-scale linear systems, which are resolved by a parallelised conjugate gradient method. The performance of this fully parallelised implementation is evaluated and compared to the conventional serial coding. Specifically, the investigation using several field seismic data sets with different sizes indicates that GPU-based parallelisation can significantly reduce the computational time with an overall factor up to 145. This efficiency improvement demonstrates a great potential of the basis-pursuit inversion method in practical application to large-scale seismic reflectivity inversion problems.

Over the last two decades, anomalous diffusion processes in which the mean squares variance grows slower or faster than that in a Gaussian process have found many applications. At a macroscopic level, these processes are adequately described by fractional differential equations, which involves fractional derivatives in time or/and space. The fractional derivatives describe either history mechanism or long range interactions of particle motions at a microscopic level. The new physics can change dramatically the behavior of the forward problems. For example, the solution operator of the time fractional diffusion diffusion equation has only limited smoothing property, whereas the solution for the space fractional diffusion equation may contain weak singularity. Naturally one expects that the new physics will impact related inverse problems in terms of uniqueness, stability, and degree of ill-posedness. The last aspect is especially important from a practical point of view, i.e., stably reconstructing the quantities of interest. In this paper, we employ a formal analytic and numerical way, especially the two-parameter Mittag-Leffler function and singular value decomposition, to examine the degree of ill-posedness of several ‘classical’ inverse problems for fractional differential equations involving a Djrbashian–Caputo fractional derivative in either time or space, which represent the fractional analogues of that for classical integral order differential equations. We discuss four inverse problems, i.e., backward fractional diffusion, sideways problem, inverse source problem and inverse potential problem for time fractional diffusion, and inverse Sturm–Liouville problem, Cauchy problem, backward fractional diffusion and sideways problem for space fractional diffusion. It is found that contrary to the wide belief, the influence of anomalous diffusion on the degree of ill-posedness is not definitive: it can either significantly improve or worsen the conditioning of related inverse problems, depending crucially on the specific type of given data and quantity of interest. Further, the study exhibits distinct new features of ‘fractional’ inverse problems, and a partial list of surprising observations is given below. (a) Classical backward diffusion is exponentially ill-posed, whereas time fractional backward diffusion is only mildly ill-posed in the sense of norms on the domain and range spaces. However, this does not imply that the latter always allows a more effective reconstruction. (b) Theoretically, the time fractional sideways problem is severely ill-posed like its classical counterpart, but numerically can be nearly well-posed. (c) The classical Sturm–Liouville problem requires two pieces of spectral data to uniquely determine a general potential, but in the fractional case, one single Dirichlet spectrum may suffice. (d) The space fractional sideways problem can be far more or far less ill-posed than the classical counterpart, depending on the location of the lateral Cauchy data. In many cases, the precise mechanism of these surprising observations is unclear, and awaits further analytical and numerical exploration, which requires new mathematical tools and ingenuities. Further, our findings indicate fractional diffusion inverse problems also provide an excellent case study in the differences between theoretical ill-conditioning involving domain and range norms and the numerical analysis of a finite-dimensional reconstruction procedure. Throughout we will also describe known analytical and numerical results in the literature.

Characterisation of the transmissivity field of a fractured and karstic aquifer, Southern France

This paper develops two accelerated Bregman Operator Splitting (BOS) algorithms with backtracking for solving regularized large-scale linear inverse problems, where the regularization term may not be smooth. The first algorithm improves the rate of convergence for BOSVS [5] in terms of the smooth component in the objective function by incorporating Nesterov's multi-step acceleration scheme under the assumption that the feasible set is bounded. The second algorithm is capable of dealing with the case where the feasible set is unbounded. Moreover, it allows more aggressive stepsize than that in the first scheme by properly selecting the penalty parameter and jointly updating the acceleration parameter and stepsize. Both algorithms exhibit better practical performance than BOSVS and AADMM [21], while preserve the same accelerated rate of convergence as that for AADMM. The numerical results on total-variation based image reconstruction problems indicate the effectiveness of the proposed algorithms.

Inverse problems in nano-biomolecular circuits typically involve stochastic functional elements that admit non-linear relationships between different circuit variables. In this regard, a factor graph representation serves as an important visualization and analysis tool that can be used to compute inferences over an arbitrary large-scale biomolecular circuit. Solving the inverse problem using a factor graph entails passing of messages/signals between the internal nodes of the biomolecular circuit, and the steady-state distribution of the messages can be used to determine the dynamics of the circuit and the final solution. In this paper, we present an open-source simulation tool that we have developed which can be used to verify the functionality of a generic nano-biomolecular circuit. As a representative example, we apply the simulation software to estimate the reliability of a 10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> size biosensor array where each element of the array is comprised of our previously reported antigen-antibody based biomolecular circuit. This example will demonstrate the utility of the proposed software in emulating the functionality of micro and nano biosensor arrays without resorting to time-consuming and laborious fabrication procedure and laboratory experiments.

Ill-posed linear inverse problems appear frequently in various signal processing applications. It can be very useful to have theoretical characterizations that quantify the level of ill-posedness for a given inverse problem and the degree of ambiguity that may exist about its solution. Traditional measures of ill-posedness, such as the condition number of a matrix, provide characterizations that are global in nature. While such characterizations can be powerful, they can also fail to provide full insight into situations where certain entries of the solution vector are more or less ambiguous than others. In this work, we derive novel theoretical lower- and upper-bounds that apply to individual entries of the solution vector, and are valid for all potential solution vectors that are nearly data-consistent. These bounds are agnostic to the noise statistics and the specific method used to solve the inverse problem, and are also shown to be tight. In addition, our results also lead us to introduce an entrywise version of the traditional condition number, which provides a substantially more nuanced characterization of scenarios where certain elements of the solution vector are less sensitive to perturbations than others. Our results are illustrated in an application to magnetic resonance imaging reconstruction, and we include discussions of practical computation methods for large-scale inverse problems, connections between our new theory and the traditional Cramér-Rao bound under statistical modeling assumptions, and potential extensions to cases involving constraints beyond just data-consistency.

The inverse problem for the acoustic equation is considered. We propose a method of reconstruction of the density approximating 2D inverse acoustic problem by a finite system of one dimensional inverse acoustic problems. The 2D analogy of the Gel'fand–Levitan–Krein method is established. The inverse acoustic problem is formulated and the short outline of the history and development in this field are given in Section 1. In Section 2 we consider the 2D analogy of the Gel'fand–Levitan–Krein equation. The N-approximation of the Gel'fand–Levitan–Krein equation is obtained for inverse acoustic problem in Section 3. The numerical results are presented in Section 4.

This paper introduces a new efficient algorithm to approximate a solution of linear least squares problems subject to box constraints. Starting from an equivalent reformulation of the associated KKT conditions as a nonlinear system of equations, the new approach formulates a fixed-point iteration scheme that involves the solution of an adaptively preconditioned linear sys-tem, which is handled by flexible CGLS. The resulting method is dubbed ‘box-FCGLS’. Box-FCGLS is applied to solve large-scale linear inverse problems arising in imaging applications, where box constraints encode prior information about the solution. The results of extensive numerical testings show the performance of box-FCGLS that, when compared to accelerated gradient-based optimization schemes for box-constrained least squares problems, efficiently delivers results of equal or better quality.