Fast Optimization Methods for L1 Regularization: A Comparative Study and Two New Approaches

Bioluminescence imaging (BLI) is an optical molecular imaging modality for monitoring physiological and pathological activities at the molecular level. The information of bioluminescent probe distribution in small animals can be threedimensionally and quantitatively obtained by bioluminescence tomography (BLT). Due to ill-posed nature, BLT may bear multiple solutions and aberrant reconstruction in the presence of measurement noise and optical parameter mismatches. Among different regularization methods, L2-type regularization strategy is the most popular and commonly-applied method, which minimizes the output-least-square formulation incorporated with the l2-norm regularization term to stabilize the problem. However, it often imposes over-smoothing on the reconstruction results. In contrast, for many practical applications, such as early detection of tumors, the volumes of the bioluminescent sources are very small compared with the whole body. In this paper, L1 regularization is used to fully take advantage of the sparsity prior knowledge and improve both efficiency and stability. And then a reconstruction method based on the augmented Lagrangian approach is proposed, which considers the BLT problem as the constrained optimization problem and employs the Bregman iterative method to deal with it. By using "divide and conquer" approach, the optimization problem can be exactly and fast solved by iteratively solving a sequence of unconstrained subproblems. To evaluate the performance of the proposed method in turbid mouse geometry, stimulate experiments with a heterogeneous 3D mouse atlas are conducted. In addition, physical experiments further demonstrate the potential of the proposed algorithm in practical applications.

Abstract. Accurate quantification of long-term trends in stratospheric ozone can be challenging due to their sensitivity to natural variability, the quality of the observational datasets, and non-linear changes in forcing processes as well as the statistical methodologies. Multivariate linear regression (MLR) is the most commonly used tool for ozone trend analysis; however, the complex coupling in many atmospheric processes can make it prone to the issue of over-fitting when using the conventional ordinary-least-squares (OLS) approach. To overcome this issue, here we adopt a regularized (ridge) regression method to estimate ozone trends and quantify the influence of individual processes. We use the Stratospheric Water and OzOne Satellite Homogenized (SWOOSH) merged dataset (v2.7) to derive stratospheric ozone profile trends for the period 1984–2020. Besides SWOOSH, we also analyse a machine-learning-based satellite-corrected gap-free global stratospheric ozone profile dataset from a chemical transport model (ML-TOMCAT) and output from a chemical transport model (TOMCAT) simulation forced with European Centre for Medium-Range Weather Forecasts (ECMWF) ERA5 reanalysis. For 1984–1997, we observe smaller negative trends in the SWOOSH stratospheric ozone profile using ridge regression compared to OLS. Except for the tropical lower stratosphere, the largest differences arise in the mid-latitude lowermost stratosphere (>4 % per decade difference at 100 hPa). From 1998 and the onset of ozone recovery in the upper stratosphere, the positive trends estimated using the ridge regression model (∼1 % per decade near 2 hPa) are smaller than those using OLS (∼2 % per decade). In the lower stratosphere, post-1998 negative trends with large uncertainties are observed and ridge-based trend estimates are somewhat smaller and less variable in magnitude compared to the OLS regression. Aside from the tropical lower stratosphere, the largest difference is around 2 % per decade at 100 hPa (with ∼3 % per decade uncertainties for individual trends) in northern mid-latitudes. For both time periods the SWOOSH data produce large negative trends in the tropical lower stratosphere with a correspondingly large difference between the two trend methods. In both cases the ridge method produces a smaller trend. The regression coefficients from both OLS and ridge models, which represent ozone variations associated with natural processes (e.g. the quasi-biennial oscillation, solar variability, El Niño–Southern Oscillation, Arctic Oscillation, Antarctic Oscillation, and Eliassen–Palm flux), highlight the dominance of dynamical processes in controlling lower-stratospheric ozone concentrations. Ridge regression generally yields smaller regression coefficients due to correlated explanatory variables, and care must be exercised when comparing fit coefficients and their statistical significance across different regression methods. Comparing the ML-TOMCAT-based trend estimates with the ERA5-forced model simulation, we find ML-TOMCAT shows significant improvements with much better consistency with the SWOOSH dataset, despite the ML-TOMCAT training period overlapping with SWOOSH only for the Microwave Limb Sounder (MLS) measurement period. The largest inconsistencies with respect to SWOOSH-based trends post-1998 appear in the lower stratosphere where the ERA5-forced model simulation shows positive trends for both the tropics and the mid-latitudes. The large differences between satellite-based data and the ERA5-forced model simulation confirm significant uncertainties in ozone trend estimates, especially in the lower stratosphere, underscoring the need for caution when interpreting results obtained with different regression methods and datasets.

We consider linear operator equations of the form where $K:{\cal X}\to{\cal Y}$ is a compact linear operator between Hilbert spaces ${\cal X} \hbox{ and } {\cal Y}.$ We assume y to be attainable, i.e., that problem (1) has a solution x† = K†y of minimal norm. Here K† denotes the (Moore‐Penrose) generalized inverse operator of K, which is unbounded when K is compact, with infinite dimensional range. Hence problem (1) is ill‐posed and has to be regularized in order to compute a numerical solution. We want to approximate the solution x† of the equation (1), when only an approximation yδ of y is available with ‖yδ − y‖ ≤ δ, where δ is called the noise level. Since K†yδ is not a good approximation of x†, we approximate x† with xδα := Rαyδ where {Rα} is a family of continuous operators depending on a parameter α. A classical example is the Tikhonov regularization defined by Rα = (K*K + αI)−1 K*, where I denotes the identity and K* the adjoint of K.Recently, new Tikhonov based regularization methods have been proposed in [1], [2] and [3], under the name of fractional Tikhonov, to reduce the oversmoothing property of the Tikhonov regularization in standard form, in order to preserve the details of the approximated solution. Their regularization and convergence properties have been previously investigated showing that they are of optimal order.In this paper, we firstly provide a saturation result similar to the well‐known saturation result for Tikhonov regularization [6]: indeed, Tikhonov regularization under suitable a‐priori assumption and a‐priori choice rule, α = α(δ) ∼ c(δ)2/3, is of optimal order and the best possible convergence rate obtainable is ‖xδα−x†‖ = O(δ2/3). On the other hand, let R(K) be the range of K and let Q be the orthogonal projector onto $\overline{R(K)},$, if then x† = 0, as long as R(K) is not closed, and this shows how Tikhonov regularization for an ill‐posed problem with compact operator never yields a convergence rate which is faster than O(δ2/3), since it saturates at this rate. Such results motivated us to introduce the iterated versions of fractional Tikhonov methods in the same spirit of the iterated Tikhonov method. We prove that those iterated methods can overcome the afore‐mentioned saturation results.Afterwards, inspired by the works [4,5] we introduce the nonstationary variants of our iterated methods. Differently from the nonstationary iterated Tikhonov, we have two nonstationary sequences of parameters. In the noise free case, we give sufficient conditions on these sequences to guarantee the convergence providing also the corresponding convergence rates. In the noise case, we show the stability of the proposed iterative schemes proving that they are regularization methods. For the brief space we have at disposal, we will only cite some of the main results we obtained for one kind of fractional Tikhonov algorithms, that we call weighted Tikhonov. For the results about the other fractional Tikhonov algorithm analysis, proofs, numerical examples and full details, we invite the interested reader to look at the full paper [7]. (© 2015 Wiley‐VCH Verlag GmbH & Co. KGaA, Weinheim)

The Newsletter is a key element in further enhancing the value of the journal to the inverse problems community. So why not be a part of this exciting forum by sending to our Bristol office material suitable for inclusion under any of the categories mentioned above. Your contributions will be very welcome. Book review Introduction to Inverse Problems in Imaging M Bertero and P Boccacci 1998 Bristol: Institute of Physics 362 pp ISBN 0-7503-0435-9 (pbk) £25.00, $49.00 This book shows that several problems in imaging are in fact linear inverse problems. In general inverse problems are ill-posed in the sense that small perturbations in the (measured) data have a significant influence on the output. Consequently, numerical methods developed in a general setting of inverse problems which cure the ill-posedness (such methods are called regularization methods) can be applied to solve problems in imaging. This is an excellent textbook on the principle of linear inverse problems, methods of their numerical solution and practical applications in imaging. It introduces basic ideas and methods for the solution of inverse and ill-posed problems while avoiding the mathematics of functional treatment of operator equations. This goal is achieved by a bottom-up presentation: the authors focus on the two problems of image deconvolution and tomography (bottom) and develop regularization theory specifically for these two problems. Most other books which are concerned with the numerical solution of inverse problems present a top-down approach. That is, a general regularization theory (top) is developed which is then applied to particular problems. The distinguished presentation makes the book very useful to students in applied mathematics, practitioners in engineering science and image processing. This book has the best prerequisites to establish a close link between the mathematical fields of imaging and inverse problems. Mainly the book focuses on two topics: image deconvolution and linear imaging systems. 1. In image deconvolution the goal is to restore a space invariant blurred image. This part of the book contains mathematical tools in image deconvolution, examples of space invariant imaging systems, a comparison of image deconvolution techniques and low-pass filtering techniques, the analysis of the ill-posedness of the deconvolution problem, regularization methods for deconvolution such as constrained least squares regularization, Tikhonov regularization, iterative regularization and statistical methods. Since the authors focus on particular problems they are able to analyse various methods for deconvolution on a very concrete level. As an example I would like to mention that the authors analyse Fourier methods for deconvolution. Thus they provide an optimal starting point to the theory of inverse problems for people working in the area of signal and image processing. 2. The second part of the book considers linear imaging systems that are not of a convolution type such as tomography problems, diffraction problems and inverse scattering problems. Efficient numerical methods for the solution of this class of problems such as singular value decomposition, Tikhonov regularization methods and Fourier-based methods are presented. O Scherzer Universität Linz

For interior tomography based on truncated Hilbert transform (THT), the recently proposed truncated singular value decomposition (TSVD) reconstruction method uses a regularizati on parameter given directly. In this paper, a method of choosing the regularization parameter is presented based on L-curve to get an optimal regularization parameter in theoretical sense. Furthermore, we develop a Tikhonov regula rization method in comparison to TSVD. Simulation results indicate that both of the two regularization methods with the optimal regularization parameters have good performances on the image quality for both cases of noise-free and noisy projections. Keywords: Interior Tomography, Truncated Hilbert Transform, L-curve, Truncated Singular Value Decomposition, Tikhonov Regularization 1. INTRODUCTION It had been believed that the solution to the interior tomography is not unique fo r a long time. Recently, inspired by the concept of the differential backprojection (DBP) proposed by Noo et al [1] and Pan et al [2], a novel solution for the interior problem was proposed with numerical results demonstrat ed that the interior problem can be uniquely and stably solved if a sub-region inside a region of interest (ROI) is known [3]-[5]. Similar results were also independently proposed by other researchers [6]-[8]. In these papers, image reconstruction was reduced to the inversion of THT via the projection onto convex set (POCS) method, which is iterativ e and computationally expensive. To over above limitations, TSVD as a regularization method has been proposed for inversing the THT, which runs two orders of magnitude faster than the iterative method [9]. However the regularization parame ter in this method is given directly without theoretical analysis. The selection of the regularization parameter that controls the trade-off between fidelity to the data and constraint on the reconstruction result is a primary problem in the regularization method. In this paper, we present a method of selecting the optimal regularization based on the L-curve for TSVD method for solving the interior tomography based on truncated Hilbert transform. In this method, the optimal regularization parameter is computed theoretically using the maximum curvature of the L-curve. Moreover, we develop a Tikhonov regularization method in comparison to TSVD. Root Mean Sq uare Error (RMSE) of reconstr ucted image with respect to different regularization parameters validates our selecting regularization parameter method. The paper is organized as follows: section 2 gives Mathematical formulation, TSVD, Tikhonov regularization, the L-curve. Section 3 exhibits the simulation results. Section 4 is conclusion.

Determination of a molecular weight distribution from equilibrium sedimentation has been significantly improved by incorporating linear programming into Tikhonov's regularization method. The conventional way of computation failed because the Fujita equations which relate the distribution function f(m) to the experimental data are improperly posed problems in the Hadamard sense. Their inverse solution necessary for determination of a molecular weight distribution is unstable. As a result such a solution leads to an erratic curve instead of the desired f(m). This erratic behavior is significantly improved by applying Tikhonov's regularization method. However, a certain amount of oscillation in the f(m) curve still remains uncorrected. Linear programming, which was originally introduced to eliminate undesired negative f(m) values, also served as an additional smoothing factor and improved the calculated f(m) curve. Good results were obtained in cases of a unimodal, a bimodal, and a trimodal distribution.

The availability and quality of experimental data pose challenges for identifying the role of individual proteins in signal transduction pathways. To address this issue, this paper formulates and solves an inverse problem to determine the dynamics of transcription factors, i.e., one important class of proteins, from fluorescence intensity measurements of green fluorescent protein (GFP) reporter systems. In the presented approach, a model describing transcription and translation of GFP is discretized and concentrations of transcription factor are estimated at discrete time points. Unlike previous works, this approach has no restrictions with regard to a particular shape of the profiles. However, the resulting inverse problem is ill-conditioned and requires the use of regularization techniques. Two regularization methods - truncated singular value decomposition and Tikhonov regularization - are investigated in this work and the characteristics of the results obtained are discussed in detail.

In this paper, the distribution of relaxation times (DRTs) functions are calculated numerically in Matlab for synthetic impedance data from single parallel RC circuit and two parallel RC circuits connected in series, experimental impedance data from supercapacitors and α-LiFeO2 anode based Li ion batteries. The quality of the impedance data is checked with the Kramers–Krönig (KK) relations. The DRTs are calculated within the KK compatible regime for all the systems using Tikhonov regularization (TR) method. Here we use a fast and simple L-curve method to estimate the TR parameter (λ) for regularization of the Fredholm integral equations of first kind in impedance. Estimation of the regularization parameters are performed effectively from the offset of the global corner of the L-curve rather than simply using the global corner. The physical significances of DRT peaks are also discussed by calculating the effective resistances and capacitances coupled with peak fitting program. For instance, two peaks in the DRTs justify the electrical double layer capacitance and ion diffusion phenomena for supercapacitors in low to intermediate frequencies respectively. Moreover, the surface film effect, Li/electrolyte and electrode/electrolyte charge transfer related processes are identified for α-LiFeO2 anode based Li-ion batteries. This estimation of the offset of the global corner extends the L-curve approach coupled with the Tikhonov regularization in the field of electrochemistry and can also be applied in similar process detection methods.

This paper presents a stable method for the identification of sources located on the separation interface of two homogeneous media (where one of them is contained by the other one), from measurement yielded by those sources on the exterior boundary of the media. This is an ill-posed problem because numerical instability is presented, i.e., minimal errors in the measurement can result in significant changes in the solution. To obtain the proposed stable method the identification problem is categorized into three subproblems, two of which present numerical instability and regularization methods must be applied to obtain their solution in a stable form. To manage the numerical instability due to the ill-posedness of these subproblems, the Tikhonov regularization and sequential smoothing methods are used. We illustrate this methodology in a circular and irregular region to demonstrate the feasibility of the proposed method, which yields convergent and stable solutions for input data with and without noise.

Assessment of Tikhonov-type regularization methods for solving atmospheric inverse problems

The demand for healthcare facilities in the world is growing daily, and this growth in demand exceeds general population growth. This situation has made hospital trips a vital issue during urban transportation planning. In this context, the aim of this study is twofold: (1) to investigate the hospital trip behavior as home-based hospital trips within the scope of trip production and attraction (trip generation) models and (2) to evaluate the predictive power of some robust and regularization methods in trip generation modeling. Within this scope, approximately 49,000 valid Household Survey Data (HSD) from 2001 (base year) and 2015 (target year) were used to develop comprehensive trip generation models for the home-based hospital trips. Multiple linear regression (MLR), ridge regression (RR), Liu, least trimmed squares (LTS), least trimmed squares–ridge (LTS–R), and least trimmed square–Liu (LTS–L) methods were used for the analysis, and results were evaluated in terms of specialized mean squared error (MSE) and root mean square error (RMSE). As a result, LTS–R and LTS–L increased future predictive power by approximately 55% compared with OLS. This study fills an important gap in the literature in terms of not only the use of regularization and robust methods in trip demand modeling but also examining characteristics that affect hospital trip behaviors. In addition, the paper proposes a “Hospital Mobility Coefficient” projection that could be useful in demand estimation for future hospital trips.

We propose Absum, which is a regularization method for improving adversarial robustness of convolutional neural networks (CNNs). Although CNNs can accurately recognize images, recent studies have shown that the convolution operations in CNNs commonly have structural sensitivity to specific noise composed of Fourier basis functions. By exploiting this sensitivity, they proposed a simple black-box adversarial attack: Single Fourier attack. To reduce structural sensitivity, we can use regularization of convolution filter weights since the sensitivity of linear transform can be assessed by the norm of the weights. However, standard regularization methods can prevent minimization of the loss function because they impose a tight constraint for obtaining high robustness. To solve this problem, Absum imposes a loose constraint; it penalizes the absolute values of the summation of the parameters in the convolution layers. Absum can improve robustness against single Fourier attack while being as simple and efficient as standard regularization methods (e.g., weight decay and L1 regularization). Our experiments demonstrate that Absum improves robustness against single Fourier attack more than standard regularization methods. Furthermore, we reveal that robust CNNs with Absum are more robust against transferred attacks due to decreasing the common sensitivity and against high-frequency noise than standard regularization methods. We also reveal that Absum can improve robustness against gradient-based attacks (projected gradient descent) when used with adversarial training.

Solutions to Linear Inverse Problems on the Sphere by Tikhonov Regularization, Wiener filtering and Spectral Smoothing and Combination — A ComparisonSolutions to linear inverse problems on the sphere, common in geodesy and geophysics, are compared for Tikhonov's method of regularization, Wiener filtering and spectral smoothing and combination as well as harmonic analysis. It is concluded that Wiener and spectral smoothing, although based on different assumptions and target functions, yield the same estimator. Also, provided that the extra information on the signal and error degree variances is available, the standard Tikhonov method is inferior to the other methods, which, in contrast to Tikhonov's approach, match the spectral errors and signals in an optimum way. We show that the corresponding Tikhonov matrix for optimum regularization can only be determined approximately. Moreover, as Tikhonov's method solves an integral equation, it is less computationally efficient than the other methods, which use forward integration. Also harmonic analysis uses direct integration and is not hampered, as previous methods, with spectral leakage. Spectral combination, in addition to filtering, has the advantage of combining different data sets by least squares spectral weighting.

As one of the indispensable components in convolutional neural network (CNN), loss function assists in updating parameters of CNN models during the training phase. Generally, different loss functions can assist convolutional neural network (CNN) to learn different feature representations, and different feature representations can be treated as different knowledge learned from objects. In this paper we introduce a novel training framework, namely Loss Transferring (LT), to improve the generalization ability of CNN. LT contains multiple training phases, and each training phase uses a different loss function. Under this framework, CNN models can combine different knowledge of objects by transferring the knowledge learned via one loss function to another. LT contains two components, i.e., loss function set and training strategy. In order to build appropriate loss function set, we establish two basic guides. And according to these basic guides, we design a new loss function in the last layer of CNN models (layer before softmax operation), namely Near Classifier Hyper-Plane (N-CHP) loss, which makes the learned object features belonging to the same category have the minimum intra-class distance and be near the classifier hyper-plane. Based on the two loss function set $\{\text {MSE, softmax}\}$ and $\{\text {N-CHP, softmax}\}$ , we setup two specific training methods, $LT_{\text {MSE, softmax}}$ and $LT_{\text {N-CHP, softmax}}$ , which can be universally applied to different CNN models with low additional computation cost. Meanwhile, two training strategies, multi-phase strategy 1 and multi-phase strategy 2, are further proposed to improve the training efficiency of LT. Extensive experimental results on shallow, moderate and deep models with four benchmark datasets, including MNIST, SVHN, CIFAR-10 and CIFAR-100, demonstrate that CNN models can bring obvious performance improvements when working with $LT_{\text {MSE, softmax}}$ and $LT_{\text {N-CHP, softmax}}$ , which verifies the effectiveness of LT and the proposed two basic guides.

NUMBER of methods have been developed for damage detection in the last two decades. Many of the methods are based on the observation of dynamic behavior of a structure (Doebling et al. [1], Brincker et al. [2], Maeck and DeRoeck [3]), among which the sensitivity approach via a model-updating technique is commonly accepted and applied extensively in the engineering industry because of its clear mathematical background andquantitativeindications.However,thistypeofmethodisweakat accommodating the influence of measurement errors, leading to illconditioned problems, as demonstrated by Friswell et al. [4] and Humar et al. [5]. This shortcoming means that the existence and uniqueness of the solution is not ensured and numerical instability is likely to take place in the course of the solution process [6,7]. Investigations have since been conducted to deal with the illconditioned problems in model updating. Hansen [8,9] and Vogel [10] proposed regularization methods for obtaining a solution of the inverse problem. It is recognized in regularization theory that the conventional output error, which is usually the vector of differences between thecomputedandmeasuredresponses,canbemadeunrealistically small if the process of damage identification is allowed to behave badly, such that the variable has arbitrarily large deviations from the true set of parameter change, or there may be infinite sets of ill-posed solutions. A stable solution scheme can be achieved by imposing certain constraints in the form of added penalty terms with adjustable weighting parameters based on posterior knowledge. Recently,TiturusandFriswell[11]presentedthesensitivity-based model-updating method with an additional regularization criterion and computed the solutions based on the generalized singular value decomposition.Specificfeaturesoftheparameterandresponsepaths were explored when the regularization parameter varies. The four different types of spaces that arise in the solution were discussed, together with the characteristics of the families of the regularized solution. Weber et al. [12] applied the Tikhonov regularization and truncated singular value decomposition consistently to a nonlinear updating problem. Line-search and stopping criteria known from numericaloptimizationwereadaptedtotheregularizedproblem.The optimal regularization parameter was determined by generalized cross-validation. From experiences gained in model updating with laboratory test structures, the authors found that Tikhonov regularization can give the optimal solution when there is no noise or very small noise in the measurement. But in the updating procedure for a nonlinear inverse problem with the inclusion of noise and model error, the signal-tonoise ratio is getting smaller [13,14] with each iteration, and the solution obtained from a poor regularization parameter is usually unacceptable. The algorithm does not accurately converge, and the results depend strongly on the convergence criteria and tolerances. In this paper, two techniques are proposed in a new regularization method for the identification of local damages in a structure. One technique proposed a new side condition that classified the elements aspossiblydamagedandundamagedelements,whichwillbetreated differently later. The other technique restricts the range of variation of the regularization parameter, such that the regularization solution will be in a realistic range, and the correct optimal point from the curvature of the L-curve is consistently chosen to ensure a continuousconvergingprocess.Bothtechniquesmakefulluseofthe information from results obtained in previous iteration steps. A plane frame structure is studied with two damaged elements and different levels of noise and model errors to illustrate the application of the proposed method. Numerical results show that the proposed consistent regularization method is very effective at improving the results in the inverse problem with ill-conditioning, compared with the conventional Tikhonov regularization.