Robust Mean and Covariance Structure Analysis Through Iteratively Reweighted Least Squares

A description of a system of subroutines to compute solutions to the iteratively reweighted least squares problem is presented. The weights are determined from the data and linear fit arid are computed as functions of the scaled residuals. Iteratively reweighted least squares is a part of robust statistics where vvrobustnessfl nans relative insensitivity to iroderate departures from assumptions. The software for iteratively reweighted least squares is cast as semi-portable Fortran code whose performance is unaffected (in the sense that performance will not be deaded) by the computer or operating-system environment in which it is used.

Iteratively reweighted least squares (IRLS) method is a popular approach for sparse signal recovery, provably achieves superior performance in a series of applications. Nevertheless, iteratively reweighted least squares involves a matrix inversion at each iteration, which makes it impractical in massive data. To address this issue, we introduced approximate inverse-free accelerated IRLS methods in this paper. Specifically, by adopting the approximate inverse preconditioning with conjugate gradient, we obtain an approximate inverse-free IRLS method. Moreover, a nesterov acceleration scheme and decreased parameters with fixed scale strategy are independently adopted to accelerate the approximate inverse-free IRLS method, which leads to two computationally efficient approximate inverse-free accelerated IRLS algorithms. Experimental results clearly demonstrate the superiority of our proposed approaches, which not only achieve a comparable sample complexity but also have faster convergence with less execution time when compared to traditional fast algorithms.

Measurements from the electrical network are generally transmitted towards the control centres using special communication links. These measurements allow determining the state of the network in real time. However, these measurements often contain uncertainties due to the meter and communication errors, incomplete metering or unavailability of some of these measurements, etc. This paper presents the application of the Particle Swarm Optimization (PSO) algorithm in minimizing the raw measurement errors in order to identify or estimate the optimal operating state of the power system. Two different objective function formulations are assessed by PSO. The first formulation is the Weighted Least Square (WLS) and the second one is the Iteratively Reweighted Least Squares (IRLS) implementation of the Weighted Least Absolute Value (WLAV). Both solutions are compared with a Newton-Raphson (NR) power flow solution using an IEEE 6-bus test system.

In this paper, we present a new method for sparse representation of rational function in Hardy-2 space by using Iteratively Reweighted Least Squares (IRLS) algorithm and shifted Cauchy kernels. This problem originates from identification of transfer functions of FIR models. We also give two examples and MATLAB codes to illustrate the idea.

Basis Pursuit and Basis Pursuit Denoising, well established techniques for computing sparse representations, minimize an {ell}{sup 2} data fidelity term subject to an {ell}{sup 1} sparsity constraint or regularization term on the solution by mapping the problem to a linear or quadratic program. Basis Pursuit Denoising with an {ell}{sup 1} data fidelity term has recently been proposed, also implemented via a mapping to a linear program. They introduce an alternative approach via an iteratively Reweighted Least Squares algorithm, providing greater flexibility in the choice of data fidelity term norm, and computational advantages in certain circumstances.

We present and analyze an efficient implementation of an iteratively reweighted least squares algorithm for recovering a matrix from a small number of linear measurements. The algorithm is designed for the simultaneous promotion of both a minimal nuclear norm and an approximately low-rank solution. Under the assumption that the linear measurements fulfill a suitable generalization of the null space property known in the context of compressed sensing, the algorithm is guaranteed to recover iteratively any matrix with an error of the order of the best $k$-rank approximation. In certain relevant cases, for instance, for the matrix completion problem, our version of this algorithm can take advantage of the Woodbury matrix identity, which allows us to expedite the solution of the least squares problems required at each iteration. We present numerical experiments which confirm the robustness of the algorithm for the solution of matrix completion problems, and we demonstrate its competitiveness with respect to other techniques proposed recently in the literature.

This paper presents a way of using the Iteratively Reweighted Least Squares (IRLS) method to minimize several robust cost functions such as the Huber function, the Cauchy function and others. It is known that IRLS (otherwise known as Weiszfeld) techniques are generally more robust to outliers than the corresponding least squares methods, but the full range of robust M-estimators that are amenable to IRLS has not been investigated. In this paper we address this question and show that IRLS methods can be used to minimize most common robust M-estimators. An exact condition is given and proved for decrease of the cost, from which convergence follows. In addition to the advantage of increased robustness, the proposed algorithm is far simpler than the standard L1 Weiszfeld algorithm. We show the applicability of the proposed algorithm to the rotation averaging, triangulation and point cloud alignment problems.

We study the problem of personalized Quality of Service (QoS) estimation for web services. State-of-the-art methods use matrix factorization or collaborative prediction to estimate web service response times and throughput for each user based on partial measurements collected from past invocations. We point out that in reality, both the response times and through-put of web services follow highly skewed distributions. In this case, the conditional mean QoS estimates generated by traditional matrix completion approaches can be heavily biased toward a few outliers, leading to poor web service recommendation performance. In this paper, we propose the Quantile Matrix Factorization (QMF) technique for web service recommendation by introducing quantile regression into the matrix factorization framework. We propose a novel and efficient algorithm based on Iterative Reweighted Least Squares (IRLS) to solve the QMF problem involving a non-smooth objective function. We further extend the proposed QMF approach to take into account user and service side attributes. Extensive evaluation based on a large-scale QoS dataset has shown that our schemes significantly outperform various state-of-the-art web service QoS estimation schemes in terms of personalized recommendation performance.

The \(\ell _p\) -norm regression problem is a classic problem in optimization with wide ranging applications in machine learning and theoretical computer science. The goal is to compute \(\boldsymbol {\mathit {x}}^{\star } =\arg \min _{\boldsymbol {\mathit {A}}\boldsymbol {\mathit {x}}=\boldsymbol {\mathit {b}}}\Vert \boldsymbol {\mathit {x}}\Vert _p^p\) , where \(\boldsymbol {\mathit {x}}^{\star }\in \mathbb {R}^n,\boldsymbol {\mathit {A}}\in \mathbb {R}^{d\times n},\boldsymbol {\mathit {b}}\in \mathbb {R}^d\) and \(d\le n\) . Efficient high-accuracy algorithms for the problem have been challenging both in theory and practice and the state-of-the-art algorithms require \(poly(p)\cdot n^{\frac{1}{2}-\frac{1}{p}}\) linear system solves for \(p\ge 2\) . In this article, we provide new algorithms for \(\ell _p\) -regression (and a more general formulation of the problem) that obtain a high-accuracy solution in \(O(p n^{ {(p-2)}{(3p-2)}})\) linear system solves. We further propose a new inverse maintenance procedure that speeds-up our algorithm to \(\widetilde{O}(n^{\omega })\) total runtime, where \(O(n^{\omega })\) denotes the running time for multiplying \(n \times n\) matrices. Additionally, we give the first Iteratively Reweighted Least Squares (IRLS) algorithm that is guaranteed to converge to an optimum in a few iterations. Our IRLS algorithm has shown exceptional practical performance, beating the currently available implementations in MATLAB/CVX by 10–50×.

Support vector machine quantile regression approach for functional data: Simulation and application studies

Richard M. Heiberger, Richard A. Becker, Design of an S Function for Robust Regression Using Iteratively Reweighted Least Squares, Journal of Computational and Graphical Statistics, Vol. 1, No. 3 (Sep., 1992), pp. 181-196

A variety of feature selection methods based on sparsity regularization have been developed with different loss functions and sparse regularization functions. Capitalizing on the existing sparsity regularized feature selection methods, we propose a general sparsity feature selection (GSR-FS) algorithm that optimizes a ℓ2,r (0 < r ≤ 2) based loss function with a ℓ2,p-norm (0 < p ≤ 2) sparse regularization. The ℓ2,r-norm (0 <

Robust Principal Component Analysis: An IRLS Approach

The accurate transformation of multi-camera 2D coordinates into 3D coordinates is critical for applications like animation, gaming, and medical rehabilitation. This study unveils an enhanced multi-camera calibration method that alleviates the shortcomings of existing approaches by incorporating a comprehensive cost function and Adaptive Iteratively Reweighted Least Squares (AIRLS) optimization. By integrating static error components (3D coordinate, distance, angle, and reprojection errors) with dynamic wand distance errors, the proposed comprehensive cost function facilitates precise multi-camera parameter calculations. The AIRLS optimization effectively balances the optimization of both static and dynamic error elements, enhancing the calibration’s robustness and efficiency. Comparative validation against advanced multi-camera calibration methods shows this method’s superior accuracy (average error 0.27 ± 0.22 mm) and robustness. Evaluation metrics including average distance error, standard deviation, and range (minimum and maximum) of errors, complemented by statistical analysis using ANOVA and post-hoc tests, underscore its efficacy. The method markedly enhances the accuracy of calculating intrinsic, extrinsic, and distortion parameters, proving highly effective for precise 3D reconstruction in diverse applications. This study represents substantial progression in multi-camera calibration, offering a dependable and efficient solution for intricate calibration challenges.

Covariance structure analysis is used to evaluate hypothesized influences among unmeasured latent and observed variables. As implemented, it is not robust to outliers and bad data. Several robust methods in model fitting and testing are proposed. These include direct estimation of M-estimators of structured parameters and a two-stage procedure based on robust M- and S-estimators of population covariances. The large sample properties of these estimators are obtained. The equivalence between a direct M-estimator and a two-stage estimator based on an M-estimator of population covariance is established when sampling from an elliptical distribution. Two test statistics are presented in judging the adequacy of a hypothesized model; both are asymptotically distribution free if using distribution free weight matrices. So these test statistics possess both finite sample and large sample robustness. The two-stage procedures can be easily adapted into standard software packages by modifying existing asymptotically distribution free procedures. To demonstrate the two-stage procedure, S-estimator and M-estimators under different weight functions are calculated for some real data sets.