Least quantile regression via modern optimization

Say that a regression method is "unstable" at a data set if a small change in the data can cause a relatively large change in the fitted plane. A well-known example of this is the instability of least squares regression (LS) near (multi)collinear data sets. It is known that least absolute deviation (LAD) and least median of squares (LMS) linear regression can exhibit instability at data sets that are far from collinear. Clear-cut instability occurs at a "singularity"--a data set, arbitrarily small changes to which can substantially change the fit. For example, the collinear data sets are the singularities of LS. One way to measure the extent of instability of a regression method is to measure the size of its "singular set" (set of singularities). The dimension of the singular set is a tractable measure of its size that can be estimated without distributional assumptions or asymptotics. By applying a general theorem on the dimension of singular sets, we find that the singular sets of LAD and LMS are at least as large as that of LS and often much larger. Thus, prima facie, LAD and LMS are frequently unstable. This casts doubt on the trustworthiness of LAD and LMS as exploratory regression tools.

Robust Regression for the Detection of Independent 3D Motion by a Binocular Observer

In exciting recent work, Bertsimas, King and Mazumder (Ann. Statist. 44 (2016) 813–852) showed that the classical best subset selection problem in regression modeling can be formulated as a mixed integer optimization (MIO) problem. Using recent advances in MIO algorithms, they demonstrated that best subset selection can now be solved at much larger problem sizes than what was thought possible in the statistics community. They presented empirical comparisons of best subset with other popular variable selection procedures, in particular, the lasso and forward stepwise selection. Surprisingly (to us), their simulations suggested that best subset consistently outperformed both methods in terms of prediction accuracy. Here, we present an expanded set of simulations to shed more light on these comparisons. The summary is roughly as follows: •neither best subset nor the lasso uniformly dominate the other, with best subset generally performing better in very high signal-to-noise (SNR) ratio regimes, and the lasso better in low SNR regimes; •for a large proportion of the settings considered, best subset and forward stepwise perform similarly, but in certain cases in the high SNR regime, best subset performs better; •forward stepwise and best subsets tend to yield sparser models (when tuned on a validation set), especially in the high SNR regime; •the relaxed lasso (actually, a simplified version of the original relaxed estimator defined in Meinshausen (Comput. Statist. Data Anal. 52 (2007) 374–393)) is the overall winner, performing just about as well as the lasso in low SNR scenarios, and nearly as well as best subset in high SNR scenarios.

Network Topology Error Identification in Robust State Estimation for Power Systems

This paper considers the robustness properties in the time series context of the least median of squares (LMS) estimator. The influence function of the LMS estimator is derived under additive outlier contamination. This influence function is redescending and bounded for fixed values of the AR parameters. The gross-error sensitivity, however, is an unbounded function of the AR parameters. In order to asses the global robustness behavior of the LMS estimator, we consider several notions of breakdown. The breakdown points of the LMS estimator depend on the value of the underlying AR parameter. Generally, the breakdown point is below one half for high values of the AR parameter. The bias curves of the LMS estimator reveal, however, that the magnitude of outliers has to be considerable in order to cause breakdown.

Outlier detection is an important task for fitting a model to a set of data. Two different outlier detection approaches are given as tests for outliers and robust methods. For these approaches, usually outliers are considered as additive bias terms neglected in the original adjustment model. However, there is another approach that outlier is considered as a model error in the Gauss–Markov model. This model error is represented as an unknown parameter. As it cannot be known before which observation includes outlier; this method is applied on the data for each observation separately and tested with t-test or F-test. It is successful if the sample includes only one outlier. To detect multiple outliers more successfully, in this article, a new outlier detection method is introduced. In this method, all the possible combinations of multiple outliers are considered as model errors and it is accepted that the smallest variance of them gives the solution for a certain number of outliers, then the estimated model errors are detected by comparing with a critical value. The critical value is chosen as 3σo. To compare the results of the new method, with those of the Least Median of Squares (LMS) and Huber M-estimators, Monte Carlo simulation technique is used for linear regression. The Mean Success Rate is proposed to measure the reliabilities of the methods. We showed that the new method is robust and includes the property of high breakdown point as LMS; and more efficient than LMS.

In this paper, the computation of the least median of squares (LMS) estimator for regression and the minimum volume ellipsoid estimator for multivariate location and covariance are considered, using the P3R and PAR programs from the widely available statistical package BMDP. Multivariate outliers are detected using robust distances based on the MVE estimators of the explanatory variables. The identification of leverage points in linear regression is based on the combined use of the LMS standardized residuals and the robust distances of the explanatory regression variables. FORTRAN procedures and the required BMDP input to produce the corresponding robust estimates are presented. The Hawkins-Bradu-Kass data is used as a test example.

Finding and fitting circles from a set of points is a frequent problem in the data analysis of high-energy physics experiments. In a tracker immersed in a homogeneous magnetic field, tracks are close to perfect circles if projected to the bending plane. In a ring-imaging Cherenkov (RICH) detector, circles of photons around the crossing point of charged particles have to be found and their radii estimated. In both cases, non-negligible background may be present that tends to complicate the pattern recognition and to bias the circle fit. In this contribution we present a robust circle fit based on a modified Riemann fit that removes or significantly reduces the effect of background points. As in the standard Riemann fit, the measured points are projected to the Riemann sphere or paraboloid, and a plane is fitted to the projected points. The fit is made robust by replacing the usual least-squares regression by a least median of squares (LMS) regression. Because of the high breakdown point of the LMS estimator, the fit is insensitive to background points. The LMS plane is used to initialize the weights of an M-estimator that refits the plane in order to suppress eventual remaining outliers and to obtain the final circle parameters. The method is demonstrated on three sets of artificial data: points on a circle plus a comparable number of background points; points on two overlapping circles with additional background; and points obtained by the simulation of tracks in a drift chamber with mirror points and additional background. The results show high circle finding efficiency and small contamination of the final fitted circles.

Natural mortality rate (M) and fishing catchability coefficient (q) were estimated by using the Paloheimo pairwise linear regression model with simulated and real catch–effort data. The four estimation methods used were least squares (LS), least absolute value (LAV), least median of squares (LMS), and LMS-based reweighted least squares (RLS). When catch or effort data or both contained homogeneous error variances, mean squared errors (MSE) of LS and RLS were significantly smaller than those of LAV and LMS. However, there was no significant difference in MSE between LS and RLS. When catch–effort data contained variances inconsistent among years, MSEs of the robust methods were much smaller than those of LS. Reweighted least squares substantially reduced the variances in the LS-estimated q and M, and it had the smallest MSE among the four estimation methods. We suggest using the following procedures in estimating q and M based on the Paloheimo pairwise regression model: (1) applying LMS to the model to identify outliers in the data; and (2) weighting the defined outliers by a factor of 0 and normal data points by a factor of 1 and applying LS to the model to estimate q and M and their associated variances.

We propose a new method to fill missing or invalid values in depth images generated from the Kinect depth sensor. To fill the missing depth values, we use a robust least median of squares (LMedS) approach. We apply our method for telepresence environments, where Kinects are used very often for reconstructing the captured scene in 3D. We introduce a modified 1D LMedS approach for efficient traversal of consecutive image frames. Our approach solves the unstable nature of depth values in static scenes that is perceived as flickering. We obtain very good result both for static and moving objects inside a scene.

Least trimmed squares (LTS) provides a parametric family of high breakdown estimators in regression with better asymptotic properties than least median of squares (LMS) estimators. We adapt the forward search algorithm of Atkinson (1994) to LTS and provide methods for determining the amount of data to be trimmed. We examine the efficiency of different trimming proportions by simulation and demonstrate the increasing efficiency of parameter estimation as larger proportions of data are fitted using the LTS criterion. Some standard data examples are analysed. One shows that LTS provides more stable solutions than LMS.

This paper proposes a novel approach to co-register InSAR image pair. We use the SIFT descriptor to extract scale- and rotation-invariant point correspondences from images. Since the images are acquired spatial or temporal differently, there are unavoidable differences between them, which result in some mismatches in the correspondences. Generally, these mismatches must be removed so as to achieve a correct co-registration. However, this step is not necessary in our approach. Base on the obtained correspondences and the geometric transform relationship between images, a series of equations are formulated to calculate the samples of co-registration parameters: the rotation, scale, azimuth translation and range translation. We transform the parameter estimation problem into a linear regression problem, and the robust Least Median of Squares (LMedS) is introduced to obtain the precise value of co-registration parameters from these contaminated samples. Experiments show that the approach is able to co-register images with high precision and robustness even if substantial nonoverlapped areas exist between images.

Algorithms such as Least Median of Squares (LMedS) and Random Sample Consensus (RANSAC) have been very successful for low-dimensional robust regression problems. However, the combinatorial nature of these algorithms makes them practically unusable for high-dimensional applications. In this paper, we introduce algorithms that have cubic time complexity in the dimension of the problem, which make them computationally efficient for high-dimensional problems. We formulate the robust regression problem by projecting the dependent variable onto the null space of the independent variables which receives significant contributions only from the outliers. We then identify the outliers using sparse representation/learning based algorithms. Under certain conditions, that follow from the theory of sparse representation, these polynomial algorithms can accurately solve the robust regression problem which is, in general, a combinatorial problem. We present experimental results that demonstrate the efficacy of the proposed algorithms. We also analyze the intrinsic parameter space of robust regression and identify an efficient and accurate class of algorithms for different operating conditions. An application to facial age estimation is presented.

Summary In this paper, we present a method for detecting automatically CDP gathers with clear discrepancies between the observations and the model used to define the AVA attributes. The technique is presented using a linear model of AVA; however, any regression model can be used. Bootstrap and least median square (LMS) regression are the main components of the method. An application to data from the Glitne field (North Sea) is presented to illustrate the applicability of the methodology.

<p>Recent investigations have shown it is possible to simultaneously calibrate model structures and model parameters to identify appropriate models for a given task (Spieler et al., 2020). However, this is computationally challenging, as different model structures may use a different number of parameters. While some parameters may be shared between model structures, others might be relevant for only a few structures, which theoretically requires the calibration of conditionally active parameters. Additionally, shared model parameters might cause different effects in different model structures, causing their optimal values to differ across structures. In this study, we tested how two current “of the shelf” mixed-integer optimization algorithms perform when having to handle these peculiarities during the automatic model structure identification (AMSI) process recently introduced by Spieler et al. (2020).</p><p>To validate the current performance of the AMSI approach, we conduct a benchmark experiment with a model space consisting of 6912 different model structures.  First, all model structures are independently calibrated and validated for three hydro-climatically differing catchments using the CMA-ES algorithm and KGE as the objective function. This is referred to as standard calibration procedure. We identify the best performing model structure(s) based on validation performance and analyze the range of performance as well as the number of structures performing in a similar range. Secondly, we run AMSI on all three catchments to automatically identify the most feasible model structure based on the KGE performance. Two different mixed-integer optimization algorithms are used – namely DDS and CMA-ES. Afterwards, we compare the results to the best performing models of the standard calibration of all 6912 model structures.</p><p>Within this experimental setup, we analyze if the best performing model structure(s) AMSI identifies are identical to the best performing structures of the standard calibration and if there are differences in performance when using different optimization algorithms for AMSI. We also validate if AMSI can identify the best performing model structures for a catchment at a fraction of the computational cost than the standard calibration procedure requires by using “off the shelf” mixed-integer optimization algorithms.</p><p> </p><p> </p><p> </p><p>Spieler, D., Mai, J., Craig, J. R., Tolson, B. A., & Schütze, N. (2020). Automatic Model Structure Identification for Conceptual Hydrologic Models. Water Resources Research, 56(9). https://doi.org/10.1029/2019WR027009</p>