Efficient Randomized Algorithms for the Repeated Median Line Estimator

Dimension reduction procedures have received increasing consideration over the past decades. Despite this attention, the effect of data contamination or outlying data points in dimension reduction is, however, not well understood, and is compounded by the issue that outliers can be difficult to classify in the presence of many variables. This paper formally investigates the influence of data contamination for sliced inverse regression (SIR), which is a prototypical dimension reduction procedure that targets a lower-dimensional subspace of a set of regressors needed to explain a response variable. We establish a general theory for how estimated reduction subspaces can be distorted through both the number and direction of outlying data points. The results depend critically on the regressor covariance structure and the most harmful types of data contamination are shown to differ in cases where this covariance structure is known or unknown. For example, if the covariance structure is estimated, data contamination is proven to produce an estimated subspace that is automatically orthogonal to the directions of outlying data points, constituting a potentially serious loss of information. Our main results demonstrate the degree to which data contamination indeed causes incorrect dimension reduction, depending on the amount, magnitude, and direction of contamination. Further, by metricizing distances between dimension reduction subspaces, worst case results for data contamination can be formulated to define a finite sample breakdown point for SIR as a measure of global robustness. Our theoretical findings are illustrated through simulation.

Finding a new drug which is both safe and efficient is an expensive and time-consuming endeavour. In particular, establishing the ‘dose-effect relationship’ – how beneficial a drug is at different dosages – can be challenging. Predicting this curve requires gathering experimental data by exposing and recording how cells respond to various levels of the drug. However, extreme values are often observed at low and high dosages, potentially introducing errors that are hard to correct in the prediction process. Yet, these extreme observations are sometimes genuine so researchers cannot just ignore them. To improve dose-effect estimation, Zhou, Liu, Fang et al. developed a new general-purpose approach. It uses advanced statistical modelling to account for extremes in lab data. This strategy outperformed other methods when dealing with these observations while also providing higher efficiency in data analysis with more uniform data in experiments. To facilitate implementation, Zhou, Liu, Fang et al. set up a user-friendly tool baptized ‘REAP’; this free online resource allows scientists without advanced statistical experience to harness the new approach and to perform dose-effect analysis more easily and accurately. This could boost research across many different disciplines that examine the effects of chemicals on cells.

Nowadays, high-dimensional data appear in many practical applications such as biosciences. In the regression analysis literature, the well-known ordinary least-squares estimation may be misleading when the full ranking of the design matrix is missed. As a popular issue, outliers may corrupt normal distribution of the residuals. Thus, since not being sensitive to the outlying data points, robust estimators are frequently applied in confrontation with the issue. Ill-conditioning in high-dimensional data is another common problem in modern regression analysis under which applying the least-squares estimator is hardly possible. So, it is necessary to deal with estimation methods to tackle these problems. As known, a successful approach for high-dimension cases is the penalized scheme with the aim of obtaining a subset of effective explanatory variables that predict the response as the best, while setting the other parameters to zero. Here, we develop several penalized mixed-integer nonlinear programming models to be used in high-dimension regression analysis. The given matrix approximations have simple structures, decreasing computational cost of the models. Moreover, the models are effectively solvable by metaheuristic algorithms. Numerical tests are made to shed light on performance of the proposed methods on simulated and real world high-dimensional data sets.

Robust covariance estimates based on resampling

We explore why many recently proposed robust estimation problems are efficiently solvable, even though the underlying optimization problems are non-convex. We study the loss landscape of these robust estimation problems, and identify the existence of ’generalized quasi-gradients’. Whenever these quasi-gradients exist, a large family of no-regret algorithms are guaranteed to approximate the global minimum; this includes the commonly used filtering algorithm. For robust mean estimation of distributions under bounded covariance, we show that any first-order stationary point of the associated optimization problem is an approximate global minimum if and only if the corruption level $\epsilon < 1/3$. Consequently, any optimization algorithm that approaches a stationary point yields an efficient robust estimator with breakdown point $1/3$. With carefully designed initialization and step size, we improve this to $1/2$, which is optimal. For other tasks, including linear regression and joint mean and covariance estimation, the loss landscape is more rugged: there are stationary points arbitrarily far from the global minimum. Nevertheless, we show that generalized quasi-gradients exist and construct efficient algorithms. These algorithms are simpler than previous ones in the literature, and for linear regression we improve the estimation error from $O(\sqrt{\epsilon })$ to the optimal rate of $O(\epsilon )$ for small $\epsilon $ assuming certified hypercontractivity. For mean estimation with near-identity covariance, we show that a simple gradient descent algorithm achieves breakdown point $1/3$ and iteration complexity $\tilde{O}(d/\epsilon ^2)$.

Extreme data points, or outliers, can have a disproportionate influence on the conclusions drawn from a set of bivariate correlational data. This paper addresses two aspects of outlier detection. The results of a survey regarding how published researchers prefer to deal with outliers are presented, and a set of 183 test validity studies is examined to document the effects of different approaches to the detection and exclusion of outliers on effect size measures. The study indicates that: (a) there is disagreement among researchers as to the appropriateness of deleting data points from a study; (b) researchers report greater use of visual examination of data than of numeric diagnostic techniques for detecting outliers; and (c) while outlier removal influenced effect size measures in individual studies, outlying data points were not found to be a substantial source of variance in a large test validity data set.

What mathematicians, scientists, engineers, and statisticians mean by “inverse problem” differs. For a statistician, an inverse problem is an inference or estimation problem. The data are finite in number and contain errors, as they do in classical estimation or inference problems, and the unknown typically is infinite-dimensional, as it is in nonparametric regression. The additional complication in an inverse problem is that the data are only indirectly related to the unknown. Standard statistical concepts, questions, and considerations such as bias, variance, mean-squared error, identifiability, consistency, efficiency, and various forms of optimality apply to inverse problems. This article discusses inverse problems as statistical estimation and inference problems, and points to the literature for a variety of techniques and results.

The measurement of power spectra is a problem of steadily increasing importance which appears to some to be primarily a problem in statistical estimation. Others may see it as a problem of instrumentation, recording and analysis which vitally involves the ideas of transmission theory. Actually, ideas and techniques from both fields are needed. When they are combined, they provide a basis for developing the insight necessary (i) to plan both the acquisition of adequate data and sound procedures for its reduction to meaningful estimates and (ii) to interpret these estimates correctly and usefully. This account attempts to provide and relate the necessary ideas and techniques in reasonable detail. Part II of this article wilt appear in the March issue of THE JOURNAL.

The overwhelming need to improve the quality of complex data structures in healthcare is more important than ever. Although data quality has been the point of interest in many studies, none of them has focused on the development of quantitative and explainable methods for data imputation. In this work, we propose a "smart" imputation workflow to address missing data across complex data structures in the context of in silico clinical trials. AI algorithms were utilized to produce high-quality virtual patient profiles. A search algorithm was then developed to extract the best virtual patient profiles through the definition of a profile matching score (PMS). A case study was conducted, where the real dataset was randomly contaminated with multiple missing values (e.g., 10 to 50%). In total, 10000 virtual patient profiles with less than 0.02 Kullback-Leibler (KL) divergence were produced to estimate the PMS distribution. The best generator achieved the lowest average squared absolute difference (0.4) and average correlation difference (0.02) with the real dataset highlighting its increased effectiveness for data imputation across complex clinical data structures.

Smart Home Systems (SHSs) automation platforms are now enabling users to automate the control of their SHS devices by installing autonomous third-party applications (called SmartApps). However, intentional/unintentional issues would make SmartApps deviate from their expected behavior, putting the SHS owner’s security at risk. To address this issue, in this paper, we introduce an ML behavior-based approach to prevent malicious control of SHS devices by misbehaved SmartApps. To do so, control commands issued by the SmartApps during regular operation are captured to build a One-Class Support Vector Machine (OCSVM) model as a baseline for each installed SmartApp. Then, anomalous commands (i.e., outlier data points) should be detected and rejected, while normal commands (i.e., inlier data points) are allowed to be executed. Through an experimental evaluation conducted on an adapted SHS automation history, our proposed approach exhibits low false acceptance and rejection rates.

Continuing bank failures point to the need for early warning problem bank identification models to guide the actions of regulators and investors. Several models have been shown to work well, but most require extensive data preparation/manipulation and custom computer programmes to analyse the data, which hinders widespread adoption. In this article, we show that robust Locally Weighted Scatter Plot Smooth, a type of Local Regression Smoothing, which requires minimal data preparation and can be run in many off the shelf statistical packages such as SAS and SPSS, can be just as effective as an early warning system.

Visual performance in patients with bilateral central field loss is related to fixation stability. This study evaluated the repeatability of visual-fixation parameters in patients with bilateral central field loss, measured with the MP-1 microperimeter for fixation durations on the order of 15 to 30 seconds. Bivariate contour ellipse area (BCEA), and the eccentricity and meridian of the preferred retinal locus (PRL) were determined in 56 eyes of 56 patients, sampled at two investigational sites. Repeatability and agreement were assessed by estimating 95% limits of agreement for each parameter from two fixation examinations conducted on the same day. The 95% confidence intervals (CI) for log BCEA and for PRL eccentricity and meridian were ±0.67 log deg2, ±2.0°, and ±65.9°, respectively. Each CI decreased substantially if a small number of outlying data points were excluded. For all parameters, the mean difference between the two fixation examinations was close to zero. For most patients with bilateral central field loss, the repeatability of estimated BCEA and PRL eccentricity and meridian is good. When repeated estimates of fixation parameters do not agree, the absence of a well-developed PRL or the use of multiple PRLs may be suspected.

The paper presents our experience with identifying and verifying outlying data points. Firstly we recall the grand tour method implemented in a dynamic graphics environment and endowed with dynamically changing concentration ellipses and count plots — as proposed by Bartkowiak & Szustalewicz (1997). The method permits to select and identify some data points as suspected outliers. Next we propose to carry out a sort of classification of the found outliers by performing cluster analysis based on angular similarities of the suspected outliers. The procedure returns bundles of data vectors similar with respect to their outlyingness. The considerations are illustrated with the Milk container data, analyzed formerly, a.o. by Atkinson (1994) and Muruzabal and Munoz (1997).

Terrestrial LiDAR provides many disciplines with an effective and efficient means of producing realistic three-dimensional models of real world objects. With the advent of mobile terrestrial LiDAR, this ability has been expanded to include the rapid collection of three-dimensional models of large urban scenes. For all its usefulness, it does have drawbacks. One of the major problems faced by the LiDAR industry today is the automatic removal of outlying data points from LiDAR point clouds. This paper discusses the development and combined implementation of two methods of performing outlier detection in georeferenced point clouds. These methods made use of the raw data available from most time-of-flight mobile terrestrial LiDAR scanners in both the temporal and spatial domains. The first method involved a moving fixed interval smoother derived from the well-known position velocity acceleration Kalman Filter. The second method fitted a quadratic curved surface to sections of LiDAR data. The combined use of these routines is discussed through examples with real LiDAR data.

This paper describes important aspects of preliminary data analysis to be taken after data are checked for clerical entry errors and before the primary statistical analysis is performed. These include description and graphic display of each variable, recoding categorical data, transforming continuous data into another continuous variable and recoding continuous to categorical data. Missing values and outlying data points are identified and several techniques are recommended to minimize mistakes in variable recoding. Related variables measured with different units may be combined by using the z transformation and converted back to one of the original units for ease of interpretation. Finally, both categorical and continuous variables are checked for reliability by using kappa or the intraclass R.