Monitoring Robust Estimates for Compositional Data

Finding a new drug which is both safe and efficient is an expensive and time-consuming endeavour. In particular, establishing the ‘dose-effect relationship’ – how beneficial a drug is at different dosages – can be challenging. Predicting this curve requires gathering experimental data by exposing and recording how cells respond to various levels of the drug. However, extreme values are often observed at low and high dosages, potentially introducing errors that are hard to correct in the prediction process. Yet, these extreme observations are sometimes genuine so researchers cannot just ignore them. To improve dose-effect estimation, Zhou, Liu, Fang et al. developed a new general-purpose approach. It uses advanced statistical modelling to account for extremes in lab data. This strategy outperformed other methods when dealing with these observations while also providing higher efficiency in data analysis with more uniform data in experiments. To facilitate implementation, Zhou, Liu, Fang et al. set up a user-friendly tool baptized ‘REAP’; this free online resource allows scientists without advanced statistical experience to harness the new approach and to perform dose-effect analysis more easily and accurately. This could boost research across many different disciplines that examine the effects of chemicals on cells.

SUMMARY The solution of the remote reference method, a frequently used technique in magnetotelluric (MT) data processing, can be viewed as a product of the two-input–multiple-output relationship between the local electromagnetic field and the reference field at a remote station. By applying a robust estimator to the two-input–multiple-output system, one can suppress the influence of outliers in the local magnetic field as well as those in the local electric field based on regression residuals. Therefore, this study develops a new robust remote reference estimator with the aid of robust multivariate linear regression. By applying the robust multivariate regression S-estimator to the multiple-output system, the present work derives a set of equations for robust estimates of the transfer function, noise variances, and the scale of the Mahalanobis distance simultaneously. The noise variances are necessary for the multivariate analysis to normalize the residuals of dependent variables. The Mahalanobis distance, a distance measure for multivariate data, is a commonly used indicator of outliers in multivariate statistics. By updating those robust estimates iteratively, the new robust remote reference estimator seeks the transfer function that minimizes the robust scale estimate of the Mahalanobis distance. The developed estimator can avoid bias in the MT transfer function even if there are significant noises in the reference magnetic field and handle outlying data more robustly than previously proposed robust remote reference estimators. The authors applied the developed method to a synthetic data set and real-world data. The test results demonstrate that the developed method downweights outliers in the local electric and magnetic fields and gives an unbiased transfer function.

SummaryMapReduce has become a popular model for large‐scale data processing in recent years. Many works on MapReduce scheduling (e.g., load balancing and deadline‐aware scheduling) have emphasized the importance of predicting workload received by individual reducers. However, because the input characteristics and user‐specified map function of a given job are unknown to the MapReduce framework before the job starts, accurately predicting workload of reducers can be a difficult challenge. To address this challenge, we present ROUTE, a run‐time robust reducer workload estimation technique for MapReduce. ROUTE progressively samples the partition size of the early completed mappers, allowing ROUTE to perform estimation at run time yet fulfilling the accuracy requirement specified by users. Moreover, by using robust estimation and bootstrapping resampling techniques, ROUTE can achieve high applicability to a wide variety of applications. Through experiments using both real and synthetic data on an 11‐node Hadoop cluster, we show ROUTE can achieve high accuracy with error rate no more than 10.92% and an improvement of 40.6% in terms of error rate while compared with the state‐of‐the‐art solution. Besides, through simulations using synthetic data, we show that ROUTE is robust to a variety of skewed distributions. Finally, we apply ROUTE to existing load balancing and deadline‐aware scheduling frameworks and show ROUTE significantly improves the performance of these frameworks. Copyright © 2016 John Wiley & Sons, Ltd.

Detecting influential observations in principal components and common principal components

One of the most robust empirical findings in political science is that in multiparty democracies cabinet ministries are distributed in rough proportion to parties’ legislative seat shares, a pattern known as Gamson’s Law. Yet existing research often overlooks the fact that portfolio and seat shares are compositional—mutually dependent parts of a whole. Standard methods treat them as unconstrained, risking bias, misleading uncertainty estimates, and flawed inference. Unfortunately, the most common strategy for handling compositions—the additive log-ratio (ALR) transformation combined with seemingly unrelated regression (SUR)—fails when the number and identity of compositional elements (like parties) vary across cases. We propose the isometric log-ratio (ILR) transformation—new to political science—as an alternative that both respects compositional geometry and adapts to differing compositional structures. Monte Carlo simulations show that ILR sharply outperforms standard approaches, reducing bias, improving coverage probability, and increasing statistical power. While we apply it to portfolio allocation, ILR provides a general solution for modeling compositional outcomes with other potential uses, including in electoral competition, where ALR+SUR has required strong assumptions or ad hoc adjustments. Using this improved methodology, we find that seat–portfolio proportionality is weaker overall than conventionally reported and varies substantially across governments.

There is a wide range of outliers in spatial data, and these potential outliers will have a great impact on parameter estimation and corresponding statistical inference. Relying on the framework of maximum likelihood estimation (MLE), we investigate the asymptotic distribution of robust ML estimator under the mixed spatial autoregressive models with outliers and compare it with that of the ML estimator. Furthermore, based on the asymptotic theoretical result, we conduct the confidence interval of robust MLE and MLE. Similar to the results of MLE, we construct the second-order-corrected robust confidence interval using the parametric and semi-parametric bootstrap method. Simulation studies using Monte Carlo show that the robust estimator with the Huber loss function is more accurate and outperforms the MLE in most sample settings when data is contaminated by outliers. Then the use of the method is demonstrated in the analysis of the Neighborhood Crimes Data and the Boston Housing Price Data. The results further support the eligibility of the robust method in practical situations.

Today's power system predominantly employing the weighted least squares (WLS) technique for estimating the state of the power system over robust state estimation techniques. The computational inexpensiveness of WLS made it a good choice for the operators compared to the robust state estimators. When the system is completely observable by PMUs, the relationship between measurements and state variables changes from non-linear to linear. This made robust state estimators computationally competitive with the WLS estimator (WLSE). In this paper, quadratic-constant (QC) robust estimator has been improved to make it suitable for linear state estimation. The constant part of the estimator after the break-even point is changed to decaying exponential. The proposed quadratically- decaying exponential criterion (QE) is tested on IEEE 14, 30, 57 and 118- bus systems for two cases. (a) measurement data having only Gaussian noise (b) with corrupt measurement data. It is demonstrated that the proposed estimator is able to suppress the bad data and exhibiting better estimation accuracy and computational performance compared to least absolute value (LAV) and WLSE.

Anomaly Detection for Compositional Data using VSI MEWMA control chart

Prediction of true classes of surficial and deep earth materials using multivariate spatial data is a common challenge for geoscience modelers. Most geological processes leave a footprint that can be explored by geochemical data analysis. These footprints are normally complex statistical and spatial patterns buried deep in the high-dimensional compositional space. This paper proposes a spatial predictive model for classification of surficial and deep earth materials derived from the geochemical composition of surface regolith. The model is based on a combination of geostatistical simulation and machine learning approaches. A random forest predictive model is trained, and features are ranked based on their contribution to the predictive model. To generate potential and uncertainty maps, compositional data are simulated at unsampled locations via a chain of transformations (isometric log-ratio transformation followed by the flow anamorphosis) and geostatistical simulation. The simulated results are subsequently back-transformed to the original compositional space. The trained predictive model is used to estimate the probability of classes for simulated compositions. The proposed approach is illustrated through two case studies. In the first case study, the major crustal blocks of the Australian continent are predicted from the surface regolith geochemistry of the National Geochemical Survey of Australia project. The aim of the second case study is to discover the superficial deposits (peat) from the regional-scale soil geochemical data of the Tellus Project. The accuracy of the results in these two case studies confirms the usefulness of the proposed method for geological class prediction and geological process discovery.

Linear discriminant analysis, such as Fisher's criterion, is a statistical learning tool traditionally devoted to separating a training dataset into two or even several classes by the way of linear decision boundaries. In this paper, we show that this tool can formalize the robust linear regression problem as a robust estimator will do. More precisely, we develop a one-class Fischer's criterion in which the maximization provides both the regression parameters and the separation of the data in two classes: typical data and atypical data or outliers. This new criterion is built on the statistical properties of the subspace decomposition of the hat matrix. From this angle, we improve the discriminative properties of the hat matrix which is traditionally used as outlier diagnostic measure in linear regression. Naturally, we call this new approach discriminative hat matrix. The proposed algorithm is fully nonsupervised and needs only the initialization of one parameter. Synthetic and real datasets are used to study the performance both in terms of regression and classification of the proposed approach. We also illustrate its potential application to image recognition and fundamental matrix estimation in computer vision.

Thetdistribution is a useful extension of the normal distribution, which can be used for statistical modeling of data sets with heavy tails, and provides robust estimation. In this paper, in view of the advantages of Bayesian analysis, we propose a new robust coefficient estimation and variable selection method based on Bayesian adaptive Lassotregression. A Gibbs sampler is developed based on the Bayesian hierarchical model framework, where we treat thetdistribution as a mixture of normal and gamma distributions and put different penalization parameters for different regression coefficients. We also consider the Bayesiantregression with adaptive group Lasso and obtain the Gibbs sampler from the posterior distributions. Both simulation studies and real data example show that our method performs well compared with other existing methods when the error distribution has heavy tails and/or outliers.

Many signal and image processing applications, including texture analysis, radar detection or EEG signal classification, require the computation of a centroid from a set of covariance matrices. The most popular approach consists in considering the center of mass. While efficient, this estimator is not robust to outliers arising from the inherent variability of the data or from faulty measurements. To overcome this, some authors have proposed to use the median as a more robust estimator. Here, we propose an estimator which takes advantage of both efficiency and robustness by combining the concepts of Riemannian center of mass and median. Based on the theory of M-estimators, this robust centroid estimator is issued from the so-called Huber's function. We present a gradient descent algorithm to estimate it. In addition, an experiment on both simulated and real data is carried out to evaluate the influence of outliers on the estimation and classification performances.

The analysis of the gut microbiome with respect to health care prevention and diagnostic purposes is increasingly the focus of current research. We analyzed around 2000 stool samples from the KORA (Cooperative Health Research in the Region of Augsburg) cohort using high-throughput 16S rRNA gene amplicon sequencing representing a total microbial diversity of 2089 operational taxonomic units (OTUs). We evaluated the combination of three different components to assess the reflection of obesity related to microbiota profiles: (i) four prediction methods (i.e., partial least squares (PLS), support vector machine regression (SVMReg), random forest (RF), and M5Rules); (ii) five OTU data transformation approaches (i.e., no transformation, relative abundance without and with log-transformation, as well as centered and isometric log-ratio transformations); and (iii) predictions from nine measurements of obesity (i.e., body mass index, three measures of body shape, and five measures of body composition). Our results showed a substantial impact of all three components. The applications of SVMReg and PLS in combination with logarithmic data transformations resulted in considerably predictive models for waist circumference-related endpoints. These combinations were at best able to explain almost 40% of the variance in obesity measurements based on stool microbiota data (i.e., OTUs) only. A reduced loss in predictive performance was seen after sex-stratification in waist–height ratio compared to other waist-related measurements. Moreover, our analysis showed that the contribution of OTUs less prevalent and abundant is minor concerning the predictive power of our models.

Joint simulation of compositional and categorical data via direct sampling technique – Application to improve mineral resource confidence

The estimation of β plays a basic role in the evaluation of expected return and market risk. Typically this is performed by ordinary least squares (OLS). To cope with the high sensitivity of OLS to outlying observations and to deviations from the normality assumptions, several methods suggest to use robust estimators. It is argued that, from a predictive point of view, the simple use of either OLS or robust estimators is not sufficient but that some shrinking of the robust estimators toward OLS is necessary to reduce the mean squared error. The performance of the proposed shrinkage robust estimator is shown by means of a small simulation study and on a real data set.