Sparse Additive Gaussian Process with Soft Interactions

The assumption of equal variances is not always appropriate and different approaches for modelling variance heterogeneity have been widely studied in the literature. One of these approaches is joint location and scale model defined with the idea that both the location and the scale depend on explanatory variables through parametric linear models. Because the joint location and scale model includes two models, it does not deal well with a large number of irrelevant variables. Therefore, determining the variables that are important for the location and the scale is as important as estimating the parameters of these models. From this point of view, a combine robust estimation and variable selection method is proposed to simultaneously estimate the parameters and select the important variables. This is done using the least favorable distribution and least absolute shrinkage and selection operator method. Under appropriate conditions, we study the consistency, asymptotic distribution and the sparsity property of the proposed robust estimator. Simulation studies and a real data example are provided to demonstrate the advantages of the proposed method over existing methods in literature.

This paper is about optimal estimation of the additive components of a nonparametric, additive isotone regression model. It is shown that asymp- totically up to first order, each additive component can be estimated as well as it could be by a least squares estimator if the other components were known. The algorithm for the calculation of the estimator uses backfitting. Conver- gence of the algorithm is shown. Finite sample properties are also compared through simulation experiments. In this paper we discuss nonparametric additive monotone regression models. We discuss a backfitting estimator that is based on iterative application of the pool adjacent violator algorithm to the additive components of the model. Our main result states the following oracle property. Asymptotically up to first order, each additive component is estimated as well as it would be (by a least squares estimator) if the other components were known. This goes beyond the classical finding that the estimator achieves the same rate of convergence, independently of the number of additive components. The result states that the asymptotic distribution of the estimator does not depend on the number of components. We have two motivations for considering this model. First of all we think that this is a useful model for some applications. For a discussion of isotonic additive regres- sion from a more applied point, see also Bacchetti (1), Morton-Jones et al. (32) and De Boer, Besten and Ter Braak (7). But our main motivation comes from statistical theory. We think that the study of nonparametric models with several nonparamet- ric components is not fully understood. The oracle property that is stated in this paper for additive isotone models has been shown for smoothing estimators in some other nonparametric models. This property is expected to hold if the estimation of the different nonparametric components is based on local smoothing where the lo- calization takes place in different scales. An example are additive models of smooth functions where each localization takes place with respect to another covariate. In Mammen, Linton and Nielsen (28) the oracle property has been verified for the lo- cal linear smooth backfitting estimator. As local linear estimators, also the isotonic least squares is a local smoother. The estimator is a local average of the response variable but in contrast to local linear estimators the local neighborhood is chosen � Research of this paper was supported by the Deutsche Forschungsgemeinschaft project MA

Fractional data that are restricted in the standard unit interval (0, 1) with a highly skewed distribution are commonly encountered. Such data arise in various areas, such as economics, finance, and medicine, among others. One natural idea to model such data is to use the beta family due to its flexibility to accommodate various density shapes. In this paper, we propose a nonparametric additive beta regression model along with a variable selection procedure, where the mean response is related to covariates through the combination of unknown functions of covariates, which can be approximated on a B-spline basis. By using this approximation method, we transform the problem of variable selection into the problem of selecting the groups of coefficients in the expansion. Based on the penalized likelihood method for group variable selection, we successfully select the significant covariates. Moreover, the estimation and selection consistencies and the properties of the penalized estimators are established. The simulation studies demonstrate that the performance of our proposed method is quite good. Finally, we apply the proposed method to body fat data, and we obtain several important findings with satisfactory selection and prediction performance.

Accurate prediction of the breast cancer patient's life expectancy is essential for treatment decisions. This study aims to develop a novel model estimation and variable selection method for the partially linear additive quantile regression model when the survival times are subject to right censoring. Rather than most of the existing methods using the formulation of synthetic data points or weighting schemes to tackle censoring, we use an adapted loss function to solve censoring. Moreover, we adopt the B-spline to approximate the nonparametric additive components. To further improve the prediction accuracy, we use the group smoothly clipped absolute deviation (SCAD) penalty to select significant variables in the nonparametric additive components. To implement the proposed method, we develop an effective block-wise majorize-minimize (MM) algorithm. Furthermore, we establish the asymptotic properties for the resultant estimators. Numerical simulations illustrate that the finite sample performance of the proposed method outperforms alternative methods. Finally, we apply our method for the personalized treatment of female malignant metastatic breast cancer patients, using the Surveillance, Epidemiology, and End Results (SEER) research data.

ABSTRACTRegularization methods for simultaneous variable selection and coefficient estimation have been shown to be effective in quantile regression in improving the prediction accuracy. In this article, we propose the Bayesian bridge for variable selection and coefficient estimation in quantile regression. A simple and efficient Gibbs sampling algorithm was developed for posterior inference using a scale mixture of uniform representation of the Bayesian bridge prior. This is the first work to discuss regularized quantile regression with the bridge penalty. Both simulated and real data examples show that the proposed method often outperforms quantile regression without regularization, lasso quantile regression, and Bayesian lasso quantile regression.

We put forward a variable selection method for selecting explanatory variables in a normality-assumed multivariate linear regression. It is cumbersome to calculate variable selection criteria for all subsets of explanatory variables when the number of explanatory variables is large. Therefore, we propose a fast and consistent variable selection method based on a generalized $C_{p}$ criterion. The consistency of the method is provided by a high-dimensional asymptotic framework such that the sample size and the sum of the dimensions of response vectors and explanatory vectors divided by the sample size tend to infinity and some positive constant which are less than one, respectively. Through numerical simulations, it is shown that the proposed method has a high probability of selecting the true subset of explanatory variables and is fast under a moderate sample size even when the number of dimensions is large.

Variable selection is important in QSAR modeling since it can improve model performance and transparency, as well as reduce the computational cost of model fitting and predictions. Which variable selection methods that perform well in QSAR settings is largely unknown. To address this question we, in a total of 1728 benchmarking experiments, rigorously investigated how eight variable selection methods affect the predictive performance and transparency of random forest models fitted to seven QSAR datasets covering different endpoints, descriptors sets, types of response variables, and number of chemical compounds. The results show that univariate variable selection methods are suboptimal and that the number of variables in the benchmarked datasets can be reduced with about 60 % without significant loss in model performance when using multivariate adaptive regression splines MARS and forward selection.

This study adopts two approaches to analyze the occurrence of algae at Haman Weir for Nakdong River; one is the traditional statistical method, such as logistic regression, while the other is machine learning technique, such as kNN, ANN, RF, Bagging, Boosting, and SVM. In order to compare the performance of the models, this study measured the accuracy, specificity, sensitivity, and AUC, which are representative model evaluation tools. The ROC curve is created by plotting association of sensitivity and (1‐specificity). The AUC that is area of ROC curve represents sensitivity and specificity. This measure has two competitive advantages compared to other evaluation tools. One is that it is scale‐invariant. It means that purpose of AUC is how well the model predicts. The other is that the AUC is classification‐threshold‐invariant. It shows that the AUC is independent of threshold because it is plotted association of sensitivity and (1‐specificity) obtained by threshold. We chose AUC as a final model evaluation tool with two advantages. Also, variable selection was conducted using the Boruta algorithm. In addition, we tried to distinguish the better model by comparing the model with the variable selection method and the model without the variable selection method. As a result of the analysis, Boruta algorithm as a variable selection method suggested PO4‐P, DO, BOD, NH3‐N, Susp, pH, TOC, Temp, TN, and TP as significant explanatory variables. A comparison was made between the model with and without these selected variables. Among the models without variable selection method, the accuracy of RF analysis was highest, and ANN analysis showed the highest AUC. In conclusion, ANN analysis using the variable selection method showed the best performance among the models with and without variable selection method.

We propose and investigate additive density regression, a novel additive functional regression model for situations where the responses are random distributions that can be viewed as random densities and the predictors are vectors. Data in the form of samples of densities or distributions are increasingly encountered in statistical analysis and there is a need for flexible regression models that accommodate random densities as responses. Such models are of special interest for multivariate continuous predictors, where unrestricted nonparametric regression approaches are subject to the curse of dimensionality. Additive models can be expected to maintain one-dimensional rates of convergence while permitting a substantial degree of flexibility. This motivates the development of additive regression models for situations where multivariate continuous predictors are coupled with densities as responses. To overcome the problem that distributions do not form a vector space, we utilize a class of transformations that map densities to unrestricted square integrable functions and then deploy an additive functional regression model to fit the responses in the unrestricted space, finally transforming back to density space. We implement the proposed additive model with an extended version of smooth backfitting and establish the consistency of this approach, including rates of convergence. The proposed method is illustrated with an application to the distributions of baby names in the United States.

Variable selection is fundamental to high dimensional statistical modeling, and many approaches have been proposed. However, existing variable selection methods do not perform well in presence of outliers in response variable or/and covariates. In order to ensure a high probability of correct selection and efficient parameter estimation, we investigate a robust variable selection method based on a modified Huber's function with an exponential squared loss tail. We also prove that the proposed method has oracle properties. Furthermore, we carry out simulation studies to evaluate the performance of the proposed method for both p<n and p>n. Our simulation results indicate that the proposed method is efficient and robust against outliers and heavy-tailed distributions. Finally, a real dataset from an air pollution mortality study is used to illustrate the proposed method.

The Lasso and Ridge estimators are effective variable selection methods for a simple linear model. While the multiple linear model is often used in practice, only few studies have examined its problem of variable selection. To address this gap, we study such problem by taking advantage of the Lasso and Ridge estimate. We propose two variable selection methods in the multiple linear model and introduce the multiple random simulation (MRS) algorithm, whose efficiency is similar to that of the least angle regression (LARS) algorithm in a simple linear model. We verify the performance of these two algorithms by applying them to real diabetes data and find that the LARS algorithm cannot be easily applied in a multiple linear model. We also illustrate the excellent performance of MRS by applying this algorithm on a simulated dataset.

Variable selection in high-dimensional partially linear additive models for composite quantile regression

Partial linear model is very flexible when the relation between the covariates and responses, either parametric and nonparametric. However, estimation of the regression coefficients is challenging since one must also estimate the nonparametric component simultaneously. As a remedy, the differencing approach, to eliminate the nonparametric component and estimate the regression coefficients, can be used. Here, suppose the regression vector-parameter is subjected to lie in a sub-space hypothesis. In situations where the use of difference-based least absolute and shrinkage selection operator (D-LASSO) is desired for, we propose a restricted D-LASSO estimator. To improve its performance, LASSO-type shrinkage estimators are also developed. The relative dominance picture of suggested estimators is investigated. In particular, the suitability of estimating the nonparametric component based on the Speckman approach is explored. A real data example is given to compare the proposed estimators. From the numerical analysis, it is obtained that the partial difference-based shrinkage estimators perform better than the difference-based regression model in average prediction error sense.

Agencies that regulate electricity providers often apply nonparametric data envelopment analysis (DEA) to assess the relative efficiency of each firm. The reliability and validity of DEA are contingent upon selecting relevant input variables. In the era of big (wide) data, the assumptions of traditional variable selection techniques are often violated due to challenges related to high-dimensional data and their standard empirical properties. Currently, regulators have access to a large number of potential input variables. Therefore, our aim is to introduce new machine learning methods for regulators of the energy market. We also propose a new two-step analytical approach where, in the first step, the machine learning-based adaptive least absolute shrinkage and selection operator (ALASSO) is used to select variables and, in the second step, selected variables are used in a DEA model. In contrast to previous research, we find, by using a more realistic data-generating process common for production functions (i.e., Cobb–Douglas and Translog), that the performance of different machine learning techniques differs substantially in different empirically relevant situations. Simulations also reveal that the ALASSO is superior to other machine learning and regression-based methods when the collinearity is low or moderate. However, in situations of multicollinearity, the LASSO approach exhibits the best performance. We also use real data from the Swedish electricity distribution market to illustrate the empirical relevance of selecting the most appropriate variable selection method.

The accuracy of response variables is crucially important to train regression models. In some situations, including the high-dimensional case, response observations tend to be inaccurate, which would lead to biased estimators by directly fitting a conventional model. For analyzing data with anomalous responses in the high-dimensional case, in this work, we adopt γ-divergence to conduct variable selection and estimation methods. The proposed method possesses good robustness to anomalous responses, and the proportion of abnormal data does not need to be modeled. It is implemented by an efficient coordinate descent algorithm. In the setting where the dimensionality p can grow exponentially fast with the sample size n, we rigorously establish variable selection consistency and estimation bounds. Numerical simulations and an application on real data are presented to demonstrate the performance of the proposed method.