Multivariate Plug-in Bandwidth for Local Linear Regression

Crown profile of loblolly pine (Pinus taeda L.) was modeled by nonparametric regression analysis. Nonparametric regression may be applicable when an underlying parametric model cannot be identified and uses only the data to fit a curve. A class of local-polynomial estimators which contains the popular kernel estimator as a special case was investigated. Kernel regression appears to fit closely to the interior data points but often possesses bias problems at the boundaries of the data, a feature less exhibited by local linear or local quadratic regression. A selection of trees was used to show that nonparametric regression captures more variation in crown shape than multiple linear regression. When using nonparametric regression, decisions must be made regarding polynomial order and bandwidth. Such decisions depend on the presence of local curvature, desired degree of smoothing, and for bandwidth in particular, the minimization of some global error criterion. In the present study, a penalized PRESS criterion (PRESS*) was selected as the global error criterion. For. Sci. 44(3):445-453.

Double-smoothing for bias reduction in local linear regression

This thesis proposes a novel unified boosting framework. We apply this framework to the several face processing tasks, face detection, facial feature localisation, and pose classification, and use the same boosting algorithm and the same pool of features (local binary features). This is in contrast with the standard approaches that make use of a variety of features and models, for example AdaBoost, cascades of boosted classifiers and Active Appearance Models. The unified boosting framework covers multivariate classification and regression problems and it is achieved by interpreting boosting as optimization in the functional space of the weak learners. Thus a wide range of smooth loss functions can be optimized with the same algorithm. There are two general optimization strategies we propose that extend recent works on TaylorBoost and Variational AdaBoost. The first proposition is an empirical expectation formulation that minimizes the average loss and the second is a variational formulation that includes an additional penalty for large variations between predictions. These two boosting formulations are used to train real-time models using local binary features. This is achieved using look-up-tables as weak learners and multi-block Local Binary Patterns as features. The resulting boosting algorithms are simple, efficient and easily scalable with the available resources. Furthermore, we introduce a novel coarse-to-fine feature selection method to handle high resolution models and a bootstrapping algorithm to sample representative training data from very large pools of data. The proposed approach is evaluated for several face processing tasks. These tasks include frontal face detection (binary classification), facial feature localization (multivariate regression) and pose estimation (multivariate classification). Several studies are performed to assess different optimization algorithms, bootstrapping parametrizations and feature sharing methods (for the multivariate case). The results show good performance for all of these tasks. In addition to this, two other contributions are presented. First, we propose a context-based model for removing the false alarms generated by a given generic face detector. Second, we propose a new face detector that predicts the Jaccard distance between the current location and the ground truth. This allows us to formulate the face detection problem as a regression task.

State-of-the-art data analysis tools have to deal with high-dimensional data. Fortunately, the inherent dimensionality of data is often much smaller, as it has an internal structure limiting its degrees of freedom. In most cases, this structure can be approximated using a graph, i.e, a set of nodes connected by edges. Based on this idea, graphs have been largely used in semi-supervised and unsupervised learning. The canonical assumption when incorporating a graph prior is to assume that the signal is smooth with respect to the graph, i.e, that connected pieces of data have similar values. Nevertheless, this hypothesis is insufficient to characterize more complex relations between the data and its structure, the graph. To this end, the field of Graph Signal Processing (GSP) extends the graph smoothness hypothesis to a more general assumption. Its key idea is to think of the signal as a sum of harmonic modes, which are obtained from the eigen-decomposition of the Laplacian, an operator representing the second order derivative. Thus, the essence of GSP resides in the Fourier transform defined by the projection on the Laplacian eigenvectors basis, which allows us to filter graph signals. However, GSP suffers from a counterintuitive irregularity. Contrarily to the classical Fourier basis, the graph Laplacian eigenvectors are not spreading uniformly on the vertex set and can be highly concentrated in a region of the graph. This non-uniformity results in a lack of any intuitive translation operator and consequently poses challenges to generalize some of the classical signal processing theory, e.g., stationarity, uncertainty principles, or sampling. In this thesis, we answer these difficulties by accepting that each node is unique and should be treated specially according to its surrounding structure. This is achieved using an operator called localization that shifts a kernel defined in the spectral domain around a specific node while adapting to the local graph structure. In the first part of the thesis, we focus on harmonic analysis in GSP. We start by making use of the norm of localized kernels to capture the spectral local features of the graph vertices. To illustrate the relevance of this analysis we then introduce structural clustering, an algorithm that groups nodes according to the role they play in the graph. Then, we bound the vertex-frequency concentration of graph filter banks atoms. Because of the irregular structure of graphs, we construct a local uncertainty principle that is, again, driven by the norm of a localized kernel. In the second part of the thesis, we tackle GSP problems from a machine learning perspective. We use the localization operator to extend the notion of stationarity to graph signals. It consists of a graph signal probabilistic framework allowing us to optimally solve inverse problems. Finally, we show how the graph total variation can be used in semi-supervised learning and prove its consistency with respect to the underlying graph manifold.

To achieve adaptive prediction and error correction of tobacco moisture content after rotary drying in cigarette production, hierarchical cluster analysis was employed to prune the drying parameters of cut tobacco. Leveraging the data characteristics observed during the drying process, we constructed a multivariate linear regression model and a transfer function model based on principal component analysis. For seamless implementation in a computer program, both forecasting models were discretized. The prediction error was corrected online using the moving average method and the second-order exponential smoothing method to enhance the models’ adaptive capability and prediction accuracy. Results from the production environment indicate that while the multivariate linear regression model can generally capture the overall trend of data changes, it falls short in accurately predicting local and real-time moisture variations. In contrast, the transfer function model, enhanced with second-order exponential smoothing, improved the root mean square error (RMSE) and the coefficient of determination (R2) to 0.052 and 0.779, respectively. This adaptive approach effectively accommodates the dynamic characteristics of the rotary drying process, captures random fluctuations in moisture, and delivers precise moisture predictions.

A novel approach for real-time monitoring of long-term voltage stability using local linear regression and adaptive training database is proposed in this paper. Comparing to previous methods, the proposed local predictive regression model could properly balance the simplicity/transparency and accuracy, and it is also adaptive to the changes of system and operating conditions. The approach is tested on IEEE 30-bus system. Preliminary results show that it can be successfully employed in online voltage stability monitoring.

The quality of object proposal plays an important role in boosting the performance of many computer vision tasks, such as, object detection and recognition. Due to the absence of manually annotated bounding-box in practice, the quality metric towards blind assessment of object proposal is highly desirable for singling out the optimal proposals. In this paper, we propose a blind proposal quality assessment algorithm based on the Deep Objectness Representation and Local Linear Regression (DORLLR). Inspired by the hierarchy model of the human vision system, a deep convolutional neural network is developed to extract the objectness-aware image feature. Then, the local linear regression method is utilized to map the image feature to a quality score, which tries to evaluate each individual test window based on its k-nearest-neighbors. Experimental results on a large-scale IoU labeled dataset verify that the proposed method significantly outperforms the state-of-the-art blind proposal evaluation metrics.

Multivariate regressive deep stochastic artificial learning for energy and cost efficient 6G communication

Let where f is an unknown regression function, (ξ1,...,ξn) are iid centered gaussian variables independent of the design (X 1,……X n), Consider the problem of estimating the level set from (X 1, Y 1),….. (X n, Y n)Consider under certain assumptions on the boundary smoothness of G. We propose piecewise-polynomial estimators based on the maximization of local empirical excess masses. With assumptions on the design we show that these estimators have optimal rates of convergence in an asymptotically minimax meaning, within studied classes of regressions. For “bad” design we obtain other, non-optimal, rates. We generalize these results to the N-dimensional case, N ≠ 2.

In theory, recent results in nonparametric regression show that neural network estimates are able to achieve good rates of convergence provided suitable assumptions on the structure of the regression function are imposed. However, these theoretical analyses cannot explain the practical success of neural networks since the theoretically studied estimates are defined by minimizing the empirical L_2 risk over a class of neural networks and in practice, solving this kind of minimization problem is not feasible. Consequently, the neural networks examined in theory cannot be implemented as they are defined. This means that neural network in applications differ from the ones that are analyzed theoretically. In this thesis we narrow the gap between theory and practice. We deal with neural network regression estimates for (p,C)-smooth regression functions m that satisfy a projection pursuit model. We construct three implementable neural network estimates and show that each of them achieve up to a logarithmic factor the optimal univariate rate of convergence. Firstly, for univariate regression functions with p contained in [-1/2,1] we construct a neural network estimate with one hidden layer where the weights are learned via gradient descent. The starting weights are randomly chosen from an interval independently of the data. The interval is large enough to guarantee that the estimate is close to a piecewise constant approximation. Secondly, for multivariate regression functions with p contained in (0,1] we construct a neural network estimate with one hidden layer where the weights are learned via gradient descent. The initial weights are chosen from specific intervals dependently on the data and the projection directions. This choice guarantees that the estimate is close to a piecewise constant approximation. The projection directions are repeatedly chosen randomly. Lastly, for multivariate regression functions with p>0 we construct a multilayer neural network estimate. The value of the inner weights are prescribed dependently on the projection directions by a new approximation result for a projection pursuit model by piecewise polynomials. The outer weights are chosen by solving a linear equation system. The projection directions are repeatedly chosen randomly. Since we are able to show a rate of convergence that is independent of the dimension of the data our second and third estimates are able to circumvent the curse of dimensionality.

To learn things incrementally without the catastrophic interference, we have proposed Resource Allocating Network with Long-Term Memory (RAN-LTM). In RAN-LTM, not only training data but also memory items stored in long-term memory are trained. In this paper, we propose an extended RAN-LTM called Resource Allocating Network by Local Linear Regression (RAN-LLR), in which its centers are not trained but selected based on output errors and the connections are updated by solving a linear regression problem. To reduce the computation and memory costs, the modified connections are restricted based on RBF activity. In the experiments, we first apply RAN-LLR to a one-dimensional function approximation problem to see how the negative interference is effectively suppressed. Then, the performance of RAN-LLR is evaluated for a real-world prediction problem. The experimental results demonstrate that the proposed RAN-LLR can learn fast and accurately with less memory costs compared with the conventional models.

In the case of the random design nonparametric regression, locally weighted regression (LWR) in Cleveland is considered. Under some regularity conditions, we show that the regression function estimator derived from LWR has the same asymptotic variance quantity as the k-nearest neighbor (k-NN) smoother. But, it has better asymptotic bias quality than the k-NN smoother. By these results, this regression function estimator has two advantages. One is that it is better than the k-NN smoother in a minimax sense. The other is that it does not suffer from boundary effects when boundary points of the support of the design density are known in advance. The motivation that the regression function estimator derived from LWR has nice asymptotic bias quality and boundary properties comes from Fan and Gijbels and Fan Furthermore, estimators of derivatives of the regression function derived from LWR are considered.

Increased accuracy of geostatistical prediction of nitrogen dioxide in the United Kingdom with secondary data

In this paper, we present a novel nonparametric algorithm for short term electricity demand forecasting. The algorithm is based on local linear regression using sliding window with variable length. The method for selecting optimal window length for each local fit offers close insight into trade-off between bias and standard deviation of local regressions. Optimal window length is selected for each value in the load time-series: large window for linear change of load to reduce variability and small window when load departs from linear function to control bias. In the presented algorithm local linear regression is used to estimate trend component of the load time series and to forecast trend component by extrapolating with the fitted local linear function. Some features of the algorithm are demonstrated in the paper using examples from the historic load data recorded in the Namibian Power Utility.

Local linear regression models, a kind of nonparametric structures that locally perform a linear estimation of the target function, are analyzed in the context of empirical risk minimization (ERM) for function learning. The analysis is carried out with emphasis on geometric properties of the available data. In particular, the discrepancy of the observation points used both to build the local regression models and compute the empirical risk is considered. This allows to treat indifferently the case in which the samples come from a random external source and the one in which the input space can be freely explored. Both consistency of the ERM procedure and approximating capabilities of the estimator are analyzed, proving conditions to ensure convergence. Since the theoretical analysis shows that the estimation improves as the discrepancy of the observation points becomes smaller, low-discrepancy sequences, a family of sampling methods commonly employed for efficient numerical integration, are also analyzed. Simulation results involving two different examples of function learning are provided.