An Additive Global and Local Gaussian Process model for large data sets

Optimization of complex real-time control systems often requires efficient response to any system changes over time. By combining pattern search optimization with a fast estimated Gaussian Process model, we are able to perform global optimization more efficiently for response surfaces with multiple local minimums or even dramatic changes over the design space. Our approach extends pattern search for global optimization problems by incorporating the global and local information provided by an additive global and local Gaussian Process model. We further develop a global search method to identify multiple promising local regions for parallel implementation of local pattern search. We demonstrate our methods on a standard test problem.

This paper deals with a nonparametric identification of continuous-time nonlinear systems using multiple local Gaussian process (GP) models. Multiple sets of training input and output data are collected to train the local GP prior models. Each local GP prior model is trained by minimizing the negative log marginal likelihood of each set of the training data. The final nonlinear function with confidence measure is estimated by weighted mean of the local estimated nonlinear functions using the predictive variances of local GP posterior distributions. Compared to the standard GP-based identification method, the proposed method can reduce the computational cost and improve the accuracy of identification. Simulation results are shown to illustrate the effectiveness of the proposed identification method.

Taking another step: A simple approach to high-dimensional Bayesian optimization

Gaussian process (GP) regression models make for powerful predictors in out of sample exercises, but cubic runtimes for dense matrix decompositions severely limit the size of data - training and testing - on which they can be deployed. That means that in computer experiment, spatial/geo-physical, and machine learning contexts, GPs no longer enjoy privileged status as data sets continue to balloon in size. We discuss an implementation of local approximate Gaussian process models, in the laGP package for R, that offers a particular sparse-matrix remedy uniquely positioned to leverage modern parallel computing architectures. The laGP approach can be seen as an update on the spatial statistical method of local kriging neighborhoods. We briefly review the method, and provide extensive illustrations of the features in the package through worked-code examples. The appendix covers custom building options for symmetric multi-processor and graphical processing units, and built-in wrapper routines that automate distribution over a simple network of workstations.

The mechanical design of a modern robot system is becoming more and more complex as robots are often required to do physical interactions with an unforeseen changing environment. To deal with challenging contexts in the field of robotics, capability to learn a model of kinematics and dynamics is becoming mandatory for a robot. In this paper, we proposed a novel on-line learning algorithm based on a mixture of local sparse Gaussian process. By segmenting a data domain into smaller regions and handling each region with an independent local Gaussian process model, the algorithm achieved linear computational cost with comparable accuracy. The algorithm was validated in experiments with real robot data. In the experiments, the algorithm achieved comparable performance to existing on-line regression approaches.

High performance and compliant robot control requires accurate dynamics models which cannot be obtained analytically for sufficiently complex robot systems. In such cases, machine learning offers a promising alternative for approximating the robot dynamics using measured data. This approach offers a natural framework to incorporate unknown nonlinearities as well as to continually adapt online for changes in the robot dynamics. However, the most accurate regression methods, e.g. Gaussian processes regression (GPR) and support vector regression (SVR), suffer from exceptional high computational complexity which prevents their usage for large numbers of samples or online learning to date. Inspired by locally linear regression techniques, we propose an approximation to the standard GPR using local Gaussian processes models inspired by. Due to reduced computational cost, local Gaussian processes (LGP) can be applied for larger sample-sizes and online learning. Comparisons with other nonparametric regressions, e.g. standard GPR, nu-SVR and locally weighted projection regression (LWPR), show that LGP has higher accuracy than LWPR and close to the performance of standard GPR and nu-SVR while being sufficiently fast for online learning.

Depth sensor based 3D human motion estimation hardware such as Kinect has made interactive applications more popular recently. However, it is still challenging to accurately recognize postures from a single depth camera due to the inherently noisy data derived from depth images and self-occluding action performed by the user. In this paper, we propose a new real-time probabilistic framework to enhance the accuracy of live captured postures that belong to one of the action classes in the database. We adopt the Gaussian Process model as a prior to leverage the position data obtained from Kinect and marker-based motion capture system. We also incorporate a temporal consistency term into the optimization framework to constrain the velocity variations between successive frames. To ensure that the reconstructed posture resembles the accurate parts of the observed posture, we embed a set of joint reliability measurements into the optimization framework. A major drawback of Gaussian Process is its cubic learning complexity when dealing with a large database due to the inverse of a covariance matrix. To solve the problem, we propose a new method based on a local mixture of Gaussian Processes, in which Gaussian Processes are defined in local regions of the state space. Due to the significantly decreased sample size in each local Gaussian Process, the learning time is greatly reduced. At the same time, the prediction speed is enhanced as the weighted mean prediction for a given sample is determined by the nearby local models only. Our system also allows incrementally updating a specific local Gaussian Process in real time, which enhances the likelihood of adapting to run-time postures that are different from those in the database. Experimental results demonstrate that our system can generate high quality postures even under severe self-occlusion situations, which is beneficial for real-time applications such as motion-based gaming and sport training.

Traffic speed is a key indicator for the efficiency of an urban transportation system. Accurate modeling of the spatiotemporally varying traffic speed thus plays a crucial role in urban planning and development. This paper addresses the problem of efficient fine-grained traffic speed prediction using big traffic data obtained from static sensors. Gaussian processes (GPs) have been previously used to model various traffic phenomena, including flow and speed. However, GPs do not scale with big traffic data due to their cubic time complexity. In this work, we address their efficiency issues by proposing local GPs to learn from and make predictions for correlated subsets of data. The main idea is to quickly group speed variables in both spatial and temporal dimensions into a finite number of clusters, so that future and unobserved traffic speed queries can be heuristically mapped to one of such clusters. A local GP corresponding to that cluster can then be trained on the fly to make predictions in real-time. We call this method localization. We use non-negative matrix factorization for localization and propose simple heuristics for cluster mapping. We additionally leverage on the expressiveness of GP kernel functions to model road network topology and incorporate side information. Extensive experiments using real-world traffic data collected in the two U.S. cities of Pittsburgh and Washington, D.C., show that our proposed local GPs significantly improve both runtime performances and prediction accuracies compared to the baseline global and local GPs.

When powerful numerical tools like the finite element method encounter their limits for the evaluation of physical systems it is very common to use surrogate models as an approximation. There are many possible choices concerning the model approach, among which Gaussian process models are the most popular ones due to their clear statistical basis. A very desirable attribute of such surrogates is a high flexibility for making them applicable to a great class of underlying problems while obtaining interpretable results. To achieve this Gaussian processes are used as basis functions of an additive model in this work. Another important property of a surrogate is stability, which can be especially challenging when it comes to the estimation of the correlation parameters. To solve this we use a Bayesian approach where a reference prior is assigned to each component of the additive model assuring robust correlation matrices. Due to the additive structure of the model a simplified parameter estimation process is proposed that reduces the usually high-dimensional optimization problem to a few sub-routines of low dimension. Finally, we demonstrate this concept by modeling the magnetic field of a magnetic linear position detection system.

Device sizing is a challenging problem for analog circuit design. Traditional methods depend on domain knowledge and intensive simulations to search for feasible parameters. Recent studies apply the Bayesian optimization (BO) and a Gaussian process (GP) model in analog circuit synthesis to improve efficiency. The BO framework automatically selects the parameter candidates by inferring the surrogate GP model. However, naive BO employs a sequential updating strategy which is inefficient in a multicore environment. Besides, the widely used GP model requires costly high fidelity data, which are obtained from fine simulations. In this article, we propose a constrained batch BO approach with a multifidelity (MF) model to solve the above difficulties. The batch BO exploits parallel computing and selects promising parameters by multiple acquisition function ensemble. In addition, the MF GP model adapts the low fidelity data obtained from coarse simulations. Specifically, the proposed method incorporates information gain in a weighted clustering algorithm to refine the parameter candidates. As a result, the proposed method maintains the candidates’ quality and diversity, which speeds up the optimization convergence. In the experiments, we demonstrate the efficiency of the proposed approach on three real-world circuits. The results show that our approach reduces the simulation costs by at least 54.6% compared to the state-of-the-art baselines.

Sequential pattern mining and temporal association rule mining techniques are established techniques to mine insights into the data collected by smart home environments and ambient living systems. Local process models are a recent technique that uses constructs from the field of business process modeling to represent frequent patterns that go beyond sequential patterns and can additionally express rich ordering relations that include concurrent execution, choices, and repetition. To gain insight into the behavior within smart home environments using local process models, it is vital to focus on patterns where the activities that are described by the pattern are truly related. We show on real-life smart home data that existing techniques for mining local process models fail to do so, and address this problem by proposing novel techniques for constraint-based local process model mining, focusing on event gap constraints and time gap constraints. We observe that in order to get an accurate support count of a local process model, it is sufficient to only consider specific parts of the datasets instead of the full dataset, which allows to speed up the counting of the support of a pattern. We provide a novel algorithm to extract those relevant parts of the data for support counting. We evaluate our approach on a collection of real-life smart home data sets. We show that more insightful local process models can be mined when applying such constraints, and show the novel algorithm allows to mine them efficiently.

From the viewpoint of psychology, it is thought that perception analysis of the visual world includes two information processes: global (whole) and local (part) processes. It is assumed that the global process is carried out in the right hemisphere, and the local process, in the left hemisphere. In the present study, gamma EEG band activities during location memory (LM) task, as a global form, and shape memory (SM) task, as a local form, were calculated from the temporal, parietal and occipital areas using stimuli consisting of categorical patterns of small shapes. Gamma band activity during the SM task was greater than that during the LM task. It was assumed that the SM task requires a higher memory load condition than the LM task. In terms of the laterality ratio obtained from the whole electrode array, the gamma band was significantly activated in the right hemisphere during the LM task, and in the left hemisphere during the SM task. The gamma activation in the occipital area was significantly high in the right hemisphere for both tasks. High gamma band activation was observed in the right parietal area during the LM task and in the left temporal area during the SM task. It was concluded that global and local information processes occur in the left temporal areas and in the right occipitoparietal areas, respectively. The results of this study are useful in the assessment of visual cognition deficits in patients with cerebral hemispheric lesions in the physical therapy.

Data-driven uncertainty quantification in computational human head models

The Gaussian process (GP) model provides a powerful methodology for calibrating a computer model in the presence of model uncertainties. However, if the data contain systematic experimental errors, then the GP model can lead to an unnecessarily complex adjustment of the computer model. In this work, we introduce an adjustment procedure that brings the computer model closer to the data by making minimal changes to it. This is achieved by applying a lasso-based variable selection on the systematic experimental error terms while fitting the GP model. Two real examples and simulations are presented to demonstrate the advantages of the proposed approach. This article has supplementary material available online.

Current process discovery techniques are unable to produce meaningful models for semi-structured processes, as they are either too inaccurate or too complex. In this paper we use the idea of local process models (LPMs) to model fragments of a semi-structured process and explore the potential of sets of LPMs. Automatic LPM discovery finds many small patterns but doesn’t find patterns larger than 4–5 events, it produces too many models, and the discovered models describe some events from the log multiple times while leaving others unexplained. We manually construct a set of LPMs for the well-known BPIC12 event log that (1) contains a small number of models that (2) have high accuracy measures such as fitness and precision while (3) they together cover the whole event log and (4) do not cover parts of the log multiple times unnecessarily. We find that existing evaluation techniques for LPMs do not work for sets of LPMs and we propose several measures that help determine the quality of a set of LPMs both as a whole and as individual LPMs. We show that sets of LPMs can indeed be used to model semi-structured processes by not thinking of such processes as monolithic, but rather a collection of smaller processes working together.