Financial Applications of Gaussian Processes and Bayesian Optimization

Gaussian process regression is a well-established Bayesian machine learning method. We propose a new approach to Gaussian process regression using quantum kernels based on parameterized quantum circuits. By employing a hardware-efficient feature map and careful regularization of the Gram matrix, we demonstrate that the variance information of the resulting quantum Gaussian process can be preserved. We also show that quantum Gaussian processes can be used as a surrogate model for Bayesian optimization, a task that critically relies on the variance of the surrogate model. To demonstrate the performance of this quantum Bayesian optimization algorithm, we apply it to the hyperparameter optimization of a machine learning model which performs regression on a real-world dataset. We benchmark the quantum Bayesian optimization against its classical counterpart and show that quantum version can match its performance.

Abstarct Engineering design problems typically require optimizing a quality measure by finding the right combination of controllable input parameters. In Additive Manufacturing (AM), the output characteristics of the process can often be non-stationary functions of the process parameters. Bayesian Optimization (BO) is a methodology to optimize such “black-box” functions, i.e., the input–output relationship is unknown and expensive to compute. Optimization tasks involving “black-box” functions widely use BO with Gaussian Process (GP) regression surrogate model. Using GPs with standard kernels is insufficient for modeling non-stationary functions, while GPs with non-stationary kernels are typically over-parameterized. On the other hand, a Deep Gaussian Process (DGP) can overcome GPs’ shortcomings by considering a composition of multiple GPs. Inference in a DGP is challenging due to its structure resulting in a non-Gaussian posterior, and using DGP as a surrogate model for BO is not straightforward. Stochastic Imputation (SI)-based inference is promising in speed and accuracy for BO. This work proposes a bootstrap aggregation-based procedure to effectively utilize the SI-based inference for BO with a DGP surrogate model. The proposed BO algorithm DGP-SI-BO is faster and empirically better than the state-of-the-art BO method in optimizing non-stationary functions. Several analytical test functions and a case study in metal AM simulation demonstrate the applicability of the proposed method.

Earth observation from satellite sensory data pose challenging problems, where machine learning is currently a key player. In recent years, Gaussian Process (GP) regression and other kernel methods have excelled in biophysical parameter estimation tasks from space. GP regression is based on solid Bayesian statistics, and generally yield efficient and accurate parameter estimates. However, GPs are typically used for inverse modeling based on concurrent observations and in situ measurements only. Very often a forward model encoding the well-understood physical relations is available though. In this work, we review three GP models that respect and learn the physics of the underlying processes in the context of inverse modeling. First, we will introduce a Joint GP (JGP) model that combines in situ measurements and simulated data in a single GP model. Second, we present a latent force model (LFM) for GP modeling that encodes ordinary differential equations to blend data-driven modeling and physical models of the system. The LFM performs multi-output regression, adapts to the signal characteristics, is able to cope with missing data in the time series, and provides explicit latent functions that allow system analysis and evaluation. Finally, we present an Automatic Gaussian Process Emulator (AGAPE) that approximates the forward physical model via interpolation, reducing the number of necessary nodes. Empirical evidence of the performance of these models will be presented through illustrative examples of vegetation monitoring and atmospheric modeling.

With the sustainable development of the construction industry, recycled aggregate (RA) has been widely used in concrete preparation to reduce the environmental impact of construction waste. Compressive strength is an essential measure of the performance of recycled aggregate concrete (RAC). In order to understand the correspondence between relevant factors and the compressive strength of recycled concrete and accurately predict the compressive strength of RAC, this paper establishes a model for predicting the compressive strength of RAC using machine learning and hyperparameter optimization techniques. RAC experimental data from published literature as the dataset, extreme gradient boosting (XGBoost), random forest (RF), K-nearest neighbour (KNN), support vector machine regression Support Vector Regression (SVR), and gradient boosted decision tree (GBDT) RAC compressive strength prediction models were developed. The models were validated and compared using correlation coefficients (R2), Root Mean Square Error (RMSE), mean absolute error (MAE), and the gap between the experimental results of the predicted outcomes. In particular, The effects of different hyperparameter optimization techniques (Grid search, Random search, Bayesian optimization-Tree-structured Parzen Estimator, Bayesian optimization- Gaussian Process Regression) on model prediction efficiency and prediction accuracy were investigated. The results show that the optimal combination of hyperparameters can be searched in the shortest time using the Bayesian optimization algorithm based on TPE (Tree-structured Parzen Estimator); the BO-TPE-GBDT RAC compressive strength prediction model has higher prediction accuracy and generalisation ability. This high-performance compressive strength prediction model provides a basis for RAC’s research and practice and a new way to predict the performance of RAC.

Bayesian optimization (BO) is a versatile and robust global optimization method under uncertainty. However, most of the BO algorithms were developed for problems with only continuous variables. For practical engineering optimization, discrete variables are also prevalent. BO methods based on Gaussian process (GP) surrogates also suffers from the curse-of-dimensionality problem. To address these challenges, in this paper, a Bayesian discrete optimization algorithm is introduced to solve permutation-based combinatorial problems. A new kernel function is developed based on position distances for permutation. To improve the efficiency and scalability of the algorithm, a sparse GP model based on inducing points is further developed, where the simulated annealing algorithm is applied to select inducing points. The new algorithm is demonstrated and tested with the production scheduling problem for additive manufacturing. Experimental results show that the proposed algorithm can find a better solution with limited evaluations than state-of-the-art algorithms.

This paper describes a real-time data collection framework and an adaptive machining learning method for constructing a real-time energy prediction model for a machine tool. To effectively establish the energy consumption pattern of a machine tool over time, the energy prediction model is continuously updated with new measurement data to account for time-varying effects of the machine tool, such as tool wear and machine tool deterioration. In this work, a real-time data collection and processing framework is developed to retrieve raw data from a milling machine tool and its sensors and convert them into relevant input features. The extracted input features are then used to construct the energy prediction model using Gaussian Process (GP) regression. To update the GP regression model with real-time streaming data, we investigate the use of sparse representation of the covariance matrix to reduce the computational and storage demands of the GP regression. We compare computational efficiency of sparse GP to that of full GP regression model and show the effectiveness of the sparse GP regression model for tracking the variation in the energy consumption pattern of the target machine.

Bayesian Optimization for Field-Scale Geological Carbon Storage

Mechanical property prediction plays a crucial role in the steel industry, enabling better materials selection and enhancing production efficiency. In this study, we propose a novel framework that facilitates optimal materials selection for steel’s alloying elements, based on accurate predictions of the Jominy hardness curve. Leveraging Gaussian Process (GP) regression, we provide probabilistic predictions of steel hardenability characteristics from alloy element composition. Taking it a step further, our framework incorporates these accurate predictions into a constrained optimization process, yielding optimal compositions that reduce overall spending while meeting performance specifications. Through data obtained from 1080 steel samples, our GP regression model exhibits high accuracy, achieving an RMSE of 1.37 and showcasing significant improvements in the field. Moreover, our constrained optimization utilizing the GP model and historical market data reveals an average cost reduction of 18% on alloying element expenses, highlighting the tangible cost-saving potential of this approach. By leveraging Gaussian Process (GP) regression, we not only achieve accurate predictions of the Jominy hardness curve based on alloy element composition, but we also introduce a crucial element of uncertainty quantification. This empowers us to place trust in the results of our optimization process, ensuring robust and reliable materials selection. The integration of GP regression and optimization provides a powerful tool for achieving cost-effective materials selection and marks a significant advancement compared to existing studies. This research underscores the promise of machine learning in the steel industry, demonstrating its ability to yield substantial cost savings and enhance decision-making in materials selection.

PHYSBO (optimization tools for PHYSics based on Bayesian Optimization) is a Python library for fast and scalable Bayesian optimization. It has been developed mainly for application in the basic sciences such as physics and materials science. Bayesian optimization is used to select an appropriate input for experiments/simulations from candidate inputs listed in advance in order to obtain better output values with the help of machine learning prediction. PHYSBO can be used to find better solutions for both single and multi-objective optimization problems. At each cycle in the Bayesian optimization, a single proposal or multiple proposals can be obtained for the next experiments/simulations. These proposals can be obtained interactively for use in experiments. PHYSBO is available at https://github.com/issp-center-dev/PHYSBO. Program summaryProgram Title: PHYSBOCPC Library link to program files:https://doi.org/10.17632/22d72yb6k6.1Developer's repository link:https://github.com/issp-center-dev/PHYSBOLicensing provisions: GNU General Public License version 3Programming language: Python3External routines/libraries: NumPy, SciPy, MPI for Python.Nature of problem: Bayesian optimization (BO) can be used to select inputs that will yield better outputs from a list of candidate inputs with the help of machine learning prediction through a Gaussian process. Although BO is a powerful tool, two of its components, training the Gaussian process regression and optimizing the acquisition function, are generally computationally expensive. Moreover, hyperparameter tuning is necessary for the former process.Solution method: PHYSBO is a Python library for performing fast and scalable Bayesian optimization. To avoid the computationally expensive training process, PHYSBO uses a random feature map, Thompson sampling, and a one-rank Cholesky update. In addition, PHYSBO performs hyperparameter tuning automatically by maximizing the Type II likelihood, and MPI parallelization is used to reduce the calculation time for optimizing the acquisition function.

Comprehensive evaluation and systematic comparison of Gaussian process (GP) modelling applications in peptide quantitative structure-activity relationship

Cocrystallization can be used to tune key drug properties, such as aqueous solubility, without altering molecular structure; however, the space of possible coformers is enormous and design rules are empirical. We present a Bayesian optimization framework that couples Gaussian process (GP) classification and regression that can accelerate cocrystal discovery and solubility enhancement. Starting from 6338 literature-derived binary coformer pairs, vector fingerprints that combine 2D structural information (fragment and MQN fingerprints) with low-cost shape and polarity descriptors were engineered for cocrystal prediction. A GP classifier, trained on an actively constructed training set of ∼1000 coformer pairs selected by uncertainty sampling achieves up to 94% accuracy and Matthews correlation coefficient of 0.79 on a test set of >5000 unseen pairs. Property-driven coformer selection was formulated as a Bayesian optimization problem, using a machine learning model as a surrogate for aqueous solubility and Tanimoto-similarity to guide campaigns across several discovery scenarios. In simulations, the framework rapidly identifies highly soluble cocrystals, typically recovering top-5 candidates after fewer than 10 evaluations. Finally, we validate the workflow experimentally with 12 pharmaceutical and pharmaceutical-like compounds, discovering two new cocrystals, resveratrol + praziquantel and purin-6-amine + thiazole-4-carboxylic acid, with markedly enhanced aqueous solubility. These results demonstrate a practical, data-efficient route to Bayesian optimization for cocrystal design.

This study presents a data-driven approach for optimizing the operating altitude of Airborne Wind Energy (AWE) systems to maximize net energy production. Determining the optimal operating altitude of an AWE system is challenging, as the wind speed constantly varies with both time and altitude. Furthermore, without expensive auxiliary equipment, the wind speed is only measurable at the AWE system's operating altitude. The work presented in this paper shows how tools from machine learning can be blended with real-time control to optimize the AWE system's operating altitude efficiently, without the use of auxiliary wind profiling equipment. Specifically, Bayesian Optimization, which is a data-driven technique for finding the optimum of an unknown and expensive-to-evaluate objective function, is applied to the real-time control of an AWE system. The underlying objective function is modeled by a Gaussian Process (GP); then, Bayesian Optimization utilizes the predictive uncertainty information from the GP to decide the best subsequent operating altitude. In the AWE application, conventional Bayesian Optimization is extended to handle the time-varying nature of the wind shear profile (wind speed vs. time). Using real wind data, our method is validated against three baseline approaches. Our simulation results show that the Bayesian Optimization method is successful in dramatically increasing power production over these baselines.

To achieve accurate prediction of tunnel boring machine (TBM) penetration rate (PR) under complex geological conditions, this study proposes an interpretable machine learning (ML) framework optimized using Bayesian optimization (BO). A dataset comprising 411 samples collected from two TBM projects was established, incorporating key geological parameters (uniaxial compressive strength, elastic modulus, and Poisson’s ratio) and operational parameters (thrust force and rotation speed). Seven ML models, including decision tree, Gaussian process regression, k-nearest neighbors, random forest (RF), support vector regression, extreme gradient boosting, and light gradient boosting machine, were systematically optimized using BO under five-fold cross-validation. The results demonstrate that ensemble learning models exhibit superior predictive performance. Among them, the RF model achieved the highest accuracy on the testing set with an R 2 of 0.9582, followed by the light gradient boosting machine with an R 2 of 0.9565. SHapley Additive exPlanations (SHAP) analysis indicates that uniaxial compressive strength is the dominant factor controlling PR, while thrust force act as the primary controllable parameters. The proposed framework effectively integrates high predictive accuracy with strong interpretability, providing a reliable methodological basis for intelligent TBM performance prediction and construction parameter optimization.

We consider an autonomous exploration problem in which a mobile robot is guided by an information-based controller through an a priori unknown environment, choosing to collect its next measurement at the location estimated to be most informative within its current field of view. We propose a novel approach to predict mutual information (MI) using Bayesian optimization. Over several iterations, candidate sensing actions are suggested by Bayesian optimization and added to a committee that repeatedly trains a Gaussian process (GP). The GP estimates MI throughout the robot's action space, serving as the basis for an acquisition function used to select the next candidate. The best sensing action in the committee is executed by the robot. This approach is compared over several environments with two batch methods, one which chooses the most informative action from a set of pseudo-random samples whose MI is explicitly evaluated, and one that applies GP regression to this sample set. Our computational results demonstrate that the proposed method provides not only computational efficiency and rapid map entropy reduction, but also robustness in comparison with competing approaches.

This paper gives a detailed introduction of implementing mixture of Gaussian process (MGP) model and develops its application for Bayesian optimization (BayesOpt). The paper also develops techniques for MGP in finding its mixture components and introduced an alternative gating network based on the Dirichlet distributions. BayesOpt using the resultant MGP model significantly outperforms the one based on Gaussian process regression in terms of optimization efficiency in the test on tuning the hyperparameters in common machine learning algorithms. This indicates the success of the methods, implying a promising future of wider application for MGP model and the BayesOpt based on it.