Taking another step: A simple approach to high-dimensional Bayesian optimization

Optimization of complex real-time control systems often requires efficient response to any system changes over time. By combining pattern search optimization with a fast estimated Gaussian Process model, we are able to perform global optimization more efficiently for response surfaces with multiple local minimums or even dramatic changes over the design space. Our approach extends pattern search for global optimization problems by incorporating the global and local information provided by an additive global and local Gaussian Process model. We further develop a global search method to identify multiple promising local regions for parallel implementation of local pattern search. We demonstrate our methods on a standard test problem.

Many computer models of large complex systems are time consuming to experiment on. Even when surrogate models are developed to approximate the computer models, estimating an appropriate surrogate model can still be computationally challenging. In this article, we propose an Additive Global and Local Gaussian Process (AGLGP) model as a flexible surrogate for stochastic computer models. This model attempts to capture the overall global spatial trend and the local trends of the responses separately. The proposed additive structure reduces the computational complexity in model fitting, and allows for more efficient predictions with large data sets. We show that this metamodel form is effective in modelling various complicated stochastic model forms.

Just-in-time software defect prediction (JIT-SDP) is an active topic in software defect prediction, which aims to identify defect-inducing changes. Recently, some studies have found that the variability of defect data sets can affect the performance of defect predictors. By using local models, it can help improve the performance of prediction models. However, previous studies have focused on module-level defect prediction. Whether local models are still valid in the context of JIT-SDP is an important issue. To this end, we compare the performance of local and global models through a large-scale empirical study based on six open-source projects with 227417 changes. The experiment considers three evaluation scenarios of cross-validation, cross-project-validation, and timewise-cross-validation. To build local models, the experiment uses the k-medoids to divide the training set into several homogeneous regions. In addition, logistic regression and effort-aware linear regression (EALR) are used to build classification models and effort-aware prediction models, respectively. The empirical results show that local models perform worse than global models in the classification performance. However, local models have significantly better effort-aware prediction performance than global models in the cross-validation and cross-project-validation scenarios. Particularly, when the number of clusters k is set to 2, local models can obtain optimal effort-aware prediction performance. Therefore, local models are promising for effort-aware JIT-SDP.

In response to the deficiencies in visible-near infrared spectral identification of coal gangue under different working conditions, this paper employs multiple algorithms to construct local and global recognition models and conducts analysis. Simulating five different working conditions in the laboratory, including different detection angles, illumination angles, detection distances, coal powder content, and moisture content. Each working condition is further divided into four levels. Collecting spectral data of coal and gangue samples. Constructing global and local datasets, which are respectively derived from all levels and single levels under certain working conditions. Training local and global models to identify samples based on machine learning algorithms such as Lightgbm and 1DCNN. The results of the experiment indicate that there are variations in the sample spectra and recognition accuracy based on different working conditions. Furthermore, the local model exhibits lower recognition accuracy and inadequate robustness for other level samples under the same working condition. The global model is more robust and practical, as it achieves higher or equal recognition accuracy for local data compared to the local model. Among the global models, Lightgbm has the best overall performance, with an average F1-score of 99.34% and 97.9% for local and global datasets, respectively. It also has fast training time of 0.75s and recognition time of 1 ms. The Lightgbm global model can accurately identify coal and gangue in various working conditions, making it a reliable solution.

Bayesian Co-evolutionary Optimization based entropy search for high-dimensional many-objective optimization

Data heterogeneity across clients in federated learning (FL) settings is a widely acknowledged challenge. In response, personalized federated learning (PFL) emerged as a framework to curate local models for clients' tasks. In PFL, a common strategy is to develop local and global models jointly - the global model (for generalization) informs the local models, and the local models (for personalization) are aggregated to update the global model. A key observation is that if we can improve the generalization ability of local models, then we can improve the generalization of global models, which in turn builds better personalized models. In this work, we consider class imbalance, an overlooked type of data heterogeneity, in the classification setting. We propose FedNH, a novel method that improves the local models' performance for both personalization and generalization by combining the uniformity and semantics of class prototypes. FedNH initially distributes class prototypes uniformly in the latent space and smoothly infuses the class semantics into class prototypes. We show that imposing uniformity helps to combat prototype collapse while infusing class semantics improves local models. Extensive experiments were conducted on popular classification datasets under the cross-device setting. Our results demonstrate the effectiveness and stability of our method over recent works.

Evaluating a thermal image sharpening model over a mixed agricultural landscape in India

Estimating foliar phosphorus of rubber trees using locally modelling approach with hyperspectral reflectance

Federated learning is a cutting-edge method of model training, which leverages the end users to train the global model on the server. The end users are responsible for training locally on their datasets and update the shared global model. Once the local training is executed, the local trained models are forwarded back to the server to further upgrade the global model by performing aggregation. This process of global training is carried out for certain number of rounds. Practically, the datasets of clients are distributed heterogeneously. Thus, the updated local models by clients emanate broad variation among local models due to heterogeneity. In other words, the aggregation of local models plays a vital role in federated learning. Specifically, aggregating the diversified local models may deliver unsatisfactory output if not performed efficiently. This article presents a performance efficient and robust aggregation approach for heterogeneous federated learning called FedLbl. Our approach takes the diversity of data among clients into consideration before conducting the aggregation of local models. Our study compares the proposed method with conventional federated learning techniques, resulting in a 28% increase in accuracy and a 19% reduction in loss.

Device sizing is a challenging problem for analog circuit design. Traditional methods depend on domain knowledge and intensive simulations to search for feasible parameters. Recent studies apply the Bayesian optimization (BO) and a Gaussian process (GP) model in analog circuit synthesis to improve efficiency. The BO framework automatically selects the parameter candidates by inferring the surrogate GP model. However, naive BO employs a sequential updating strategy which is inefficient in a multicore environment. Besides, the widely used GP model requires costly high fidelity data, which are obtained from fine simulations. In this article, we propose a constrained batch BO approach with a multifidelity (MF) model to solve the above difficulties. The batch BO exploits parallel computing and selects promising parameters by multiple acquisition function ensemble. In addition, the MF GP model adapts the low fidelity data obtained from coarse simulations. Specifically, the proposed method incorporates information gain in a weighted clustering algorithm to refine the parameter candidates. As a result, the proposed method maintains the candidates’ quality and diversity, which speeds up the optimization convergence. In the experiments, we demonstrate the efficiency of the proposed approach on three real-world circuits. The results show that our approach reduces the simulation costs by at least 54.6% compared to the state-of-the-art baselines.

This study assessed the economic value of public urban green spaces (UGSs) in Kuala Lumpur (KL) city by using the hedonic price method (HPM). It involves 1269 house units from eight sub-districts in KL city. Based on the hedonic price method, this study formulates a global and local model. The global model and local model are analyzed using ordinary least square (OLS) regression and geographically weighted regression (GWR). By using the hedonic price method, the house price serves as a proxy for public urban green spaces' economic value. The house price is regressed against the set of three variables which are structural characteristics, neighborhood attributes, and environmental attributes. Measurements of interest in this study are environmental characteristics, including distance to public UGSs and size of public UGSs. The results of the OLS regression illustrated that Taman Rimba Kiara and Taman Tasik Titiwangsa provide the maximum economic value. On average, reducing the distance of the house location to Taman Rimba Kiara by 10m increased the house price by RM1700. Similarly, increasing the size of the Taman Tasik Titiwangsa by 1000m2 increases the house price by RM60,000. The advantage of the GWR result is the economic value of public UGSs which can be analyzed by the specific location according to sub-district. From this study, the GWR result exposed that the economic values of Taman Rimba Bukit Kiara and Taman Tasik Titiwangsa were not significant in each of the sub-district within KL city. Taman Rimba Bukit Kiara was negatively significant at all sub-districts except Setapak and certain house locations located at the sub-district of KL. In contrast, Taman Tasik Titiwangsa was positively significant at all sub-districts except certain house locations at the sub-districts of Batu, KL, Setapak, and KL city center. In conclusion, results show that the house price is influenced by the environmental attribute. However, even though both of these public UGSs generate the highest economic value based on distance and size, its significant values with an expected sign are only obtained based on the specific house location as verified by the local model. In terms of model comparison, the local model was better compared with the global model.

Federated learning (FL) is an effective distributed learning paradigm for protecting client privacy, enabling multiple clients to collaboratively train a global model without uploading private data. It has promising applications in sports image classification. However, FL faces the issue of non-independent and identically distributed (non-IID) data, which leads to excessive variance between local models and hinders the convergence of the global model. Although FedSAM and its variants attempt to reduce this variance by finding smooth solutions between local models, local smoothing does not necessarily result in global smoothing. We refer to this issue as the smoothness inconsistency problem. To address this challenge, we propose a novel FL paradigm, named A-FedSAM, which utilizes adaptive local distillation to achieve consistency in smoothing between local and global models without incurring additional communication overhead, thereby improving the convergence accuracy of the global model. Specifically, A-FedSAM employs the global model as the teacher during local training, dynamically guiding the local models to ensure that their gradients not only maintain smoothness but also align with the global objective. Extensive experiments on sports image classification tasks demonstrate that A-FedSAM outperforms state-of-the-art methods in terms of accuracy across different data heterogeneities and client sampling rates, while requiring fewer communication and computational resources to achieve the same target accuracy.

This paper deals with a nonparametric identification of continuous-time nonlinear systems using multiple local Gaussian process (GP) models. Multiple sets of training input and output data are collected to train the local GP prior models. Each local GP prior model is trained by minimizing the negative log marginal likelihood of each set of the training data. The final nonlinear function with confidence measure is estimated by weighted mean of the local estimated nonlinear functions using the predictive variances of local GP posterior distributions. Compared to the standard GP-based identification method, the proposed method can reduce the computational cost and improve the accuracy of identification. Simulation results are shown to illustrate the effectiveness of the proposed identification method.

Many industrial processes are nonlinear distributed parameter systems (DPS) that have significant spatiotemporal dynamics. Due to different production and working conditions, they often need to work at a large operating range with multiple working points. However, direct global modeling and persistently exciting experiment in a large working region are very costly in many cases. The complex spatiotemporal coupling and infinite-dimensional nature make the problem more difficult. In this study, a kernel-based spatiotemporal multimodeling approach is proposed for the nonlinear DPS with multiple working points. To obtain a reasonable operating space division, an iterative approach is proposed where the operating space division and local modeling are performed iteratively. The working range of the current local model will help to determine the next operating point required for modeling. Utilizing the potential of each local model, the number of regions can be reduced. In the local modeling, the Karhunen–Loève method is used for the space/time separation and dimension reduction, and after that unknown parameters of kernels are estimated. Due to consideration of time-scale properties in the dimension reduction, the modeling approach is particularly suitable for dissipative PDEs, particularly of parabolic type. The multimodeling and space/time separation techniques can largely reduce the complexity of global nonlinear spatiotemporal modeling. Finally, to guarantee a smooth transition between local spatiotemporal models, a scheduling integration method is used to provide a global spatiotemporal model. To design scheduling functions, a two-stage training method is proposed to reduce the design complexity. Compared with direct global modeling, the exciting experiment and modeling for each local region become easier. Compared with one local modeling, the multimodel integration will improve modeling accuracy. The effectiveness of the proposed modeling approach is verified by simulations.

Surrogate-assisted evolutionary algorithms (SAEAs) have been widely developed to solve complex and computationally expensive multiobjective optimization problems (EMOPs) in recent years. However, when dealing with high-dimensional optimization problems in decision space, the performance of these surrogate-assisted multiobjective evolutionary algorithms (MOEAs) deteriorates drastically. In this work, a novel classifier-assisted rank-based learning and local model-based multiobjective evolutionary algorithm (CLMEA) is proposed for high-dimensional EMOPs. CLMEA makes full use of the uncertainty of solutions in the decision space and objective space to explore the uncertain but informative space toward high-dimensional problems. Specifically, the offspring in different ranks uses rank-based learning strategy to generate more promising and informative candidates for function evaluations (FEs). To reduce the search region of high-dimensional problems and maintain the diversity of solutions, the most uncertain sample point from the nondominated solutions measured by the crowding distance is selected as the center to conduct local search. The experimental results of benchmark problems and a real-world application on geothermal reservoir heat extraction optimization demonstrate superior performance of CLMEA compared with the state-of-the-art surrogate-assisted MOEAs. The source code for this work is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/JellyChen7/CLMEA</uri>