A Nested Weighted Tchebycheff Multi-Objective Bayesian Optimization Approach for Flexibility of Unknown Utopia Estimation in Expensive Black-Box Design Problems

Bayesian optimization (BO) is a low-cost global optimization tool for expensive black-box objective functions, where we learn from prior evaluated designs, update a posterior surrogate Gaussian process model, and select new designs for future evaluation using an acquisition function. This research focuses upon developing a BO model with multiple black-box objective functions. In the standard multi-objective (MO) optimization problem, the weighted Tchebycheff method is efficiently used to find both convex and non-convex Pareto frontiers. This approach requires knowledge of utopia values before we start optimization. However, in the BO framework, since the functions are expensive to evaluate, it is very expensive to obtain the utopia values as a prior knowledge. Therefore, in this paper, we develop a MO-BO framework where we calibrate with multiple linear regression (MLR) models to estimate the utopia value for each objective as a function of design input variables; the models are updated iteratively with sampled training data from the proposed MO-BO. These iteratively estimated mean utopia values are used to formulate the weighted Tchebycheff MO acquisition function. The proposed approach is implemented in two numerical test examples and one engineering design problem of optimizing thin tube geometries under constant loading of temperature and pressure, with minimizing the risk of creep-fatigue failure and design cost, along with risk-based and manufacturing constraints. Finally, the model accuracy with frequentist, Bayesian and without MLR-based calibration are compared to true Pareto solutions.

The presence of black-box functions in engineering design, which are usually computation-intensive, demands efficient global optimization methods. This work proposes a new global optimization method for black-box functions. The global optimization method is based on a novel mode-pursuing sampling (MPS) method which systematically generates more sample points in the neighborhood of the function mode while statistically covers the entire search space. Quadratic regression is performed to detect the region containing the global optimum. The sampling and detection process iterates until the global optimum is obtained. Through intensive testing, this method is found to be effective, efficient, robust, and applicable to both continuous and discontinuous functions. It supports simultaneous computation and applies to both unconstrained and constrained optimization problems. Because it does not call any existing global optimization tool, it can be used as a standalone global optimization method for inexpensive problems as well. Limitation of the method is also identified and discussed.

A Pareto-optimal solution is developed in this paper for a scheduling problem on a single machine with periodic maintenance and non-preemptive jobs. Most of the scheduling problems address only one objective function, while in the real world, such problems are always associated with more than one objective. In this paper, both multi-objective functions and multi-maintenance periods are considered for a machine scheduling problem. To avoid complexities, multiple objective functions are consolidated and transformed into a single objective function after they are weighted and assigned proper weighting factors. In addition, periodic maintenance schedules are also considered in the model. The objective of the model addressed is to minimize the weighted function of the total job flow time, the maximum tardiness, and the machine idle time in a single machine problem with periodic maintenance and non-preemptive jobs. An algorithm is developed to solve this multiple criterion problem and to construct the Pareto-set. The parametric analysis of the trade-offs of all solutions with all possible weighted combination of the criteria is performed. A neighborhood search heuristic is also developed. Results are provided to explore the best schedule among all the Pareto-optimality sets and to compare the result of the modified Pareto-optimality algorithm with the result of the neighborhood search heuristic.

Surrogate algorithms such as Bayesian optimisation are especially designed for black-box optimisation problems with expensive objectives, such as hyperparameter tuning or simulation-based optimisation. In the literature, these algorithms are usually evaluated with synthetic benchmarks which are well established but have no expensive objective, and only on one or two real-life applications which vary wildly between papers. There is a clear lack of standardisation when it comes to benchmarking surrogate algorithms on real-life, expensive, black-box objective functions. This makes it very difficult to draw conclusions on the effect of algorithmic contributions and to give substantial advice on which method to use when. A new benchmark library, EXPObench, provides first steps towards such a standardisation. The library is used to provide an extensive comparison of six different surrogate algorithms on four expensive optimisation problems from different real-life applications. This has led to new insights regarding the relative importance of exploration, the evaluation time of the objective, and the used model. We also provide rules of thumb for which surrogate algorithm to use in which situation. A further contribution is that we make the algorithms and benchmark problem instances publicly available, contributing to more uniform analysis of surrogate algorithms. Most importantly, we include the results of the six algorithms on all evaluated problem instances. This unique new dataset lowers the bar for researching new methods as the number of expensive evaluations required for comparison and for the creation of new surrogate models is significantly reduced.

Improved crop forecasts for the Australian macadamia industry from ensemble models

With the explosive increase in data on the web, recommending items to users is becoming more complex. In recent times, the best recommender systems have come from ensemble learning, which combines many models and techniques to generate recommendations that can draw the best characteristics of the constituent models. These ensemble models can improve accuracy, and they are also able to reduce the biases that come with each model. However, the massive number of permutations in which recommendation models are combined to create an ensemble model adds another layer of complexity to an already complex problem. Thus, in line with the recent trend in Automated Machine Learning (AutoML) aimed at reducing the complexity associated with model selection, there is a need for a machine learning framework that can learn the best ensemble model for a given problem given the base models. We proposed a system Ensemble with Particle Swarm Optimization, which intelligently optimizes the recommendations by identifying the best ensemble architecture for the data at hand. Our proposed AutoML system can improve recommendations for the MovieLens dataset by combining the results from base techniques without any user effort. The major challenge in creating such a system is to develop a framework for generating ensemble models and finding an efficient way to reach the best performing model since the search space can be vast. To overcome this challenge, we have used hierarchical models to generate ensembles and Particle Swarm Optimization to find the optimal ensemble models.

Employing machine learning techniques in predicting the parameters of metamaterial antennas has a significant impact on the reduction of the time needed to design an antenna with optimal parameters using simulation tools. In this paper, we propose a new approach for predicting the bandwidth of metamaterial antenna using a novel ensemble model. The proposed ensemble model is composed of two levels of regression models. The first level consists of three strong models namely, random forest, support vector regression, and light gradient boosting machine. Whereas the second level is based on the ElasticNet regression model, which receives the prediction results from the models in the first level for refinement and producing the final optimal result. To achieve the best performance of these regression models, the advanced squirrel search optimization algorithm (ASSOA) is utilized to search for the optimal set of hyper-parameters of each model. Experimental results show that the proposed two-level ensemble model could achieve a robust prediction of the bandwidth of metamaterial antenna when compared with the recently published ensemble models based on the same publicly available benchmark dataset. The findings indicate that the proposed approach results in root mean square error (RMSE) of (0.013), mean absolute error (MAE) of (0.004), and mean bias error (MBE) of (0.0017). These results are superior to the other competing ensemble models and can predict the antenna bandwidth more accurately.

본 연구의 목적은 개념적인 강우-유출모형인 Tank 모형의 매개변수를 산정하기 위한 다목적 유전자알고리즘의 적용성을 평가하는 것이다. 다목적 유전자알고리즘 기법으로는 최근에 가장 많이 사용되는 기법중의 하나인 NSGA-II를 채택하여 Tank 모형과 결합하였으며, 4가지 목적함수(유출용적오차, 평균제곱근 오차, 고수유량 평균제곱근 오차 및 저수유량 평균제곱근 오차)값을 최소화하는 형태의 목적함수를 적용하였다. NSGA-II는 목적함수의 개수가 많아지면 한 번의 실행에 의해 굉장히 많은 수의 파레토최적해를 구하는 단점을 가지고 있기 때문에 구해진 파레토최적해 중에서 어떤 해가 최우선해 인지를 결정해야 할 필요가 있으며, 이러한 고차원적인 의사결정을 위하여 선호적순서화(preference ordering) 기법을 적용하였다. NSGA-II를 이용하여 Tank모형의 매개변수를 추정할 때 초기조건이 최적화과정에 미칠 수 있는 영향을 최소화하기 위해 세대수(generation number)와 개체군의 크기(population size)에 대한 민감도분석을 수행하였다. 분석결과 Tank모형의 매개변수 최적화를 위한 세대수와 개체군 크기의 초기 값을 각각 900번과 1000개로 선정하는 것이 적합한 것으로 나타났다. The objective of this study is to evaluate the applicability of multi-objective genetic algorithm(MOGA) in order to calibrate the parameters of conceptual rainfall-runoff model, Tank model. NSGA-II, one of the most imitating MOGA implementations, is combined with Tank model and four multi-objective functions such as to minimize volume error, root mean square error (RMSE), high flow RMSE, and low flow RMSE are used. When NSGA-II is employed with more than three multi-objective functions, a number of Pareto-optimal solutions usually becomes too large. Therefore, selecting several preferred Pareto-optimal solutions is essential for stakeholder, and preference-ordering approach is used in this study for the sake of getting the best preferred Pareto-optimal solutions. Sensitivity analysis is performed to examine the effect of initial genetic parameters, which are generation number and Population size, to the performance of NSGA-II for searching the proper paramters for Tank model, and the result suggests that the generation number is 900 and the population size is 1000 for this study.

This landscape study aims to provide an understanding of Large Language Models (LLMs), especially with a view point of supporting scientific and community queries for climate resilience and preparedness. Typically, for queries related to climate resilience and preparedness, the data span multiple heterogeneous sources and are complex by nature. Scientists as well as communities seek answers to real-world questions from LLMs, yet the accuracy of the feedback is not always guaranteed to be consistent and free of hallucinations. Given the complex landscape and the number of LLM models available, the choice of a model becomes complex too. In addition, LLM ensemble models are also taking on an important role where high level of trustworthiness and accuracy is needed. To address this gap, we performed a review of the LLM research landscape, identifying common themes. This was followed by a deep dive into ensemble models. We evaluate the ensembles as well as a popular single LLM model (that is, ChatGPT) with two sample queries: (i) ‘Are there any linkages between sea ice retreat and ice shelf melt in the Antarctic?’, (ii) ‘What is the best crop for sandy soils?’. The responses were assessed for robustness and accuracy by the domain expert. Through our analysis we concluded that a single LLM provides a parameter free setup whereas the ensemble models needed fine tuning to arrive to robust outcomes. However, the paper does not address ensemble performance issues and neither does it diminish their importance. Instead, this indicates that a lot more research is needed to get ensembles to a more user-friendly and approachable state of use unlike single LLMs. We also outline criteria that an ensemble model selection will need to consider such as initial model selection, retraining, ensemble optimization, energy utilization.

Variable selection in regression models has been well studied in the literature, with many non‐Bayesian and Bayesian methods available in this regard. An important class of regression models is generalized linear models, which involve situations where the response variable is discrete. To add more flexibility, generalized additive partial linear models can be considered, where some predictors can have a non‐linear effect while some predictors have a strictly linear effect. We consider Bayesian variable selection in these models. The functions in the non‐parametric additive part of the model are expanded in a B‐spline basis and multivariate Laplace prior put on the coefficients with point mass at zero. The coefficients corresponding to the strictly linear components are assigned a univariate Laplace prior with point mass at zero. The prior times the likelihood is mathematically intractable, but we find an approximation by expansion around the posterior mode, which is the group lasso solution in generalized linear model setting for the choice of prior. We thus completely avoid Markov chain Monte Carlo methods, which are extremely slow and unreliable in high‐dimensional models. We evaluate the performance of the Bayesian method by conducting simulation studies and real data analyses. Copyright © 2014 John Wiley & Sons, Ltd.

This article investigates the competence of ensemble learning techniques in solar irradiance prediction. It was seen from the literature survey, an ensemble tree model, random forests is studied more frequently as ensemble models. However, ensemble of support vector regression (SVR) and artificial neural networks (ANN) is also possible. So, this study is the first detailed evaluation of ensemble models in solar irradiance estimation domain. Boosting and bagging ensembles of SVR, ANN and decision tree (DT), are developed to estimate solar irradiance in hourly basis in five cities in Turkey. First frequently used base models (SVR, ANN, and DT) are created and tested with the use of 5 years meteorological data. Then boosting and bagging ensembles of the base models are developed and tested with the same data. The base models are compared with their ensemble counterparts in terms of average coefficient of determination (R2) and root mean squared error (RMSE). The comparative results show that boosting and bagging ensemble models improve SVR, ANN, and DT in terms of RMSE between 4.6 and 14.6% in average. The results show empirically that ensemble models improve prediction accuracies of various base regression models and it can be applied to other machine learning models used in solar irradiance prediction.

Machine learning algorithms aim to improve the power of predictors over conventional regression models. This study aims to tap the predictive potential of crash mechanism-related variables using ensemble machine learning models. The results demonstrate selected models can predict severity at a high level of accuracy. The stacking model with a linear blender is preferred for the designed ensemble combination. Most bagging, boosting, and stacking algorithms perform well, indicating ensemble models are capable of improving upon individual models.

Predictions of mechanical properties of Fiber Reinforced Concrete using ensemble learning models

Ideally, probabilistic forecasts improve the decision quality, capture upsides and mitigate downsides. Unfortunately, one of the risks associated with probabilistic forecasts is the possible creation of a large number of models with similar properties or responses. Methods to identify representative models from an ensemble of models are thus necessary. In this paper, we extend the use of distance-based methods, developed at Stanford Center for Reservoir Forecasting, and Kernel clustering techniques to the selection of models for economic decision-making. Using mathematical tools, the large set of models in a high-dimensional space is mapped to a low-dimensional space and a distance function based on several prediction-related quantities is used to quantify the similarities and dissimilarities among an ensemble of history-matched models. We then construct a small portfolio of models with a diverse range of prediction performances for less cost-intensive probabilistic forecasts. The approach is applied to an offshore gas condensate field in the Asia-Pacific region. Two possible development plans were competing: continue to produce from existing geological structures, or extend the development to accumulations connected through a saddle in the carbonate reservoir. The decision was impacted by the uncertainty on remaining volumes in place and connectivity between structures. Based on the uncertainty management plans, 925 history-matched models are generated with genetic algorithms. Within the context of distance-based modeling, an application-tailored distance function between models is designed, balancing static (scenario independent) and dynamic (scenario dependent) properties of the models. Then, eight diverse and representative models are selected among the 925 models. We show that the eight selected models, while offering an equally good history-match, cover well the uncertainty ranges of the static and dynamic properties considered. The selected models can be used to evaluate different possible development scenarios for final business decision making.

In this paper, a stochastic multi-objective structure is introduced in joint energy and reserve market to allow energy generation companies (GENCOs) participating in the short-term hydro-thermal self-scheduling with wind, photovoltaic uncertainty and small-hydro units. In addition, uncertainties including energy price, spinning and non-spinning reserve prices as well as the uncertainty of renewable energy resources such as output power of the wind, PV and small-hydro power plants are mentioned. One pivotal feature of this study is that two methods are used to generate stochastic multi-objective scenarios, namely lattice monte carlo simulation and roulette wheel mechanism. After that, the main purpose of the study is described, i.e., making GENCOs able to achieve the maximum profit and the minimum emission by using a multi-objective function considering a stochastic process. To reach this aim, the mixed integer programming which includes a set of multi stage deterministic scenarios is employed. However, some special cases should be introduced in the formulation structure of the presented scheduling regarding hydro-thermal units to make the SMO-HTSS problem with wind, PV and SH units alike real time modeling. Since optimal Pareto solutions are produced in this method, one can allude to the application of the e-constraint method. Nevertheless, in order to select one of the most appropriate solutions among Pareto solutions obtained, the utilization of fuzzy method has been presented. In the end, some tests are carried out on an IEEE 118-bus test system to verify the accuracy and validity of the proposed method.