A novel metaheuristic algorithm with applications in parameter estimation and engineering problems

The Gaussian distribution and its related functions are widely used in statistics, engineering, and physics, including hypothesis testing, quality control, and digital communications. As the standard normal cumulative distribution function (CDF) lacks a closed-form, accurate approximations are essential for practical applications. This study presents new, highly accurate improvements to existing approximations of the standard normal distribution function while preserving simplicity. The proposed methodology refines current formulas by introducing additional parameters and systematically optimizing them using R-based iterative procedures guided by the Kolmogorov-Smirnov statistic to minimize maximum and mean absolute errors. Evaluations across a dense grid confirm the precision and robustness of the proposed models. Results demonstrate significant improvements, with one enhancement reducing the maximum error by a factor of 4.86 compared to previous models, while other formulas show notable accuracy gains. The improved models remain straightforward to implement, making them suitable for real-time engineering and applied statistical computations. A practical application is demonstrated by calculating the bit error rate (BER) in Binary Phase Shift Keying (BPSK) systems under Gaussian noise, where the proposed approximations closely replicate the exact BER curve, confirming their effectiveness in telecommunications system analysis. These improvements provide engineers and scientists with precise, efficient tools, supporting modeling, risk assessment, and system reliability in applications requiring accurate Gaussian evaluations.

Many key problems in science and engineering can be formulated and solved using global optimization techniques. In the particular case of computational biology, the development of dynamic (kinetic) models is one of the current key issues. In this context, the problem of parameter estimation (model calibration) remains as a very challenging task. The complexity of the underlying models requires the use of efficient solvers to achieve adequate results in reasonable computation times. Metaheuristics have been the focus of great consideration as an efficient way of solving hard global optimization problems. Even so, in most realistic applications, metaheuristics require a very large computation time to obtain an acceptable result. Therefore, several parallel schemes have been proposed, most of them focused on traditional parallel programming interfaces and infrastructures. However, with the emergence of cloud computing, new programming models have been proposed to deal with large-scale data processing on clouds. In this paper we explore the applicability of these new models for global optimization problems using as a case study a set of challenging parameter estimation problems in systems biology. We have developed, using Spark, an island-based parallel version of Differential Evolution. Differential Evolution is a simple population-based metaheuristic that, at the same time, is very popular for being very efficient in real function global optimization. Several experiments were conducted both on a cluster and on the Microsoft Azure public cloud to evaluate the speedup and efficiency of the proposal, concluding that the Spark implementation achieves not only competitive speedup against the serial implementation, but also good scalability when the number of nodes grows. The results can be useful for those interested in using parallel metaheuristics for global optimization problems benefiting from the potential of new cloud programming models.

In North America and Europe, there is a significant number of centenary metallic bridges that require maintenance and strengthening operations. These structures need to be adapted to increasing traffic intensities, and their structural integrity should be constantly evaluated. Fatigue damages were not considered in the original design of old metallic bridges, and riveted connections are one of the most frequent sources of fatigue damages on these structures. This paper intends to be a contribution for the reliable assessment of the fatigue behavior of riveted connections by proposing S-N curves for these structural details. Experimental fatigue data is analyzed from multiple bridges across Europe, and different statistical methods were implemented. A normal distribution function was implemented following the ISO 12107 standard, and the results were compared with the implementation of the two-parameter Weibull distribution function. Different estimation methodologies were implemented to determine the parameters of the Weibull distribution. S-N curves obtained by the statistical analysis were then compared with design recommendations from North American and European standards.

Click to increase image sizeClick to decrease image size 2Paper given at the 3rd International Summer School on Problems of Model Choice and Parameter Estimation in Regression Analysis, Mühlhausen (G.D.R.), May 1977 Notes 2Paper given at the 3rd International Summer School on Problems of Model Choice and Parameter Estimation in Regression Analysis, Mühlhausen (G.D.R.), May 1977

Improved small-sample estimation method and stochastic analysis of concrete durability parameters in marine environment

In order to find a method for reasonably determining the number of soil moisture monitoring points (depth 0-20, 20-40, 40-60 cm at per points) in the farmland scale, in Beijinzhuang village and Xilv village of Cui Huangkou town, Wuqing district, Tianjin city. The monitoring area (286.67ha). Monitoring stations is divided into three layers from southeast to northwest. The soil moisture content of 61 points was measured in turn. The parameters were fit analysis based on the obtained data by using P-III type distribution function and normal distribution function. The minimum sample size of the measured sequence was analyzed. The analysis results show that the P-III type distribution function (correlation coefficient R2=0.9954, average relative error is 6.3%) is better than the normal distribution function (correlation coefficient R2=0.9918, average relative error is 6.6%), but the two are very close. Therefore, the number of reasonable soil moisture monitoring points were determined by the normal distribution function . the number of samples of the measured sequence, by which determining the reasonable soil moisture monitoring points in the farmland scale, should not be less than 30 points.

Pump-and-treat (PAT) techniques are often applied to the remediation of dissolved \nchemicals from shallow aquifers. A related management problem typically consists of \nthe selection of the pumping strategy and the most appropriate treatment method, in \norder to minimize the total cleanup cost while meeting a set of technical, economic \nand social constraints. However, due to scarcity of information about the \nhydrogeological system, stochastic modeling approaches seem more appropriate. Of \nprimary concern is the inherent spatial variability of hydraulic conductivity. In \ngeneral, the implementation of a remediation strategy assessed based on uncertain \nhydrogeological parameters leads to a decision involving the risk of constraint \nviolations. The decision-making process may then be formulated as a multiobjective \noptimization framework where the optimality of a pumping pattern is traded off \nagainst its reliability. Operations may be structured into a stochastic optimal \ncontrol problem, in which the remediation strategy is sequentially updated based \nupon new measurements collected during the actual cleanup process. The procedure \nrequires the implementation of an inverse simulation model to estimate the \nstochastic hydrogeological parameters based on a set of potential measurements. In \nthis work, we follow a geostatistical conceptual model where the spatial \ndistribution of hydraulic conductivity is represented as a realization of a log-\nnormally distributed stationary process, characterized by an exponential covariance\nfunction. Using the maximum likelihood method, the parameter estimation problem is \nsolved by determining the set of geostatistical parameters -- average, variance, \nand correlation scales. Available data may include direct measurements of hydraulic \nconductivity, water table elevation at a number of monitoring wells, and \ncontaminant mass extracted from active remediation wells. A rigorous solution to \nthis optimization problem would require a stochastic flow and transport model to be \nincluded in the optimization loop to calculate the expected values and the \ncovariance matrix of the available measurements as functions of the decision \nvariables. Because of the overwhelming computational effort involved, a surrogate \nmodel or response surface is introduced to approximate the objective function. The \nsurrogate model is estimated using a multidimensional kriging interpolation over a \nset of data points or measurements obtained from a series of stochastic flow and \ntransport simulations for pre-established combinations of the decision variables. \nSince the goodness of the solution is ultimately determined by the error of \nestimation of the objective function, which in turn depends on the number and the \nlocation of measurements, an optimal search procedure is used in order to optimize \nthe pattern of data points. The method turns out to be computationally efficient \nand produces results that well approximate the actual geostatistical distribution of\nhydraulic conductivity.

The estimation of biokinetic parameters from experimental data is known to be a tedious step in the identification of bioprocess models. Optimization of a given feed rate profile structure has been performed with respect to a novel combined criterion aiming at both decorrelation and high accuracy of the parameter estimates. For specific values of a weighting factor, the cost reduces to the modified E-criterion, the E-criterion, or the D-criterion for optimal experimental design. It is illustrated how this novel criterion leads to trade-off problems in parameter estimation, such as quality of the parameter estimates, practical feasibility, and model validity.

In recent years considerable research has been devoted to a class of problems which is concerned with ranking and/or selecting a subset of $k$ given populations where the ranking or the selection is defined in terms of a (scalar) parameter of the populations. In these problems the interest is often centered on the populations having large (small) values of the ranking parameter. One usually refers to the $t(< k)$ populations with largest (smallest) values of the ranking parameter as the $t$ best populations. In this paper we consider a problem of selecting a subset of specified size, from a given set of $k$ populations, which contains a subset of the $t$ best populations. Suppose that $\\prod_1, \\prod_2, \\cdots, \\prod_k$ is a given set of $k$ populations, where the distribution function of each observation from $\\prod_i$ is $F(\\cdot \\mid \\theta_i)$. The parameter $\\theta_i$ is unknown, but it belongs to the interval, $\\Theta$, of the real line $(1 \\leqq i \\leqq k)$. We assume that the functional form of $F$ is known. Let $\\theta_{\\lbrack 1\\rbrack} \\leqq \\theta_{\\lbrack 2\\rbrack} \\leqq \\cdots \\leqq \\theta_{lbrack k\\rbrack}$ be the ranked $\\theta_i$; we assume that it is not known with which population $\\theta_{\\lbrack i\\rbrack}$ is associated $(1 \\leqq i \\leqq k)$. The $t$ populations with largest $\\theta$-values are defined as the $t$ best populations and we refer to $\\theta$ as the ranking parameter. The problem of selecting the $t$ best populations in an unordered manner has been studied extensively from the sampling point of view, in relation to several distributions. The usual formulation of the problem is the following: the experimenter's goal is to select the $t$ best populations in an unordered manner. He specifies two positive constants $d^\\ast$ and $P^\\ast$, where $\\binom{k}{t}^{-1} < P^\\ast < 1$. He desires to have a fixed-sample procedure which has a probability of at least $P^\\ast$ of selecting the $t$ best populations whenever $\\theta_{\\lbrack k - t + 1\\rbrack}$ is at a distance not less than $d^\\ast$ from $\\theta_{\\lbrack k - t\\rbrack}$. Bechofer [3] developed a procedure based on predetermined number of observations from each population when $F$ is the normal distribution function with unknown mean $\\theta$ and known variance. Bechhofer and Sobel [4] considered a similar problem in relation to normal populations where the ranking parameter is the variance. Sobel and Huyett [9], Sobel [8], Rizvi [7], Baar and Rizvi [2] considered similar problems for other distributions. Here we consider a generalized version of the above selection problem. We solve the problem in broad generality by not considering a specific family of distributions $F(\\cdot \\mid \\theta)$, but by assuming certain properties of $F$. Thus many of the available results (concerning the above selection problem) can be obtained as special cases of our results. Let $c, s, t$ be integers such that $\\max (1, s + t + 1 - k) \\leqq c \\leqq \\min (s, t)$, which implies that $\\max (s, t) \\leqq k - 1$. The experimenter's goal, which is referred as Goal I, is to select a subset of size $s$ which contains at least $c$ of the $t$ best populations. The experimenter specifies two positive constants $d^\\ast$ and $P^\\ast(<1)$. He desires to have a fixed-sample procedure for which the probability of selecting the desired type of subset of populations is not less than $P^\\ast$ whenever the distance between $\\theta_{\\lbrack k - t\\rbrack}$ and $\\theta_{\\lbrack k - t + 1\\rbrack}$ is at least $d^\\ast$. Two particular cases of this goal are of special interest: (1) Selection of a subset of size $s(\\geqq t)$ which contains the $t$ best populations, (2) Selection of a subset of size $s(\\leqq t)$ which includes any $s$ of the $t$ best populations. Sobel pointed (see the footnote on page 22 of [3]) out that sometimes Case 2 is of interest. These two cases correspond to $c = t, s \\geqq t$ and $c = s, s \\leqq t$. Further when $c = s = t$, Goal I reduces to the goal of selecting the $t$ best in an unordered manner. The proposed selection procedure, $R_s$, is based on suitable statistics $T_1, T_2, \\cdots, T_k$, where $T_i$ is computed from a random sample of size $n$ from $\\prod_i(1 \\leqq i \\leqq k)$. The procedure $R_s$ selects the subset of populations which corresponds to the $s$ largest $T$-values. Having specified the procedure as above, the problem is to determine the common sample size $n$ so that the probability requirement imposed on the procedure is satisfied. This problem has been solved under the assumption that $T_i$ is an absolutely continuous random variable and its distribution function is stochastically increasing in $\\theta_i$ for each value of $n$. This has been done in Section 5. It should be noted (by considering the subset not selected) that the above selection problem is logically equivalent to similar selection problem where the experimenter's goal is to select a subset of size $(k - s)$ which contains at least $(k - t) - (s - c)$ of the $(k - t)$ populations with smallest $\\theta$-values. Thus solutions to the above problem for all admissible values of $c, s$ and $t$ (with fixed $k$) will provide solutions to the selection problem where the goal is the selection of a subset of size $s$, which contains at least $c$ of those $t$ populations with smallest values for the ranking parameter. Section 6 gives results for the two above mentioned particular cases of Goal I. A theorem, which relates the sample sizes necessary to achieve Goal I, and its particular cases, is given in that section. Section 7 deals with an easily verifiable sufficient condition for the existence of the required sample size. In Section 8, the general results have been applied to normal distributions with unknown mean and known common variance.

The literature abounds with the application of optimization methods for estimating model parameters in equation systems. The utility of these methods is frequently demonstrated on pathological examples using simulated data generated from a known model with a random error component and a known statistical distribution. Unfortunately, parameter estimation problems encountered in practice do not have this advantage. The true model is frequently not known. In fact, one is faced with choosing among various candidate models, all of which may be wrong. Moreover, the error structure is generally unknown and must be estimated from the data. Finally, a great deal of mathematical expertise is required to transform the model and select meaningful starting guesses before parameter estimation can be successful.In order to demonstrate the difficulties of parameter estimation in the industrial environment and the limitations of existing methods, a parameter estimation problem formulated by the Dow Chemical Company is presented and solved. This test problem consists of a stiff differential/algebraic (DAE) model that describes complex kinetics and requires the estimation of nine parameters from batch reactor data. Here the model was inadequate to describe the data, the error structure was not specified and the starting guesses led to a nontrivial optimization problem.The Dow parameter estimation problem was distributed in 1981 to 165 researchers as a followup to the 1980 FOCAPD conference. Of those researchers, eleven agreed to apply their methodologies and expertise to this problem. However, only five acceptable solutions were finally submitted. Here we present and compare these results. Each solution was obtained using different strategies. In most cases the form of the model was also changed to accommodate the algorithms used and to ease the solution procedure. Therefore, while this case study does not present a direct numerical comparison of algorithms, it does offer guidelines and insight towards the solution of difficult parameter estimation problems.

In this thesis we analyze and develop methods based on model order reduction (MOR) for the solution of optimization problems constrained by time-dependent partial differential equations (PDEs). The methods combine a direct solution approach with model reduction via proper orthogonal decomposition (POD) and the discrete empirical interpolation method (DEIM). The reduced-order models (ROMs) are used to approximate the high-dimensional dynamic systems originating from a spatial discretization of a PDE. However, when used in an optimization algorithm, conventional POD/DEIM ROMs often lack the ability to give adequate approximations of the gradient. We propose methods for a suitable enhancement of the ROMs for the optimization purpose which are based on the inclusion of derivative information in the POD and DEIM subspaces. We distinguish two types of error between quantities evaluated with the high-dimensional model and its ROM approximation in dependency on the optimization variable q: The reconstruction error which is evaluated with the same q0 which is used constructing the ROM and the prediction error which assesses approximations at q with a ROM constructed at q0 different to q. The novel reconstruction results we present include estimates for solutions of the adjoint equation and the sensitivity equations as well as for the gradient of the objective function. Based on the estimates we explain how the POD and DEIM bases should be extended with either adjoint or sensitivity information. The enhanced ROMs allow control of the reconstruction error for the objective and its gradient up to machine precision. Moreover, we propose a POD prediction estimate for the objective of the optimization problem in a neighborhood of q where the ROM is constructed. In case of sensitivity-extended POD and DEIM bases we give an analogous result for solutions of the states. The derivative-extended ROMs are then used to develop adaptive algorithms for the solution of optimal control and parameter estimation problems which results in great runtime improvements for the optimization while ensuring high approximation quality of the solution of the original problem. For the parameter estimation case a novel a posteriori error estimate is proposed which assesses the quality of suboptimal solutions obtained with the ROM. A further fundamental contribution is a discussion of discretize-then-optimize (DTO) vs. optimize-then-discretize (OTD) approaches in the context of MOR for optimization. We analyze advantages and disadvantages of both approaches and discuss to which extent our methods exhibit properties of either strategy. We also give examples of representative optimization problems in which standard POD/DEIM ROMs show an inacceptable behavior and can be successfully solved by derivative-extended ROMs. We have further implemented the developed methods emphasizing an efficient realization which is important for the investigation of the MOR potential. We showcase the practical performance of the proposed algorithms and the superiority of derivative-extended over conventional ROMs on two academic and one industry-relevant application which exhibit a variety of challenges for the model reduction approach in optimization.

This thesis aims at developing efficient mathematical methods for solving hierarchical dynamic optimization problems. The main motivation is to model processes in nature, for which there is evidence to assume that they run optimally. We describe models of such processes by optimal control problems (called optimal control models (OCMs)). However, an OCM typically includes unknown parameters that cannot be derived entirely on a theoretical basis, which is in particular the case for the cost function. Therefore, we develop parameter estimation techniques to estimate the unknowns in an OCM from observation data of the process. Mathematically, this leads to a hierarchical dynamic optimization problem with a parameter estimation problem on the upper level and an optimal control problem on the lower level. We focus on multi-stage equality and inequality constrained optimal control problems based on nonlinear ordinary differential equations. The main goal of this thesis is to derive numerically efficient mathematical methods for solving hierarchical dynamic optimization problems, and to use these methods to estimate parameters in high-dimensional OCMs from real-world measurement data. We develop parameter-dependent OCMs for the gait of cerebral palsy patients and able-bodied subjects. The unknown parameters in the OCMs are then estimated from real-world motion capture data provided by the Heidelberg MotionLab of the Orthopedic University Clinic Heidelberg by using the mathematical methods developed within this work. The main novelties and contributions of this thesis to the field of hierarchical dynamic optimization are summarized herein. - We establish a novel mathematical method, a so-called direct all-at-once approach, for solving hierarchical dynamic optimization problems based on the direct multiple shooting method and first-order optimality conditions. - Furthermore, we propose an efficient numerical algorithm for large-scale hierarchical dynamic optimization problems, which fully exploits the structures inherited from both the hierarchical setting and the discretization. - Pontryagin's maximum principle is used to analyze solution properties of hierarchical dynamic optimization problems like second-order optimality conditions of the lower-level problem. - In addition, we propose and discuss alternative methods for hierarchical dynamic optimization that are based on derivative-free optimization and a bundle approach. These methods keep the hierarchical problem setting and do not reformulate the lower-level problem using first-order optimality conditions. - We establish a novel lifting method for regularizing mathematical programs with complementarity constraints, which is discussed and numerically investigated by means of a well-known collection of benchmark problems. - Proofs of regularity and convergence results for sequential quadratic programming methods applied to lifted mathematical programs with complementarity constraints are provided. - Efficient state-of-the-art implementations of all mathematical methods derived in this thesis, as well as a benchmark collection of hierarchical dynamic optimization problems are presented. - High-dimensional optimal control gait models for cerebral palsy patients and able-bodied subjects are developed. The mathematical methods derived in this thesis are used to estimate the unknown model parameters from real-world motion capture data provided by the Heidelberg MotionLab of the Orthopedic University Clinic Heidelberg. The theoretical and practical results presented in this thesis can be considered an initial motivating step towards answering open questions in current medical research in fields like treatment planning, classification of gaits, or the evaluation of surgeries by means of hierarchical dynamic optimization.

Focus of Presentation Epidemiological risk estimates are often adjusted for age (and, if necessary, sex) by design or analysis, and for other factors. Consequently, risk estimates do not pertain to the crude risk factor, but to its population residual after adjusting for age, sex and other covariates. For disease risk, the change in Odds PER Adjusted standard deviation (OPERA) estimates the risk gradient on an appropriate scale; log(OPERA) is the natural measure to compare estimates. Findings Under a multiplicative risk model for a normally distributed (adjusted) risk factor, log(OPERA) = the difference in mean between cases and controls. The area under the receiver operating curve (AUC) = □(log(OPERA)/√2), where □ is the standard normal cumulative distribution function. The risk discrimination from combining risk factors can be predicted from their OPERAs. The polygenic standard deviation estimated from pedigree data = log(OPERA). The OPERA for knowing all familial risk factors can be calculated. Conclusions/Implications We present examples from the breast cancer literature where the wrong conclusions can be made by not using the OPERA concept. We give examples of the value of OPERA estimates in predicting the risk discrimination of their combination and demonstrate why the better one predicts the disease the harder it is to predict it better. Key messages OPERA overcomes problems about interpreting risk factors, and combinations of risk factors, in a way not apparent using changes in AUC. OPERA also puts an upper bound on the role of genetic factors in explaining differences in risk across the population.

Presently, the problem of parameter estimation is addressed for a more general nonlinear model. In this respect, the proposed nonlinear system is composed of the parallel connection of linear and nonlinear blocks. The interconnected structure of linear and nonlinear blocks systems leads to a highly nonlinear problem involving several unknown parameters and inaccessible internal signals. In this parameter estimation method, the linear and nonlinear block parameters are estimated in one stage by using simple a sine signal or multi-cosine wave.

Finite mixtures of MaxEnt distributions