The EM Algorithm, Its Randomized Implementation and Global Optimization: Some Challenges and Opportunities for Operations Research

Traffic signal timing optimization problem aims at alleviating traffic congestion and shortening the average traffic time. However, most existing research considered only the information of one or few intersections at a time. Those local optimization methods may experience a decrease in performance when facing large-scale traffic networks. In this paper, we propose a cellular automaton traffic simulation system and conduct tests on two different optimization schemes. We use Genetic Algorithm (GA) for global optimization and Expectation Maximization (EM) as well as car flow for local optimization. Empirical results show that the GA method outperforms the EM method. Then, we use linear regression to learn from the global optimal solution obtained by GA and propose a new adjustment strategy that outperforms recent optimization methods.

Operations Research applications: Opportunities and accomplishments

In this paper, we first establish some sufficient and some necessary global optimality conditions for quadratic integer programming problems. Then we present a new local optimization method for quadratic integer programming problems according to its necessary global optimality conditions. A new global optimization method is proposed by combining its sufficient global optimality conditions, local optimization method and an auxiliary function. The numerical examples are also presented to show that the proposed optimization methods for quadratic integer programming problems are very efficient and stable.

Hammerstein–Wiener models constitute a significant class of block-structured dynamic models, as they approximate process nonlinearities on the basis of input–output data without requiring identification of a full nonlinear process model. Optimization problems with Hammerstein–Wiener models embedded are nonconvex, and thus local optimization methods may obtain suboptimal solutions. In this work, we develop a deterministic global optimization strategy that exploits the specific structure of Hammerstein–Wiener models to extend existing theory on global optimization of systems with linear dynamics. At first, we discuss alternative formulations of the dynamic optimization problem with Hammerstein–Wiener models embedded, demonstrating that careful selection of the optimization variables of the problem can offer significant numerical advantages to the solution approach. Then, we develop convex relaxations for the proposed optimization problem and discuss implementation aspects to obtain the global solution focusing on a control parametrization technique. Finally, we apply our optimization strategy to case studies comprising both offline and online dynamic optimization problems. The results confirm an improved computational performance of the proposed solution approach over alternative options not exploiting the linear dynamics for all considered examples. They also underline the tractability of deterministic global dynamic optimization when using few control intervals in online applications like nonlinear model predictive control.

*† ‡ § An efficient and high-fidelity design approach is proposed by combining global and local optimization methods for wing planform and surface design. For enhanced design results, aerodynamic shape optimization process is carried out via 2-stage with different optimization strategy. In the first stage, global optimization techniques are applied to planform design with a few geometric design variables. In the second stage, local optimization techniques are used for wing surface design with a lot of design variables to maintain a sufficient design space with high DOF (Degree of Freedom) geometric change. For global optimization, meta-modeling techniques such as RS (Response Surface) and Kriging methods are used in conjunction with Genetic Algorithm (GA). For local optimization, a discrete adjoint variable method is used. By the successive combination of global and local optimization techniques, drag minimization is performed for a multi-body aircraft configuration while maintaining the baseline lift and the wing weight at the same time. Through the design process, performances of the test models are remarkably improved in comparison with the single stage design approach. The capability of proposed design framework including wing planform design variables can be evaluated by the drag decomposition method which can provide improvement of induced drag and wave drag, respectively.

Multivariate quartic polynomial optimization problems, as a special case of the general polynomial optimization, have a lot of practical applications in real world and are proved to be NP-hard. In this paper, some necessary local optimality conditions and some necessary global optimality conditions for quartic polynomial optimization problems with mixed variables are established. Then some local optimization methods, including weakly local optimization methods for general problems with mixed variables and strongly local optimization methods for quartic polynomial optimization problems with mixed variables, are proposed by exploiting these necessary local optimality conditions and necessary global optimality conditions. A global optimization method is proposed for quartic polynomial optimization problems by combining these local optimization methods together with some auxiliary functions. Some numerical examples are also given to illustrate that these approaches are very efficient.

Simulation optimization studies the problem of optimizing simulation-based objectives. Simulation optimization is a new and hot topic in the field of system simulation and operational research. To improve the search efficiency, this paper presents a hybrid approach which combined genetic algorithm and local optimization technique for simulation optimization problems. Through the combination of genetic algorithms and with the local optimization method, it can maximally use the good global property of random searching and the convergence rate of a local method. This study considers the sampling procedure based on orthogonal design and quantization technology, the use of orthogonal genetic algorithm with quantization for the global exploration, and the application of local optimization technique for local exploitation. The final experimental results demonstrated that the proposed approach can find optimal or close-to-optimal solutions, and is superior to other recent algorithms in simulation optimization. Key words: Simulation optimization, genetic algorithms, local optimization, orthogonal design.

GL-GAN: Adaptive global and local bilevel optimization for generative adversarial network

New Opportunities for Operations Research – Feature Issue of the Joint EURO/INFORMS Conference, Istanbul, July 2003

A reconsideration on the resolution of phase stability analysis using stochastic global optimization methods: Proposal of a reliable set of benchmark problems

Features of several well-known global and local optimization methods were discussed and their robustness and efficiency were tested on several artificial test functions in Matlab environment. The tested local methods were the interior-point, the quasi-Newton method, Nelder–Mead simplex, the pattern search, the NEWUOA and the BOBYQA methods. The global methods were the genetic algorithm (GA), the simulated annealing (SA), the particle swarm optimization (PSO), and the covariance matrix adaptation evolutionary strategy (CMA-ES) methods (see subsections 2.2 and 2.3 for their details). Furthermore, a novel global optimization method, called FOCusing robusT Optimization with Uncertainty-based Sampling (FOCTOPUS), which proved to be very efficient in the optimization of constrained and highly correlated parameters of combustion kinetic models, was also tested. The test functions were selected in such a way that they had a variety of features: uni-modal and multi-modal, differentiable and non-differentiable, separable and non-separable, low dimensional and high dimensional. The following test functions were used: the 20D Alpine function, the 4D Ackley function, the Cross-in-tray 2D function, the Hartmann-6D function, the Holder table 2D function, the 5D Rastrigin function, the 5D Rosenbrock function, the 4D modified Rosenbrock function, the 20D Zakharov function and a typical 2D multi-modal function. The general conclusion here is that, the global methods performed well on the multi-modal and high-dimensional test functions while the local methods were superior in the case of low-dimensional and uni-modal test functions. For the highly multi-modal test functions, the GA was better than all the other methods. The FOCTOPUS method proved to be inferior to GA for most of the test functions, thus its application cannot be generally recommended.

EM algorithm is a very popular iteration-based method to estimate the parameters of Gaussian Mixture Model from a large observation set. However, in most cases, EM algorithm is not guaranteed to converge to the global optimum. Instead, it stops at some local optimums, which can be much worse than the global optimum. Therefore, it is usually required to run multiple procedures of EM algorithm with different initial configurations and return the best solution. To improve the efficiency of this scheme, we propose a new method which can estimate an upper bound on the logarithm likelihood of the local optimum, based on the current configuration after the latest EM iteration. This is accomplished by first deriving some region bounding the possible locations of local optimum, followed by some upper bound estimation on the maximum likelihood. With this estimation, we can terminate an EM algorithm procedure if the estimated local optimum is definitely worse than the best solution seen so far. Extensive experiments show that our method can effectively and efficiently accelerate conventional multiple restart EM algorithm.

Global optimisation problems in high-dimensional and infinite-dimensional spaces arise in various real-world applications such as engineering, economics and finance, geophysics, biology, machine learning, optimal control, etc. Among stochastic approaches to global optimisation, biology-inspired methods are currently very popular in the literature. Bio-inspired approaches imitate natural ecological and evolutionary processes and are reported to be efficient in a large number of practical study cases. However, many bio-inspired methods can possess some vital drawbacks. For example, due to their semi-empirical nature, convergence to the globally optimal solution cannot always be guaranteed. Another major obstacle is that the existing methods often struggle with higher dimensionality of the space of parameters, which results in a slow convergence. Moreover, it is often difficult to adjust the dimensionality of the space of parameters in the corresponding computer code for a practical realisation of the optimisation method. Here, we present a bio-inspired global stochastic optimisation method applicable in Hilbert function spaces. The proposed method is an evolutionary algorithm inspired by the Darwin’s famous idea of survival of the fittest and is, therefore, referred to as the ‘Survival of the Fittest Algorithm’ (SoFA). Mathematically, the convergence of SoFA is a consequence of the fundamental property of localisation of probabilistic measure in a Hilbert space and we rigorously prove the convergence of the introduced algorithm for a generic class of functionals. The approach is simple in terms of practical coding. As an insightful, real-world problem, we apply our method to find the globally optimal trajectory for the daily vertical migration of zooplankton in the ocean and lakes, this phenomenon is considered to be the largest synchronised movement of biomass on Earth. We maximise fitness in a function space derived from a von-Foerster stage-structured population model with biologically realistic parameters. We show that for problems of fitness maximisation in high-dimensional spaces, SoFA provides better performance as compared to some other stochastic global optimisation algorithms. We highlight the links between the new optimisation algorithm and natural selection processes in ecosystems occurring within a population via gradual exclusion of competitive con-specific strains.

Cross-Hill: A heuristic method for global optimization

Utilization of surface wave dispersive properties may be roughly divided into three steps: field data acquisition, dispersion-curve picking, and inversion of phase velocities (