On Efficient Solutions to the Continuous Sensitivity Equation Using Automatic Differentiation

This paper presents results for the study of equivalence between the total form continuum sensitivity equation (CSE) method and the discrete analytic method of shape design sensitivity analysis. For the discrete analytic method, the sensitivity equations are obtained by taking analytic derivatives of the discretized equilibrium equations with respect to the shape design parameters. For the CSE method, the equilibrium equations are firstly differentiated to form a set of linear continuous sensitivity equations and then discretized for solving the shape sensitivities. The sensitivity equations can be derived by taking the material derivatives (total form) or the partial derivatives (local form) of the equilibrium equations. The total form CSE method is shown for the first time to be equivalent, after finite element discretization, to the discrete analytic method for nonlinear second-order differential equations of a particular form with design dependent loads when they use the same: (1) finite element discretization, (2) numerical integration of element matrices, (3) design velocity fields that are linear with respect to the design variable and (4) shape functions for domain transformation and for design velocity field calculations. The shape sensitivity equations for three-dimensional geometric nonlinear elastic structures and linear potential flow are derived by using both total form CSE and discrete analytic method to show the equivalence of the two methods for these specific examples. The accuracy of shape sensitivity analysis is verified by potential flow around an airfoil and a joined beam with an airfoil under gust load. The results show that analytic sensitivity results are consistent with the complex step results.

A Continuum Sensitivity Equation (CSE) method was developed for the uid ow problems to support computationally e cient optimization of aerodynamic design problems. The continuum sensitivity equations and sensitivity boundary conditions were derived using both local and total derivative form for potential ow around an airfoil. The nite element method was adopted for solving both of the original ow analysis problem and the sensitivity equations. In this paper, the pressure coe cient sensitivity with respect to the angle of attack was calculated for veri cation. Shape sensitivities with respect to airfoil shape parameters were also calculated, and the accuracy and e ciency between local form and total form CSE was compared.

Sensitivity analysis is an important component of many model activities in hydrology. Numerous studies have been conducted in calculating various sensitivities. Most of these sensitivity analysis focus on the sensitivity of state variables (e.g., hydraulic head) to parameters representing medium properties such as hydraulic conductivity or prescribed values such as constant head or flux at boundaries, while few studies address the sensitivity of the state variables to some shape parameters or design parameters that control the model domain. Instead, these shape parameters are typically assumed to be known in the model. In this study, based on the flow equation, we derive the equation (and its associated initial and boundary conditions) for sensitivity of hydraulic head to shape parameters using continuous sensitivity equation (CSE) approach. These sensitivity equations can be solved numerically in general or analytically in some simplified cases. The approach has been demonstrated through two examples and the results are compared favorably to those from analytical solutions or numerical finite difference methods with perturbed model domains, while numerical shortcomings of the finite difference method are avoided.

Sensitivity analysis is essential for understanding how changes in design variables affect system performance. Numerical methods for calculating sensitivities, such as finite difference methods, are often easy to implement but can suffer from high computational costs and limited accuracy. Continuum Sensitivity Analysis (CSA) is an alternative approach for calculating analytical derivatives with respect to shape or value parameters. It is easy to implement and can be as accurate as conventional analytical sensitivity methods. By employing Spatial Gradient Reconstruction (SGR), continuous sensitivity equations can be solved in a nonintrusive manner. This method has been applied and validated on a wide range of problems. This work aims to extend the range of applicability of CSA to composite laminates. Composite materials may be characterized by several cross-coupling between loads and deformations, which leads to complex stress fields. A general formulation is introduced that applies to any plate-discretized Finite Element (FE) problem. The developed methodology utilizes the plate elements’ resultant forces and moments instead of the stresses to reconstruct spatial gradients and apply boundary conditions. Because of that, the method does not depend on the material type or lamination sequence and is also computationally inexpensive. It can be used indifferently for isotropic or composite materials, with any lamination sequence. This feature makes CSA attractive for classical FE models used in design optimization problems. This novelty extends the range applicability of CSA to any possible plate-based structural problem.

Two of the most important requirements when using gradient-based optimization for fluid–structure interaction problems are efficiency and accuracy in calculating the sensitivities. As a result, analytical continuum sensitivity formulations are finding their place due to their lower cost and accuracy. However, the necessary computation of mesh sensitivities in shape optimization is becoming a bottleneck, especially when handling complex geometries. In this research, an algorithm based on continuum sensitivity analysis for structural shape design variables is developed. In the proposed method, regularized Heaviside functions are used to modify the properties of mesh cells. Therefore, mesh dependency is removed from the sensitivity equations. This method does not require moving or modifying the mesh to handle the shape changes. This method is applied to two different validation cases where continuum sensitivity equations are formulated and solved. Validation cases are selected as a structural and thermal problem to show the applicability of this method for different physics. The results demonstrate the efficiency of the proposed algorithm for design-optimization framework.

A hybrid global–local approach for optimization of injection gate locations in liquid composite molding process simulations

The one- and two-dimensional inviscid Euler equations are formulated in an integro-differential form for shape design sensitivity analysis and optimization. The principal tool employed to derive the performance derivative sensitivity equations is the variational method, which is a continuous alternative to the discrete sensitivity analysis. Along with the sensitivity equations, the co-state equations and their boundary conditions are derived. Thereafter, based on the modal analysis of Von Neumann theory, the stability limits of the co-state equations are investigated. The stability characteristics of the co-state equations are compared with those of the Euler equations. Finally, using the criteria from the stability analysis of the state and co-state equations, some results of shape design optimization and sensitivity analysis for both quasi one- and two-dimensional Euler equations are presented. Copyright © 1999 John Wiley & Sons, Ltd.

This paper considers the numerical solution of optimal control problems based on ODEs. We assume that an explicit Runge-Kutta method is applied to integrate the state equation in the context of a recursive discretization approach. To compute the gradient of the cost function, one may employ Automatic Differentiation (AD). This paper presents the integration schemes that are automatically generated when differentiating the discretization of the state equation using AD. We show that they can be seen as discretization methods for the sensitivity and adjoint differential equation of the underlying control problem. Furthermore, we prove that the convergence rate of the scheme automatically derived for the sensitivity equation coincides with the convergence rate of the integration scheme for the state equation. Under mild additional assumptions on the coefficients of the integration scheme for the state equation, we show a similar result for the scheme automatically derived for the adjoint equation. Numerical results illustrate the presented theoretical results.

CAN-PINN: A fast physics-informed neural network based on coupled-automatic–numerical differentiation method

In this thesis we analyze and develop methods based on model order reduction (MOR) for the solution of optimization problems constrained by time-dependent partial differential equations (PDEs). The methods combine a direct solution approach with model reduction via proper orthogonal decomposition (POD) and the discrete empirical interpolation method (DEIM). The reduced-order models (ROMs) are used to approximate the high-dimensional dynamic systems originating from a spatial discretization of a PDE. However, when used in an optimization algorithm, conventional POD/DEIM ROMs often lack the ability to give adequate approximations of the gradient. We propose methods for a suitable enhancement of the ROMs for the optimization purpose which are based on the inclusion of derivative information in the POD and DEIM subspaces. We distinguish two types of error between quantities evaluated with the high-dimensional model and its ROM approximation in dependency on the optimization variable q: The reconstruction error which is evaluated with the same q0 which is used constructing the ROM and the prediction error which assesses approximations at q with a ROM constructed at q0 different to q. The novel reconstruction results we present include estimates for solutions of the adjoint equation and the sensitivity equations as well as for the gradient of the objective function. Based on the estimates we explain how the POD and DEIM bases should be extended with either adjoint or sensitivity information. The enhanced ROMs allow control of the reconstruction error for the objective and its gradient up to machine precision. Moreover, we propose a POD prediction estimate for the objective of the optimization problem in a neighborhood of q where the ROM is constructed. In case of sensitivity-extended POD and DEIM bases we give an analogous result for solutions of the states. The derivative-extended ROMs are then used to develop adaptive algorithms for the solution of optimal control and parameter estimation problems which results in great runtime improvements for the optimization while ensuring high approximation quality of the solution of the original problem. For the parameter estimation case a novel a posteriori error estimate is proposed which assesses the quality of suboptimal solutions obtained with the ROM. A further fundamental contribution is a discussion of discretize-then-optimize (DTO) vs. optimize-then-discretize (OTD) approaches in the context of MOR for optimization. We analyze advantages and disadvantages of both approaches and discuss to which extent our methods exhibit properties of either strategy. We also give examples of representative optimization problems in which standard POD/DEIM ROMs show an inacceptable behavior and can be successfully solved by derivative-extended ROMs. We have further implemented the developed methods emphasizing an efficient realization which is important for the investigation of the MOR potential. We showcase the practical performance of the proposed algorithms and the superiority of derivative-extended over conventional ROMs on two academic and one industry-relevant application which exhibit a variety of challenges for the model reduction approach in optimization.

In this paper, we propose an extrinsic approach based on physics-informed neural networks (PINNs) for solving the partial differential equations (PDEs) on surfaces embedded in high dimensional space. PINNs are one of the deep learning-based techniques. Based on the training data and physical models, PINNs introduce the standard feedforward neural networks (NNs) to approximate the solutions to the PDE systems. Using automatic differentiation, the PDEs information could be encoded into NNs and a loss function. To deal with the surface differential operators in the loss function, we combine the extrinsic approach with PINNs and then express that loss function in extrinsic form. Subsequently, the loss function could be minimized extrinsically with respect to the NN parameters. Numerical results demonstrate that the extrinsic approach based on PINNs for surface problems has good accuracy and higher efficiency compared with the embedding approach based on PINNs. In addition, the strong nonlinear mapping ability of NNs makes this approach robust in solving time-dependent nonlinear problems on more complex surfaces.

The topic for the current Mini-Workshop organized by Karl Kunisch (Univ. Graz), Angela Kunoth (Univ. Bonn) and Rolf Rannacher (Univ. Heidelberg) emerged from the Oberwolfach Workshop “Numerical Techniques for Optimization Problems with PDE Constraints” which was held February 16–22, 2003. It was realized that numerically solving control problems which are constrained by time-dependent nonlinear PDEs (Partial Differential Equations) are particularly challenging with respect to the complexity of the problem. Mathematically, one has to minimize a functional under PDE constraints and possibly additional constraints on the state and the control. Standard discretizations on uniform grids in space and time will only yield solutions where the inherent structures of the problem (nonlinearity, constraints) are not sufficiently captured. Certain optimization problems for large coupled systems of partial differential equations are currently not yet numerically treatable or do not satisfy the time constraints required in practice. Overcoming this barrier can only be achieved by designing new mathematically founded algorithmic approaches. The road towards this goal leads to many interesting problems in optimization, linear algebra, numerics, analysis, and approximation theory. The conference had 21 participants which represented continuous optimization, numerical analysis and scientific computing. 18 talks were given. The 10 longer, overview-style talks were on optimization with PDEs, focussing on These talks were intended to bridge the gap between the different research fields Optimization and Numerics. They were complemented by 8 shorter talks on more specialized research topics, ranging from efficiency indices for optimization over iterative methods for the coupled systems and multigrid acceleration to modelling issues in optimization, and data compression by means of proper orthogonal decompositions and central Voronoi tesselations. Different modern approaches to overcome the complexity issues in numerical simulations for PDE-constrained optimization have been presented and discussed. One of the approaches is to employ fast iterative solvers like multigrid on uniform grids. The methodology which conceptually provides the largest potential is to introduce adaptivity. This drastically reduces complexity but depending on the context may require solving an additional problem. Wavelet approaches particularly allow to resolve each of the variables separately and in addition provide a built-in preconditioning. Yet another approach uses compressed information in order to efficiently solve the primal-dual system. All these issues are currently very active research areas. The extensive discussions held during this workshop have produced a number of new ideas and connections. It was agreed upon that a mere black-box-style matching of efficient PDE codes with optimization tool boxes would on one hand remain much below its potential and on the other hand not help overcoming complexity barriers. Some concepts from automatic differentiation seem to carry over to adaptive methods. Even combining adaptivity with proper orthogonal decompositions may be a very promising direction. The many new ideas discussed during this workshop will have to be further elaborated in future.

This document summarizes the findings of a strategic planning exercise commissioned by the Weapons Simulation and Computing, Computational Physics (WSC/CP) program at the Lawrence Livermore National Laboratory (LLNL) in FY24. During the year, the committee met with multiple stakeholder communities to gather input, opinions, suggestions and concerns which have been incorporated throughout this document. The key findings from this exercise are summarized: • The development of multiphysics modelling and simulation (mod/sim) codes and software technologies, their deployment on exascale compute platforms, and their broad adoption across the NNSA is a major success of the Advanced Simulation and Computing (ASC) program and the Exascale Computing Project (ECP). Sustained investment in these core technologies is essential. • Today’s state of the art involves running ensembles of O(100K) simulations to perform uncertainty quantification (UQ) and design studies using multiple statistical methods such as Bayesian optimization to understand sensitivities of our models and explore parameterized design spaces. Even with exascale computing, we are practically limited to O(10) parameters in these studies since the number of simulations required to sample the space scales exponentially with the number of design parameters. • The data from these simulation ensembles is increasingly being used to train machine learned (ML) surrogates (or reduced order models, ROMs) which can then be used for optimization or real time design exploration. However, the trained surrogates are still limited in the number of parameters they can represent due to the sampling limitations previously noted. • Augmenting our suite of integrated multiphysics simulation codes, both current and emerging, with the ability to compute gradients (solution derivatives) of arbitrary simulation outputs with respect to (some or all) simulation inputs would be a breakthrough technology, opening the door to a new era of efficient and automated inverse design based on verified and validated mod/sim capabilities. • This capability, which we refer to as differentiable multiphysics codes (DMCs), would revolutionize both UQ and optimization studies by breaking the curse of dimensionality that presently limits our “gradient-free” ensemble based computing approach. A similar breakthrough occurred in the AI/ML community once the ability to compute gradients of arbitrary loss functions using back-propagation became commonplace. Gradient information from the multiphysics codes can also be used to dramatically improve the efficiency and scale of training of ML/ROM surrogates for rapid assessments. • Achieving this in our suite of codes will be a grand challenge, similar to the amount of effort that was required to transition from CPU to GPU computing. It will require buy-in from the entire WSC/CP program and beyond, including all integrated codes, physics and engineering models, third-party library dependencies and performance portability abstractions. It will also require investment in research and development of numerical methods for computing adjoints of coupled physics across multiple adaptively refined moving meshes and of stochastic (Monte Carlo) and mesh free (SPH) methods. • New software and numerical techniques, largely pioneered by the AI/ML community, make this feasible. Chief among these is automatic differentiation (AD), the ability to employ AD at point-wise locations in a physics calculation (instead of traditional black-box approaches) and the ability to perform “back-propagation in time” (or reverse mode AD) for non-linear partial differential equations (PDEs). Fundamentally, the conclusion of this strategic planning exercise is that the time is right to undertake a large scale effort in WSC, centered on the existing integrated codes, to continue the natural evolution of mod/sim in the age of AI/ML. Instead of attempting to replace mod/sim with purely data driven AI/ML models, we believe the key to success is to integrate AI/ML by building on top of the decades of hard-won knowledge and the verified/validated multiphysics modelling capability that is the hallmark of the ASC program.

During the product development processes, components and subsystems go through test-fix-test cycles until the final configurations surpass the design life to exceed the reliability goals. When failure modes are uncovered before reaching the design life, design configurations are improved to eliminate failures. Early failures can be mixed with several failure modes commonly lead to lower shape parameters. Systems that are mature that survive close to design life may have higher and steeper slope parameters. During the component-subsystem development process and design maturity, shape parameter changes associated with reliability growth are illustrated with examples. As designs reach or surpass design life with design maturity, failure modes can be single out to unique shape parameters. Separating different shapes and patterns of Weibull curves help identifying different failure modes thereby leading to failure mode-wise reliability analysis. Such analysis help prioritizing the fixing process of each failure mode based on returns on investment.On the other hand, if a product was launched before it reaches the design life and meets reliability goals, field failures become inevitable. Such field failures can also be modeled with Weibull Analysis. Parts that fail short of the design life are commonly called under-developed designs. Such designs can be mixed with multiple failure modes. Different shapes and shift of curves can be strategically isolated for separating design weaknesses and investigations before catastrophic failures erase evidence of root causes. For example, shape parameters and patterns of underdeveloped parts can be helpful in isolating manufacturing issues from design issues. This paper addresses how to perform data analysis in order to strategically enhance the further use of Weibull plots, its shapes and pattern recognition for speedier product development and customer satisfaction.

An efficient implicit high-order flux reconstruction/correction procedure via reconstruction method (FR/CPR) is developed to solve the compressible Navier-Stokes equation on unstructured hexahedral grids. The convergence rate of the proposed solver is greatly improved compared to Lower-upper Symmetry Gauss Seidel (LU-SGS) method. The restart GMRES (restarted Generalized Minimal Residual) is used to tackle the linear system by FR/CPR temporal and spatial discretization, and an automatic differential (AD) by operator overloading is proposed to construct the Jacobian matrix of the discretization system. The effects of automatic differential on the convergence rate of the developed implicit scheme have been investigated. Inviscid and viscous test cases are performed to evaluate the performance of the developed implicit solver.