Asymptotic Distribution of Stochastic Mirror Descent Iterates in Average Ensemble Models

Relative smoothness—a notion introduced in Birnbaum et al. (Proceedings of the 12th ACM conference on electronic commerce, ACM, pp 127–136, 2011) and recently rediscovered in Bauschke et al. (Math Oper Res 330–348, 2016) and Lu et al. (Relatively-smooth convex optimization by first-order methods, and applications, arXiv:1610.05708 , 2016)—generalizes the standard notion of smoothness typically used in the analysis of gradient type methods. In this work we are taking ideas from well studied field of stochastic convex optimization and using them in order to obtain faster algorithms for minimizing relatively smooth functions. We propose and analyze two new algorithms: Relative Randomized Coordinate Descent (relRCD) and Relative Stochastic Gradient Descent (relSGD), both generalizing famous algorithms in the standard smooth setting. The methods we propose can be in fact seen as particular instances of stochastic mirror descent algorithms, which has been usually analyzed under stronger assumptions: Lipschitzness of the objective and strong convexity of the reference function. As a consequence, one of the proposed methods, relRCD corresponds to the first stochastic variant of mirror descent algorithm with linear convergence rate.

The usual approach to developing and analyzing first-order methods for nonsmooth (stochastic or deterministic) convex optimization assumes that the objective function is uniformly Lipschitz continuous with parameter Mf. However, in many settings, the nondifferentiable convex function f is not uniformly Lipschitz continuous—for example, (i) the classical support vector machine problem, (ii) the problem of minimizing the maximum of convex quadratic functions, and even (iii) the univariate setting with [Formula: see text]. Herein, we develop a notion of “relative continuity” that is determined relative to a user-specified “reference function” h (that should be computationally tractable for algorithms), and we show that many nondifferentiable convex functions are relatively continuous with respect to a correspondingly fairly simple reference function h. We also similarly develop a notion of “relative stochastic continuity” for the stochastic setting. We analyze two standard algorithms—the (deterministic) mirror descent algorithm and the stochastic mirror descent algorithm—for solving optimization problems in these new settings, providing the first computational guarantees for instances where the objective function is not uniformly Lipschitz continuous. This paper is a companion paper for nondifferentiable convex optimization to the recent paper by Lu et al. [Lu H, Freund RM, Nesterov Y (2018) Relatively smooth convex optimization by first-order methods, and applications. SIAM J. Optim. 28(1): 333–354.], which developed analogous results for differentiable convex optimization.

The curse-of-dimensionality taxes computational resources heavily with exponentially increasing computational cost as the dimension increases. This poses great challenges in solving high-dimensional partial differential equations (PDEs), as Richard E. Bellman first pointed out over 60 years ago. While there has been some recent success in solving numerical PDEs in high dimensions, such computations are prohibitively expensive, and true scaling of general nonlinear PDEs to high dimensions has never been achieved. We develop a new method of scaling up physics-informed neural networks (PINNs) to solve arbitrary high-dimensional PDEs. The new method, called Stochastic Dimension Gradient Descent (SDGD), decomposes a gradient of PDEs’ and PINNs’ residual into pieces corresponding to different dimensions and randomly samples a subset of these dimensional pieces in each iteration of training PINNs. We prove theoretically the convergence and other desired properties of the proposed method. We demonstrate in various diverse tests that the proposed method can solve many notoriously hard high-dimensional PDEs, including the Hamilton–Jacobi-Bellman (HJB) and the Schrödinger equations in tens of thousands of dimensions very fast on a single GPU using the PINNs mesh-free approach. Notably, we solve nonlinear PDEs with nontrivial, anisotropic, and inseparable solutions in less than one hour for 1000 dimensions and in 12 h for 100,000 dimensions on a single GPU using SDGD with PINNs. Since SDGD is a general training methodology of PINNs, it can be applied to any current and future variants of PINNs to scale them up for arbitrary high-dimensional PDEs.

: Complex physical phenomena involving chemically reacting systems or the transport of heat or fluids are often modeled by coupled systems of time-dependent, nonlinear partial differential equations. The difficulties in understanding the stability of the differential equation systems and in designing efficient, accurate numerical methods for their solution are widely recognized and were the focus of this research. We have worked on four general aspects of the analysis and numerical approximation of systems of partial differential equations. These areas of research are: (1) modeling aspects and stability analysis for nonlinear time-dependent partial differential equations; (2) use and analysis of finite element or finite difference methods to discretize coupled systems of nonlinear differential equations; (3) development of adaptive or local grid refinement capabilities to resolve local phenomena in large-scale applications; and (4) development of data structures, preconditioners, and efficient solution algorithms for large-scale problems on new computer architectures. Emphasis has been placed upon multiphase or multicomponent, transport-dominated flow processes with dynamic local phenomena. The research also involved a mix of analysis, algorithm development, and large-scale computation using newer computer architectures.

If we follow an asexually reproducing population through time, then the amount of time that has passed since the most recent common ancestor (MRCA) of all current individuals lived will change as time progresses. The resulting “MRCA age” process has been studied previously when the population has a constant large size and evolves via the diffusion limit of standard Wright–Fisher dynamics. For any population model, the sample paths of the MRCA age process are made up of periods of linear upward drift with slope +1 punctuated by downward jumps. We build other Markov processes that have such paths from Poisson point processes on ℝ++×ℝ++ with intensity measures of the form λ⊗μ where λ is Lebesgue measure, and μ (the “family lifetime measure”) is an arbitrary, absolutely continuous measure satisfying μ((0, ∞))=∞ and μ((x, ∞))<∞ for all x>0. Special cases of this construction describe the time evolution of the MRCA age in (1+β)-stable continuous state branching processes conditioned on nonextinction—a particular case of which, β=1, is Feller’s continuous state branching process conditioned on nonextinction. As well as the continuous time process, we also consider the discrete time Markov chain that records the value of the continuous process just before and after its successive jumps. We find transition probabilities for both the continuous and discrete time processes, determine when these processes are transient and recurrent and compute stationary distributions when they exist. Moreover, we introduce a new family of Markov processes that stands in a relation with respect to the general (1+β)-stable continuous state branching process and its conditioned version that is similar to the one between the family of Bessel-squared diffusions and the unconditioned and conditioned Feller continuous state branching process.

SONets: Sub-operator learning enhanced neural networks for solving parametric partial differential equations

Nonlinear Stabilization in Infinite Dimension

This paper develops and validates through a series of presentable examples, a comprehensive high-precision, and ultrafast computing concept for solving nonlinear ordinary differential equations (ODEs) and partial differential equations (PDEs) with cellular neural networks (CNN). The core of this concept is a straightforward scheme that we call "nonlinear adaptive optimization (NAOP),” which is used for a precise template calculation for solving nonlinear ODEs and PDEs through CNN processors. One of the key contributions of this work is to demonstrate the possibility of transforming different types of nonlinearities displayed by various classical and well-known nonlinear equations (e.g., van der Pol-, Rayleigh-, Duffing-, Rössler-, Lorenz-, and Jerk-equations, just to name a few) unto first-order CNN elementary cells, and thereby enabling the easy derivation of corresponding CNN templates. Furthermore, in the case of PDE solving, the same concept also allows a mapping unto first-order CNN cells while considering one or even more nonlinear terms of the Taylor's series expansion generally used in the transformation of a PDE in a set of coupled nonlinear ODEs. Therefore, the concept of this paper does significantly contribute to the consolidation of CNN as a universal and ultrafast solver of nonlinear ODEs and/or PDEs. This clearly enables a CNN-based, real-time, ultraprecise, and low-cost computational engineering. As proof of concept, two examples of well-known ODEs are considered namely a second-order linear ODE and a second order nonlinear ODE of the van der Pol type. For each of these ODEs, the corresponding precise CNN templates are derived and are used to deduce the expected solutions. An implementation of the concept developed is possible even on embedded digital platforms (e.g., field programmable gate array (FPGA), digital signal processor (DSP), graphics processing unit (GPU), etc.). This opens a broad range of applications. Ongoing works (as outlook) are using NAOP for deriving precise templates for a selected set of practically interesting ODEs and PDEs equation models such as Lorenz-, Rössler-, Navier Stokes-, Schrödinger-, Maxwell-, etc.

An elegant and powerful technique is Homotopy Perturbation Method (HPM) to solve linear and nonlinear ordinary and partial differential equations. The method, which is a coupling of the traditional perturbation method and homotopy in topology, deforms continuously to a simple problem which can be solved easily. The method does not depend upon a small parameter in the equation. Using the initial conditions this method provides an analytical or exact solution. From the calculation and its graphical representation it is clear that how the solution of the original equation and its behavior depends on the initial conditions. Therefore there have been attempts to develop new techniques for obtaining analytical solutions which reasonably approximate the exact solutions. Many problems in natural and engineering sciences are modeled by nonlinear partial differential equations (NPDEs). The theory of nonlinear problem has recently undergone much study. Nonlinear phenomena have important applications in applied mathematics, physics, and issues related to engineering. In this paper we have applied this method to Burger’s equation and an example of highly nonlinear partial differential equation to get the most accurate solutions. The final results tell us that the proposed method is more efficient and easier to handle when is compared with the exact solutions or Adomian Decomposition Method (ADM).

SIR models with directed diffusions are important in describing the population movement. However, efficient numerical simulations of such systems of fully nonlinear second order partial differential equations (PDEs) are challenging. They are often mixed type PDEs with ill-posed or degenerate components. The solutions may develop singularities along with time evolution. Stiffness due to nonlinear diffusions in the system gives strict constraints in time step sizes for numerical methods. In this paper, we design efficient Krylov implicit integration factor (IIF) Weighted Essentially Non-Oscillatory (WENO) method to solve SIR models with directed diffusions. Numerical experiments are performed to show the good accuracy and stability of the method. Singularities in the solutions are resolved stably and sharply by the WENO approximations in the scheme. Unlike a usual implicit method for solving stiff nonlinear PDEs, the Krylov IIF WENO method avoids solving large coupled nonlinear algebraic systems at every time step. Large time step size computations are achieved for solving the fully nonlinear second-order PDEs, namely, the time step size is proportional to the spatial grid size as that for solving a pure hyperbolic PDE. Two biologically interesting cases are simulated by the developed scheme to study the finite-time blow-up time and location or discontinuity locations in the solution of the SIR model.

In this work, a novel quantum Fourier ordinary differential equation (ODE) solver is proposed to solve both linear and nonlinear partial differential equations (PDEs). Traditional quantum ODE solvers transform a PDE into an ODE system via spatial discretization and then integrate it, thereby converting the task of solving the PDE into computing the integral for the driving function f(x). These solvers rely on the quantum amplitude estimation algorithm, which requires the driving function f(x) to be within the range of [0, 1] and necessitates the construction of a quantum circuit for the oracle R that encodes f(x). This construction can be highly complex, even for simple functions like f(x) = x. An important exception arises for the specific case of f(x) = sin2(mx + c), which can be encoded more efficiently using a set of Ry rotation gates. To address these challenges, we expand the driving function f(x) as a Fourier series and propose the quantum Fourier ODE Solver. This approach not only simplifies the construction of the oracle R but also removes the restriction that f(x) must lie within [0,1]. The proposed method was evaluated by solving several representative linear and nonlinear PDEs, including the Navier–Stokes (N–S) equations. The results show that the quantum Fourier ODE solver produces results that closely match both analytical and reference solutions.

Stochastic differential equation (SDE) models are the foundation for pricing and hedging financial derivatives. The drift and volatility functions in SDE models are typically chosen to be algebraic functions with a small number ( < 5 ) of parameters which can be calibrated to market data. A more flexible approach is to use neural networks to model the drift and volatility functions, which provides more degrees-of-freedom to match observed market data. Training of models requires optimizing over an SDE, which is computationally challenging. For European options, we develop a fast stochastic gradient descent (SGD) algorithm for training the neural network-SDE model. Our SGD algorithm uses two independent SDE paths to obtain an unbiased estimate of the direction of steepest descent. For American options, we optimize over the corresponding Kolmogorov partial differential equation (PDE). The neural network appears as coefficient functions in the PDE. Models are trained on large datasets (many contracts), requiring either large simulations (many Monte Carlo samples for the stock price paths) or large numbers of PDEs (a PDE must be solved for each contract). Numerical results are presented for real market data including S&P 500 index options, S&P 100 index options, and single-stock American options. The neural-network-based SDE models are compared against the Black-Scholes model, the Dupire's local volatility model, and the Heston model. Models are evaluated in terms of how accurate they are at pricing out-of-sample financial derivatives, which is a core task in derivative pricing at financial institutions. Specifically, we calibrate a neural network-SDE model to market data for a financial derivative on an asset with price S t with a payoff function g ( s ) , and we then evaluate its generalization accuracy for a financial derivative on the same asset S t but with a different payoff function f ( s ) . In addition to comparing out-of-sample pricing accuracy, we evaluate the hedging performance of the neural network-SDE model.

The purpose of this study is to develop a numerical method for bifurcation analysis of nonlinear partial differential equations, based on the reduction of partial differential equations to ordinary ones, using the Kolmogorov-Arnold theorem. Methods. This paper describes a method for reducing partial differential equations to ordinary ones using the Kolmogorov-Arnold theorem, as well as methods for the bifurcation analysis of nonlinear boundary value problems for ordinary differential equations. Results. The paper presents a new method for solving and bifurcation analysis of nonlinear boundary value problems for partial differential equations, which allow variational formulation. The method was applied to a nonlinear two-dimensional Bratu problem with Dirichlettype boundary conditions. Conclusion. A new method of bifurcation analysis for nonlinear partial differential equations has been developed. Specifically, a method has been proposed for reducing partial different equations to ordinary equations, which allows the use of the developed apparatus of bifurcation analysis for boundary value problems of ordinary differential equations. This method allows conducting bifurcation analysis for arbitrary nonlinear partial differential equations.

Type-II hidden symmetries through weak symmetries for nonlinear partial differential equations

A recursion operator is an integro-differential operator which maps a generalized symmetry of a nonlinear partial differential equation (PDE) to a new symmetry. Therefore, the existence of a recursion operator guarantees that the PDE has infinitely many higher-order symmetries, which is a key feature of complete integrability. Completely integrable nonlinear PDEs have a bi-Hamiltonian structure and a Lax pair; they can also be solved with the inverse scattering transform and admit soliton solutions of any order.A straightforward method for the symbolic computation of polynomial recursion operators of nonlinear PDEs in (1+1) dimensions is presented. Based on conserved densities and generalized symmetries, a candidate recursion operator is built from a linear combination of scaling invariant terms with undetermined coefficients. The candidate recursion operator is substituted into its defining equation and the resulting linear system for the undetermined coefficients is solved.The method is algorithmic and is implemented in Mathematica. The resulting symbolic package PDERecursionOperator.m can be used to test the complete integrability of polynomial PDEs that can be written as nonlinear evolution equations. With PDERecursionOperator.m, recursion operators were obtained for several well-known nonlinear PDEs from mathematical physics and soliton theory.