Decreasing Entropic Regularization Averaged Gradient for Semi-Discrete Optimal Transport

In machine learning, optimal transport (OT) theory is extensively utilized to compare probability distributions across various applications, such as graph data represented by node distributions and image data represented by pixel distributions. In practical scenarios, it is often necessary to solve multiple OT problems. Traditionally, these problems are treated independently, with each OT problem being solved sequentially. However, the computational complexity required to solve a single OT problem is already substantial, making the resolution of multiple OT problems even more challenging. Although many applications of fast solutions to OT are based on the premise of a single OT problem with arbitrary distributions, few efforts handle such multiple OT problems with multiple distributions. Therefore, we propose the anchor space OT (ASOT) problem: an approximate OT problem designed for multiple OT problems. This proposal stems from our finding that in many tasks the mass transport tends to be concentrated in a reduced space from the original feature space. By restricting the mass transport to a learned anchor point space, ASOT avoids pairwise instantiations of cost matrices for multiple OT problems and simplifies the problems by canceling insignificant transports. This simplification greatly reduces its computational costs. We then prove the upper bounds of its 1-Wasserstein distance error between the proposed ASOT and the original OT problem under different conditions. Building upon this accomplishment, we propose three methods to learn anchor spaces for reducing the approximation error. Furthermore, our proposed methods present great advantages for handling distributions of different sizes with GPU parallelization.

Unbalanced Optimal Transport, from theory to numerics

Optimal partial mass transportation and obstacle Monge–Kantorovich equation

This paper focuses on multi-block optimization problems over transport polytopes, which underlie various applications including strongly correlated quantum physics and machine learning. Conventional block coordinate descent-type methods for the general multi-block problems store and operate on the matrix variables directly, resulting in formidable expenditure for large-scale settings. On the other hand, optimal transport problems, as a special case, have attracted extensive attention and numerical techniques that waive the use of the full matrices have recently emerged. However, it remains nontrivial to apply these techniques to the multi-block, possibly nonconvex problems with theoretical guarantees. In this work, we leverage the benefits of both sides and develop novel sampling-based block coordinate descent-type methods, which are equipped with either entropy regularization or Kullback-Leibler divergence. Each iteration of these methods solves subproblems restricted on the sampled degrees of freedom. Consequently, they involve only sparse matrices, which amounts to considerable complexity reductions. We explicitly characterize the sampling-induced errors and establish convergence and asymptotic properties for the methods equipped with the entropy regularization. Numerical experiments on typical strongly correlated electron systems corroborate their superior scalability over the methods utilizing full matrices. The advantage also enables the first visualization of approximate optimal transport maps between electron positions in three-dimensional contexts.

Relatively recently, there has been much activity on two particular generalizations of the classical two-marginal optimal transport problem. The first is the partial transport problem, where the total mass of the two distributions to be coupled may not match, and one is forced to choose submeasures of the constraints for coupling. The other generalization is the multi-marginal transport problem, where there are 3 or more probability distributions to be coupled together in some optimal manner. By combining the above two generalizations we obtain a natural extension: the multi-marginal optimal partial transport problem. In joint work with Brendan Pass (University of Alberta), we have obtained uniqueness of solutions (under hypotheses analogous to the two-marginal partial transport problem given by Figalli) by relating the problem to what we deem the “partial barycenter problem” for finite measures. Interestingly enough, we also observe some significantly different behavior of solutions compared to the two marginal case.

The optimal transport problem has recently developed into a powerful framework for various applications in estimation and control. Many of the recent advances in the theory and application of optimal transport are based on regularizing the problem with an entropy term, which connects it to the Schrödinger bridge problem and thus to stochastic optimal control. Moreover, the entropy regularization makes the otherwise computationally demanding optimal transport problem feasible even for large scale settings. This has led to an accelerated development of optimal transport based methods in a broad range of fields. Many of these applications have an underlying graph structure, for instance, information fusion and tracking problems can be described by trees. In this work we consider multimarginal optimal transport problems with a cost function that decouples according to a tree structure. The entropy regularized multimarginal optimal transport problem can be viewed as a generalization of the Schrödinger bridge problem with the same tree-structure, and by utilizing these connections we extend the computational methods for the classical optimal transport problem in order to solve structured multimarginal optimal transport problems in an efficient manner. In particular, the algorithm requires only matrix-vector multiplications of relatively small dimensions. We show that the multimarginal regularization introduces less diffusion, compared to the commonly used pairwise regularization, and is therefore more suitable for many applications. Numerical examples illustrate this, and we finally apply the proposed framework for the tracking of an ensemble of indistinguishable agents.

We consider the problem of optimal transport where the cost function is given by a \(\mathcal {D}^{(\alpha )}_\varPsi \)-divergence for some convex function \(\varPsi \) [21], where \(\alpha = \pm 1\) gives the Bregman divergence. For costs of this form, we introduce a new complex geometric interpretation of the optimal transport problem by considering an induced Sasaki metric on the tangent bundle of the domain of \(\varPsi \). In this framework, the Ma-Trudinger-Wang (MTW) tensor [12] is proportional to the orthogonal bisectional curvature. This geometric framework for optimal transport is complementary to the pseudo-Riemannian approach of Kim and McCann [10].

Optimal transport (OT) and Gromov-Wasserstein (GW) alignment are powerful frameworks for geometrically driven matching of probability distributions, yet their large-scale usage is hampered by high statistical and computational costs. Entropic regularization has emerged as a promising solution, allowing parametric convergence rates via the plug-in estimator, which can be computed using the Sinkhorn algorithm (or its iterations in the GW case). However, Sinkhorn's $O(n^2)$ time complexity for an $n$-sized dataset becomes prohibitive for modern, massive datasets. In this work, we propose a new computational framework for the entropic OT and GW problems that replaces the Sinkhorn step with a neural network trained via backpropagation on mini-batches. By shifting the computational load from the entire dataset to the mini-batch, our approach enables reliable estimation of both the optimal transport/alignment cost and plan at dataset sizes and dimensions far exceeding those tractable with standard Sinkhorn methods. We derive non-asymptotic error bounds for these estimates, showing they achieve minimax-optimal parametric convergence rates for compactly supported distributions. Numerical experiments confirm the accuracy of our method in high-dimensional, large-sample regimes where Sinkhorn is infeasible.

Nous analysons le probleme du transport optimal entre deux mesures aleatoires et equivariantes et demontrons des conditions qui garantissent l’existence d’une solution de type Monge. En outre, nous demontrons que l’equivariance apparait naturellement dans ce contexte en prouvant que les couplages optimaux classiques dans des ensembles bornes convergent vers le couplage optimal dans tout l’espace. Finalement nous demontrons des conditions suffisantes pour que le cout au sens $L^{p}$ soit fini en introduisant une metrique appropriee.

Given a determinate (multivariate) probability measure μ, we characterize Gaussian mixtures ν ϕ which minimize the Wasserstein distance W 2 (μ,ν ϕ ) to μ when the mixing probability measure ϕ on the parameters (m,Σ) of the Gaussians is supported on a compact set S. (i) We first show that such mixtures are optimal solutions of a particular optimal transport (OT) problem where the marginal ν ϕ of the OT problem is also unknown via the mixing measure variable ϕ. Next (ii) by using a well-known specific property of Gaussian measures, this optimal transport is then viewed as a Generalized Moment Problem (GMP) and if the set S of mixture parameters (m,Σ) is a basic compact semi-algebraic set, we provide a “mesh-free” numerical scheme to approximate as closely as desired the optimal distance by solving a hierarchy of semidefinite relaxations of increasing size. In particular, we neither assume that the mixing measure is finitely supported nor that the variance is the same for all components. If the original measure μ is not a Gaussian mixture with parameters (m,Σ)∈S, then a strictly positive distance is detected at a finite step of the hierarchy. If the original measure μ is a Gaussian mixture with parameters (m,Σ)∈S, then all semidefinite relaxations of the hierarchy have same zero optimal value. Moreover if the mixing measure is atomic with finite support, its components can sometimes be extracted from an optimal solution at some semidefinite relaxation of the hierarchy when Curto & Fialkow’s flatness condition holds for some moment matrix.

We consider in this work a system of two stochastic differential equations named the perturbed compositional gradient flow. By introducing a separation of fast and slow scales of the two equations, we show that the limit of the slow motion is given by an averaged ordinary differential equation. We then demonstrate that the deviation of the slow motion from the averaged equation, after proper rescaling, converges to a stochastic process with Gaussian inputs. This indicates that the slow motion can be approximated in the weak sense by a standard perturbed gradient flow or the continuous-time stochastic gradient descent algorithm that solves the optimization problem for a composition of two functions. As an application, the perturbed compositional gradient flow corresponds to the diffusion limit of the Stochastic Composite Gradient Descent (SCGD) algorithm for minimizing a composition of two expected-value functions in the optimization literatures. For the strongly convex case, such an analysis implies that the SCGD algorithm has the same convergence time asymptotic as the classical stochastic gradient descent algorithm. Thus it validates, at the level of continuous approximation, the effectiveness of using the SCGD algorithm in the strongly convex case.

A methodology is developed for the numerical solution to the sample‐based optimal transport and Wasserstein barycenter problems. The procedure is based on a characterization of the barycenter and of the McCann interpolants that permits the decomposition of the global problem under consideration into various local problems where the distance among successive distributions is small. These local problems can be formulated in terms of feature functions and shown to have a unique minimizer that solves a nonlinear system of equations. Both the theoretical underpinnings of the methodology and its practical implementation are developed, and illustrated with synthetic and real data sets. © 2019 Wiley Periodicals, Inc.

The objective of this paper is to develop a duality between a novel entropy martingale optimal transport (EMOT) problem and an associated optimisation problem. In EMOT, we follow the approach taken in the entropy optimal transport (EOT) problem developed in Liero et al. (Invent. Math. 211:969–1117, 2018), but we add the constraint, typical of martingale optimal transport (MOT) theory, that the infimum of the cost functional is taken over martingale probability measures. In the associated problem, the objective functional, related via Fenchel conjugacy to the entropic term in EMOT, is no longer linear as in (martingale) optimal transport. This leads to a novel optimisation problem which also has a clear financial interpretation as a nonlinear subhedging problem. Our theory allows us to establish a nonlinear robust pricing–hedging duality which also covers a wide range of known robust results. We also focus on Wasserstein-induced penalisations and study how the duality is affected by variations in the penalty terms, with a special focus on the convergence of EMOT to the extreme case of MOT.

Aligning electron density maps from Cryogenic electron microscopy (cryo-EM) is a first key step for studying multiple conformations of a biomolecule. As this step remains costly and challenging, with standard alignment tools being potentially stuck in local minima, we propose here a new procedure, called AlignOT, which relies on the use of computational optimal transport (OT) to align EM maps in 3D space. By embedding a fast estimation of OT maps within a stochastic gradient descent algorithm, our method searches for a rotation that minimizes the Wasserstein distance between two maps, represented as point clouds. We quantify the impact of various parameters on the precision and accuracy of the alignment, and show that AlignOT can outperform the standard local alignment methods, with an increased range of rotation angles leading to proper alignment. We further benchmark AlignOT on various pairs of experimental maps, which account for different types of conformational heterogeneities and geometric properties. As our experiments show good performance, we anticipate that our method can be broadly applied to align 3D EM maps.

We introduce a dual dynamical formulation for the optimal partial transport problem with Lagrangian costs cL(x,y) := ξ∈Lip([0,1];ℝN)inf {∫01 L(ξ(t), ξ˙(t))dt : ξ(0) = x, ξ(1) = y} based on a constrained Hamilton–Jacobi equation. Optimality condition is given that takes the form of a system of PDEs in some way similar to constrained mean field games. The equivalent formulations are then used to give numerical approximations to the optimal partial transport problem via augmented Lagrangian methods. One of advantages is that the approach requires only values of L and does not need to evaluate cL(x, y), for each pair of endpoints x and y, which comes from a variational problem. This method also provides at the same time active submeasures and the associated optimal transportation.