Accelerating variance-reduced stochastic gradient methods

Variance and bias reduction techniques for the harmonic gradient estimator

We provide the first theoretical analysis on the convergence rate of asynchronous mini-batch gradient descent with variance reduction (AsySVRG) for non-convex optimization. Asynchronous stochastic gradient descent (AsySGD) has been broadly used for deep learning optimization, and it is proved to converge with rate of O(1/\sqrt{T}) for non-convex optimization. Recently, variance reduction technique is proposed and it is proved to be able to accelerate the convergence of SGD greatly. It is shown that asynchronous SGD method with variance reduction technique has linear convergence rate when problem is strongly convex. However, there is still no work to analyze the convergence rate of this method for non-convex problem. In this paper, we consider two asynchronous parallel implementations of mini-batch gradient descent method with variance reduction: one is on distributed-memory architecture and the other is on shared-memory architecture. We prove that both methods can converge with a rate of O(1/T) for non-convex optimization, and linear speedup is accessible when we increase the number of workers. We evaluate our methods by optimizing multi-layer neural networks on two real datasets (MNIST and CIFAR-10), and experimental results demonstrate our theoretical analysis.

We implement the simple method to accelerate the convergence speed to the steady state and enhance the mixing rate to the stochastic gradient Langevin method. The ordinary stochastic gradient method is based on mini-batch learning for reducing the computational cost when the amount of data is extraordinary large. The stochasticity of the gradient can be mitigated by the injection of Gaussian noise, which yields the stochastic Langevin gradient method; this method can be used for Bayesian posterior sampling. However, the performance of the stochastic Langevin gradient method depends on the mixing rate of the stochastic dynamics. In this study, we propose violating the detailed balance condition to enhance the mixing rate. Recent studies have revealed that violating the detailed balance condition accelerates the convergence to a stationary state and reduces the correlation time between the samplings. We implement this violation of the detailed balance condition in the stochastic gradient Langevin method and test our method for a simple model to demonstrate its performance.

We analyze the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG method achieves a faster convergence rate than black-box SG methods. The convergence rate is improved from $$O(1/\sqrt{k})$$O(1/k) to O(1 / k) in general, and when the sum is strongly-convex the convergence rate is improved from the sub-linear O(1 / k) to a linear convergence rate of the form $$O(\rho ^k)$$O(?k) for $$\rho < 1$$?<1. Further, in many cases the convergence rate of the new method is also faster than black-box deterministic gradient methods, in terms of the number of gradient evaluations. This extends our earlier work Le Roux et al. (Adv Neural Inf Process Syst, 2012), which only lead to a faster rate for well-conditioned strongly-convex problems. Numerical experiments indicate that the new algorithm often dramatically outperforms existing SG and deterministic gradient methods, and that the performance may be further improved through the use of non-uniform sampling strategies.

We provide a thorough analysis of the effectiveness of different Variance Reduction Techniques (VRTs). We consider both stand-alone and combined applications of two input techniques, Antithetic Variates (AV) and Latin Hypercube Sampling (LHS), and two output techniques, Control Variates (CV) and Poststratified Sampling (PS). Previous research in the area mainly focuses on asymptotic variance reduction. In this experimental study, we measure the performance of VRTs under finite simulation run lengths and analyze their effects. Our findings show that the asymptotic variance reduction results do not readily apply to finite-length simulations. We consider three different types of systems (M/M/1, serial production line and (s, S) inventory control systems) and compare the VRTs under various experimental conditions. We observe that a variance reduction cannot be guaranteed for every instance a VRT is applied. Our results also indicate that the output VRTs (CV, PS) are better than input VRTs (AV, LHS) on the average for the single systems considered in this study. More interestingly, the less-sophisticated techniques (AV, CV) often perform better than the relatively more-complex techniques (LHS, PS). A comprehensive bibliography is also provided.

Stochastic optimization lies at the heart of machine learning, and its cornerstone is stochastic gradient descent ( SGD ), a method introduced over 60 years ago. The last eight years have seen an exciting new development: variance reduction for stochastic optimization methods. These variance-reduced ( VR ) methods excel in settings where more than one pass through the training data is allowed, achieving a faster convergence than SGD in theory and practice. These speedups underline the surge of interest in VR methods and the fast-growing body of work on this topic. This review covers the key principles and main developments behind VR methods for optimization with finite data sets and is aimed at nonexpert readers. We focus mainly on the convex setting and leave pointers to readers interested in extensions for minimizing nonconvex functions.

The convergence of Stochastic Gradient Descent (SGD) using convex loss functions has been widely studied. However, vanilla SGD methods using convex losses cannot perform well with noisy labels, which adversely affect the update of the primal variable in SGD methods. Unfortunately, noisy labels are ubiquitous in real world applications such as crowdsourcing. To handle noisy labels, in this paper, we present a family of robust losses for SGD methods. By employing our robust losses, SGD methods successfully reduce negative effects caused by noisy labels on each update of the primal variable. We not only reveal that the convergence rate is O(1/T) for SGD methods using robust losses, but also provide the robustness analysis on two representative robust losses. Comprehensive experimental results on six real-world datasets show that SGD methods using robust losses are obviously more robust than other baseline methods in most situations with fast convergence.

Biased stochastic conjugate gradient algorithm with adaptive step size for nonconvex problems

A new global variance reduction technique based on pseudo flux method

In this thesis a new method for the analysis of power transients in a nuclear reactor is developed, which is more accurate than the present state-of-the-art methods. Transient analysis is important tool when designing nuclear reactors, since they predict the behaviour of a reactor during changing conditions, such as a control-rod movement, induced by an operator, or an accident scenario. The current methods for transient analysis apply deterministic solvers to calculate the neutronic response, but not only do these deterministic solvers always need to discretise the problem, they also often apply more fundamental approximations such as homogenization or diffusion. Therefore a stochastic method is needed, which only has a statistical uncertainty. A challenge for applying the Monte Carlo method on transient analysis, is the different types of particles which must be simulated in a single simulation. It is common for a neutronics calculation to simulate the prompt neutrons, however the simulation of precursors is new. The main challenge when simulating precursors lies in their long lifetime, which is seconds, whereas the prompt neutrons have a lifetime of microseconds. This has been solved by dividing the transient problem into time intervals and forcing the precursors to produce a delayed neutron is all time intervals. This will ensure prompt-neutron chains in all intervals. The prompt neutrons will form prompt-neutron chains, which poses another challenge for the Monte Carlo method. The chain length of these prompt neutrons varies a lot and the can also split into many branches, which averages out in a real power reactor, but is difficult to simulate on a computer. To improve the chain-length statistics, a new variance reduction technique is developed, which dictates that a single neutron will emerge from each interaction. This emerging neutron can be the result of a scattering event or a fission event. Finally, a dynamic simulation scheme, which can run in parallel, has been devised, simulating all time intervals consecutively. Also, a method for sampling the initial conditions is created and this scheme is implemented in a purpose-built Monte Carlo code and in a general-purpose Monte Carlo code. The two codes have been tested in several cases and they behave as expected, agreeing with deterministic method where expected, but deviating when the deterministic methods are no longer valid. In order to simulate realistic transients in a power reactor, feedback has to be taken into account. To achieve this, the dynamic Monte Carlo method has been coupled to a thermal-hydraulics code, using an explicit scheme. The results of this coupled simulation are compared to a state-of-the-art coupled diffusion calculation in a NURISP-benchmark calculation and the results agree well, except for a small deviation towards the end. The exact source of this deviation should be further investigated, but the difference is small, when realising that there is already a deviation in the steady-state results. The results demonstrate the feasibility of performing a fully dynamic transient analysis, using only Monte Carlo for the neutronics part of the calculation. Transients can now be calculated, with detailed modelling of any complex geometry and with continuous energy, which is especially useful for newly developed reactor types and one-of-a-kind research reactors.

A coprime sensor array (CSA) is a sparse array geometry that interleaves two spatially undersampled uniform linear arrays (ULAs) with coprime undersampling factors. The CSA spans the same aperture and has equivalent resolution as a fully populated ULA, but uses fewer sensors. Previous works have introduced three CSA processors as spatial power spectral density (PSD) estimators: the Product, Min, and Blend processors. However, two of these processors display increased PSD esti-mate variance over the ULA PSD estimate. This dissertation focuses on reducing the variance of the CSA PSD estimators. PSD estimation traditionally relies on averag-ing uncorrelated coherent measurements (snapshots) to reduce variance. However,non-stationary underwater sonar environments often preclude increasing the num-ber of snapshots required to achieve a desirable PSD variance. In the traditional signal processing literature, there are two alternative methods to reduce variance without additional snapshot cost. The multitaper (MT) method and Welch's over-lapping segment averaging (WOSA) method improve PSD variance by O(K) at the expense of an O(K) resolution decrease by averaging K uncorrelated PSD estimates. Multitaper obtains these uncorrelated estimates by windowing the array with K or-thogonal tapers that span the entire array aperture. In contrast, WOSA obtains the uncorrelated estimates by subdividing the array into K possibly overlapping segments.This dissertation extends these two existing ULA variance reduction techniques to the CSA processors, making four main contributions. The first proposes the multitapered Product processor (MT-Product) that estimates the spatial PSD with reduced variance with respect to the traditional CSA Product processor but still uses fewer sensors than a fully populated ULA. The second proposes the multita-pered Min processor (MT-Min) that reduces the PSD estimate variance further than either MT-Product or Product. The MT-Min estimator has variance comparable to a multitapered ULA. The third proposes the multitapered Blend processor (MT-Blend) that blends MT-Product's attenuation of multiple source cross-terms with MT-Min's low variance while still achieving an O(K) variance reduction over the traditional Blend processor. The final contribution proposes the Welch overlapping segment averaging Product processor (WOSA-Product) that estimates the spatial PSD with reduced variance with respect to the traditional CSA Product processor but still uses fewer sensors than a fully populated ULA. WOSA-Product is gener-ally able to form a larger number of uncorrelated PSD estimates than MT-Product. Closed-form statistics for spatially white Gaussian processes are derived for MT-Product, MT-Min, and WOSA-Product. Simulations verify the variance reduction predicted by the analytical derivations each processor, and the effects of WOSA-Product segment length on resolution, variance reduction, and peak sidelobe levels are discussed.

Nowadays, there is a problem of a lack of computer power to conduct high-precision reactor analysis. There are several factors that increase the excessive computational load and make it difficult to calculate nuclear reactor full-scale models using Monte Carlo method. Among them is the large neutron-flux attenuation, which is observed in deep penetration problems. Various reduction techniques are used to increase the efficiency of Monte Carlo calculations in such tasks. It allows to reduce the statistical uncertainty of the functional evaluation without increasing the number of neutron histories. This work is devoted to study and testing of variance reduction techniques in the deep penetration problem. To demonstrate the possibility of using non-analogue Monte Carlo modeling a test problem was formulated. To quantify the efficacy of applying the variance reduction methods, FOM characteristic is considered, which is a function of the relative error in a flux estimate and the computational time of the simulation. The article considers non-analogue techniques implemented in the MCU (Monte Carlo Univarsal) and OpenMC codes. As part of the study, a module of the OpenMC code was developed that allows to automatically generate weight windows. It is shown that variance reduction techniques increase the calculation efficiency by several times, particularly, the weight windows method in OpenMC make it possible to achieve a 7-fold increase in the efficiency of neutron flux estimation with the same number of simulated histories. The formulated recommendations can be used in the calculation of innovative nuclear reactors full-scale models.

We present an efficient workflow that combines multiscale (MS) forward simulation and stochastic gradient computation - MS-StoSAG - for the optimization of well controls applied to waterflooding under geological uncertainty. The Iterative Multiscale Finite Volume (i-MSFV), a mass conservative reservoir simulation strategy, is employed as the forward simulation strategy. The i-MSFV method has the ability to accurately capture fine scale heterogeneities, and thus the fine scale physics of the problem. This allows for accurate and efficient simulations with fine-scale error estimates in a more computationally efficient coarse-scale simulation grid. The Stochastic Simplex Approximate Gradient (StoSAG) method, a stochastic gradient technique, inspired from Ensemble Optimization (EnOpt), is employed to compute the gradient of the objective function. Stochastic methods have recently been shown to be very efficient in terms of gradient quality and computational cost especially for robust optimization problems. They rely on the reservoir simulator as a black-box and perform a large number of forward simulations to approximate a gradient. In our numerical experiments, we investigate the impact of MS parameters such as coarsening ratio and heterogeneity contrast on the quality of the approximate MS-StoSAG gradient. Our experiments illustrate that i-MSFV simulations provide accurate forward simulation responses for the gradient computation, with the advantage of speeding up the workflow due to faster simulations. Speedups up to a factor of five on the forward simulation were achieved for the examples considered in the paper. The combination of speed and accuracy of MS forward simulation with the flexibility of the StoSAG technique (several different parameters/responses can be easily considered) allows for a flexible and efficient optimization strategy suitable for large-scale problems.

This work investigates the interplay between Robustness and Inclusion in the context of poisoning attacks targeting the convergence of Stochastic Gradient Descent (SGD). While robustness has received significant attention, the standard Byzantine defenses rely on the Independent and Identically Distributed (IID) assumption causing their performance to deteriorate on non-IID data distributions, even without any attack. This is largely due to these defenses being excessively cautious and discarding benign outliers. We introduce a penalty-based aggregation that accounts for the discrepancy between trusted clients and outliers. We propose the use of Linear Scalarization (LS) as an enhancing method to enable current defenses to simultaneously circumvent Byzantine attacks while also granting inclusion of outliers. This empowers existing defenses to not only counteract malicious adversaries effectively but also to incorporate outliers into the learning process. We conduct a theoretical analysis to demonstrate the convergence of our approach. Specifically, we establish the robustness and resilience of our method under standard assumptions. Empirical analysis further validates the viability of the proposed approach. Across mild to strong non-IID data splits, our method consistently either matches or surpasses the performance of current approaches in the literature, under state-of-the-art Byzantine attack scenarios.

Development of involved optimization algorithms is not an easy task for several reasons: First, every analyst is interested in a specific problem; Second, the capabilities of these methods may not be fully understood a priori; Third, coding of multi-purpose and more involved algorithms is not an easy job. In this paper, the optimization problem employing the near to global optimum algorithm is studied (Gadallah, M.H., 2000). The focus is to exploit 2 ideas: First, the algorithm can be modified to act as a variance reduction technique; Second, the algorithm can be modified to tackle the problem of system decomposition. Both ideas are novel within the context of statistical design of experiments. The first, if fully proved experimentally could yield the simultaneous integration of nominal and variance optimization possible. The second, can be extended to deal with multi-dimensional highly constrained systems with ease. These two ideas are explained wife the use of a simple example to illustrate the idea. An algorithm is developed that deal with the problem in several stages according to a predetermined decomposition scheme. The original objective and constraint functions are dealt with to suit each stage. Accordingly, all NP hard problems can ideally be transformed into NP complete ones with a consequence on the number of stages resulting from decomposition. Several decomposition scenarios are used and their results are compared numerically. Two orthogonal arrays and four composite arrays are used to plan experimentation; these are L27OA and L54OA and their subfamilies. These arrays are compared with respect to their statistical measures. The algorithm as such, is very promising optimization tool, especially for coupling system decomposition and variance reduction. Past work focused on either decomposition or statistical optimization. This work offers both capabilities. Several studies are reviewed and conclusions are drawn.