Tolerance Optimization: A Decomposition Scheme, Variance Reduction and Fractional Approximation

We provide a thorough analysis of the effectiveness of different Variance Reduction Techniques (VRTs). We consider both stand-alone and combined applications of two input techniques, Antithetic Variates (AV) and Latin Hypercube Sampling (LHS), and two output techniques, Control Variates (CV) and Poststratified Sampling (PS). Previous research in the area mainly focuses on asymptotic variance reduction. In this experimental study, we measure the performance of VRTs under finite simulation run lengths and analyze their effects. Our findings show that the asymptotic variance reduction results do not readily apply to finite-length simulations. We consider three different types of systems (M/M/1, serial production line and (s, S) inventory control systems) and compare the VRTs under various experimental conditions. We observe that a variance reduction cannot be guaranteed for every instance a VRT is applied. Our results also indicate that the output VRTs (CV, PS) are better than input VRTs (AV, LHS) on the average for the single systems considered in this study. More interestingly, the less-sophisticated techniques (AV, CV) often perform better than the relatively more-complex techniques (LHS, PS). A comprehensive bibliography is also provided.

We provide the first theoretical analysis on the convergence rate of asynchronous mini-batch gradient descent with variance reduction (AsySVRG) for non-convex optimization. Asynchronous stochastic gradient descent (AsySGD) has been broadly used for deep learning optimization, and it is proved to converge with rate of O(1/\sqrt{T}) for non-convex optimization. Recently, variance reduction technique is proposed and it is proved to be able to accelerate the convergence of SGD greatly. It is shown that asynchronous SGD method with variance reduction technique has linear convergence rate when problem is strongly convex. However, there is still no work to analyze the convergence rate of this method for non-convex problem. In this paper, we consider two asynchronous parallel implementations of mini-batch gradient descent method with variance reduction: one is on distributed-memory architecture and the other is on shared-memory architecture. We prove that both methods can converge with a rate of O(1/T) for non-convex optimization, and linear speedup is accessible when we increase the number of workers. We evaluate our methods by optimizing multi-layer neural networks on two real datasets (MNIST and CIFAR-10), and experimental results demonstrate our theoretical analysis.

Variance and bias reduction techniques for the harmonic gradient estimator

A new global variance reduction technique based on pseudo flux method

Solving multi-objective design optimization problems can be computationally expensive, particularly when the original objective and/or constraint functions of the problem are costly to compute. To reduce computational cost, a surrogate model can be constructed that is less costly than the original objective and/or constraint functions. The surrogate is then combined with an optimizer to solve the problem. The proposed approach is an online surrogate-based optimization method in which the surrogate is built and improved iteratively as the optimizer converges to the solution. The primary contribution of this article is a new approach based on generative adversarial networks, aided by a constraint boundary-informed support vector machine, to predict whether the solutions generated by the approach are feasible or not. The performance of the proposed method is compared with that of two other methods from the literature. The comparison is based on several quality metrics and uses a set of 30 numerical test problems. The approach is also demonstrated with a complex engineering case study for the operation of an unmanned surface vessel. The results indicate that the proposed approach outperforms the other approaches for most of the quality metrics and test problems and case study considered.

A coprime sensor array (CSA) is a sparse array geometry that interleaves two spatially undersampled uniform linear arrays (ULAs) with coprime undersampling factors. The CSA spans the same aperture and has equivalent resolution as a fully populated ULA, but uses fewer sensors. Previous works have introduced three CSA processors as spatial power spectral density (PSD) estimators: the Product, Min, and Blend processors. However, two of these processors display increased PSD esti-mate variance over the ULA PSD estimate. This dissertation focuses on reducing the variance of the CSA PSD estimators. PSD estimation traditionally relies on averag-ing uncorrelated coherent measurements (snapshots) to reduce variance. However,non-stationary underwater sonar environments often preclude increasing the num-ber of snapshots required to achieve a desirable PSD variance. In the traditional signal processing literature, there are two alternative methods to reduce variance without additional snapshot cost. The multitaper (MT) method and Welch's over-lapping segment averaging (WOSA) method improve PSD variance by O(K) at the expense of an O(K) resolution decrease by averaging K uncorrelated PSD estimates. Multitaper obtains these uncorrelated estimates by windowing the array with K or-thogonal tapers that span the entire array aperture. In contrast, WOSA obtains the uncorrelated estimates by subdividing the array into K possibly overlapping segments.This dissertation extends these two existing ULA variance reduction techniques to the CSA processors, making four main contributions. The first proposes the multitapered Product processor (MT-Product) that estimates the spatial PSD with reduced variance with respect to the traditional CSA Product processor but still uses fewer sensors than a fully populated ULA. The second proposes the multita-pered Min processor (MT-Min) that reduces the PSD estimate variance further than either MT-Product or Product. The MT-Min estimator has variance comparable to a multitapered ULA. The third proposes the multitapered Blend processor (MT-Blend) that blends MT-Product's attenuation of multiple source cross-terms with MT-Min's low variance while still achieving an O(K) variance reduction over the traditional Blend processor. The final contribution proposes the Welch overlapping segment averaging Product processor (WOSA-Product) that estimates the spatial PSD with reduced variance with respect to the traditional CSA Product processor but still uses fewer sensors than a fully populated ULA. WOSA-Product is gener-ally able to form a larger number of uncorrelated PSD estimates than MT-Product. Closed-form statistics for spatially white Gaussian processes are derived for MT-Product, MT-Min, and WOSA-Product. Simulations verify the variance reduction predicted by the analytical derivations each processor, and the effects of WOSA-Product segment length on resolution, variance reduction, and peak sidelobe levels are discussed.

To determine whether the combination of forced-collision and DXTRAN variance-reduction (VR) techniques is unbiased, and to gain insight into the operation of these techniques, proof that first-moment estimates from Monte Carlo simulations employing both techniques are unbiased is developed. A general background on the forced-collision and DXTRAN VR techniques and their combination is given. Proof of an unbiased simulation is outlined by showing the equivalence of the history score moment equations of simulations with these techniques in use. A report with detailed proof of this equivalence is available upon request. The derivation of the future time equation using a similar approach, as well as a summary of the addition of the LLNL Pulsed Sphere experiments to the MCNP verification and validation suite, is also briefly discussed.

The paper is dealing with estimation of rare event probabilities in stochastic networks. The well known variance reduction technique, called Importance Sampling (IS) is an effective tool for doing this. The main idea of IS is to simulate the random system under a modified set of parameters, so as to make the occurrence of the rare event more likely. The major problem of the IS technique is that the optimal modified parameters, called reference parameters to be used in IS are usually very difficult to obtain. Rubinstein (Eur J Oper Res 99:89–112, 1997) developed the Cross Entropy (CE) method for the solution of this problem of IS technique and then he and his collaborators applied this for estimation of rare event probabilities in stochastic networks with exponential distribution [see De Boer et al. (Ann Oper Res 134:19–67, 2005)]. In this paper, we test this simulation technique also for medium sized stochastic networks and compare its effectiveness to the simple crude Monte Carlo (CMC) simulation. The effectiveness of a variance reduction simulation algorithm is measured in the following way. We calculate the product of the necessary CPU time and the estimated variance of the estimation. This product is compared to the same for the simple Crude Monte Carlo simulation. This was originally used for comparison of different variance reduction techniques by Hammersley and Handscomb (Monte Carlo Methods. Methuen & Co Ltd, London, 1967). The main result of the paper is the extension of CE method for estimation of rare event probabilities in stochastic networks with beta distributions. In this case the calculation of reference parameters of the importance sampling distribution requires numerical solution of a nonlinear equation system. This is done by applying a Newton–Raphson iteration scheme. In this case the CPU time spent for calculation of the reference parameter values cannot be neglected. Numerical results will also be presented.

Monte Carlo simulation of ``real`` particles. He refers to the resulting variance reduction method as the Local Importance Function Transform (LIFI) method. He demonstrates the efficiency of the LIFT method for several 3-D, linearly anisotropic scattering, one-group, and multigroup problems. In these problems the LIFT method is shown to be more efficient than the AVATAR scheme, which is one of the best variance reduction techniques currently available in a state-of-the-art Monte Carlo code. For most of the problems considered, the LIFT method produces higher figures of merit than AVATAR, even when the LIFT method is used as a ``black box``. There are some problems that cause trouble for most variance reduction techniques, and the LIFT method is no exception. For example, the author demonstrates that problems with voids, or low density regions, can cause a reduction in the efficiency of the LIFT method. However, the LIFT method still performs better than survival biasing and AVATAR in these difficult cases.

Global variance reduction method based on multi-group Monte Carlo adjoint calculation

The subject of this paper is I/sub ddQ/ testing for deep sub-micron CMOS technologies. The key concept introduced is the need to reduce the variance of good and faulty I/sub ddQ/ distributions. Other I/sub ddQ/ based techniques are reviewed within the context of variance reduction. Using the SEMATECH data and production data, variance reduction techniques are demonstrated. The main contribution of the paper is the systematic use of the die location and patterns in the I/sub ddQ/ data to reduce variance. Variance reduction is completed before any I/sub ddQ/ threshold limits are set.

Chapter 5 - Variance Reduction Techniques

Although offline handwritten signature recognition has been continually researched, it still requires an improvement of recognition rate. Most of existing techniques focus on feature extraction to improve their performance. This paper proposes an alternative way to increase the recognition rate by analyzing an important characteristic of input information, namely variability of signatures. The proposed method is based on the hypothesis; reducing the variability of signatures leads to boost up the recognition rate. Therefore, the variance reduction technique is applied to normalize offline handwritten signatures by means of an adaptive dilation operator. Then the variability of signatures is analyzed in terms of coefficient of variation (CV). The optimal CV is obtained and used to be a threshold limit value for the acceptable variance reduction. Based on 5,739 signature samples with 140 classes, the experimental results show that the adaptive variance reduction procedure helps improve the recognition rate when compared to the traditional schemes without adaptive variance reduction, including histogram of gradient (HOG) and pyramid histogram of gradient (PHOG) techniques.

Investigating the Effectiveness of Variance Reduction Techniques in Manufacturing, Call Center and Cross-Docking Discrete Event Simulation Models

The ionizing radiation exposure to crew on current and future space missions can significantly increase their health risks for cancers, degenerative diseases, and other acute and late effects. A common approach for estimating risk to crew is by completing stochastic (e.g., Monte Carlo) or deterministic particle transport simulations. Within the simulated environment, a small fraction of the particle histories tracked will interact with the astronaut or detector, particularly for larger spacecraft such as the International Space Station, Tiangong Space Station or Lunar Gateway. These simulations can be computationally intensive as they require a very large number of particle histories to achieve a low statistical uncertainty. Variance reduction techniques are applied to simulations to reduce the computational time of the simulation while maintaining the same (or less) statistical uncertainty. The variance reduction technique developed herein involves applying a directional source bias to an isotropic radiation field, such as galactic cosmic rays, to reduce the quantity of particles that have a low probability of interacting with the astronaut or detector. A custom application has been developed utilizing the Geant4 Toolkit that computes the trajectories and energies of particles in three dimensions in the International Space Station using the Monte Carlo method. The results demonstrate the impact of our variance reduction technique on effective dose equivalence depending on: primary and secondary particle type (proton, neutron, photon, heavy ion, etc.), geometric volumes and spacecraft materials. Our variance reduction technique can be tuned by the user to optimize the simulation time depending on their objectives and enables rapid testing of different shield configurations and materials. This variance reduction technique is implemented easily using several input parameters for boundary conditions. Recommended values are presented for rapid implementation in simulations.