Effective Error Floor Estimation Based on Importance Sampling with the Uniform Distribution

Stochastic gradient descent (SGD) usually samples training data based on the uniform distribution, which may not be a good choice because of the high variance of its stochastic gradient. Thus, importance sampling methods are considered in the literature to improve the performance. Most previous work on SGD-based methods with importance sampling requires the knowledge of Lipschitz constants of all component gradients, which are in general difficult to estimate. In this paper, we study an adaptive importance sampling method for common SGD-based methods by exploiting the local first-order information without knowing any Lipschitz constants. In particular, we periodically changes the sampling distribution by only utilizing the gradient norms in the past few iterations. We prove that our adaptive importance sampling non-asymptotically reduces the variance of the stochastic gradients in SGD, and thus better convergence bounds than that for vanilla SGD can be obtained. We extend this sampling method to several other widely used stochastic gradient algorithms including SGD with momentum and ADAM. Experiments on common convex learning problems and deep neural networks illustrate notably enhanced performance using the adaptive sampling strategy.

Although importance sampling is an established and effective sampling and estimation technique, it becomes unstable and unreliable for high- dimensional problems. The main reason is that the likelihood ratio in the importance sampling estimator degenerates when the dimension of the problem becomes large. Various remedies to this problem have been suggested, including heuristics such as resampling. Even so, the consen- sus is that for large-dimensional problems, likelihood ratios (and hence importance sampling) should be avoided. In this paper we introduce a new adaptive simulation approach that does away with likelihood ratios, while retaining the multi-level approach of the cross-entropy method. Like the latter, the method can be used for rare-event probability estimation, optimization, and counting. Moreover, the method allows one to sam- ple from the target distribution rather than asymptotically as in Markov chain Monte Carlo. Numerical examples demonstrate the effectiveness of the method.

본 논문은 오류 마루 영역에서 우수한 성능을 가지는 비이진 LDPC (low-density parity-check) 부호의 설계 방법을 제안하고 성능을 검증한다. 제안된 설계 방법은 비이진 LDPC 부호의 이진 최소 거리(binary minimum distance)를 최대화하도록 패리티 검사 행렬의 비이진 원소 값들을 결정한다. BPSK (binary phase-shift keying) 변조 방식 하에서 제안된 방법으로 설계된 비이진 LDPC 부호가 오류 마루(error floor) 영역에서 우수한 성능을 가지는 것을 Monte Carlo 시뮬레이션과 중요도 표본 추출(importance sampling) 기법을 사용하여 검증한다. In this paper we propose a design algorithm for nonbinary LDPC (low-density parity-check) codes with low error-floors. The proposed algorithm determines the nonbinary values of the nonzero entries in the parity-check matrix in order to maximize the binary minimum distance of the designed nonbinary LDPC codes. We verify the performance of the designed nonbinary LDPC codes in the error-floor region by Monte Carlo simulation and importance sampling over BPSK (binary phase-shift keying) modulation.

Monte Carlo simulation is the technique to mimic a dynamic system or process using a computer program. Computer simulations, as an efficient and effective research tool, have been used virtually everywhere, including in biostatistics, engineering, finance, and other areas. To perform simulations, we often need to draw random samples from a certain probability distribution. Typically, the simplest and most important random sampling procedure is sampling from the uniform distribution over (0, 1), denoted by $$U\left (0,1\right )$$ . The computer-generated “random” number is not truly random because the sequence of numbers is determined by the so-called seed, an initial number. Random variates from other distributions can often be obtained by applying a transformation to uniform variates. There are usually several algorithms available to generate random numbers from a particular distribution. The algorithms differ in speed, accuracy, and the computer memory required. Many software packages have implemented various algorithms to generate random numbers with different probability distributions. Here we introduce a few commonly used algorithms.

This paper proposes an importance sampling methodology for the performance evaluation of a class of open-loop receivers with random carrier phase tracking in additive white Gaussian noise channels. The receivers, consisting of a bank of extended Kalman-Bucy filters and a decision algorithm based on the filters' innovations processes, perform symbol-by-symbol phase detection while kepping track of the random phase process within the symbol interval. We use a large deviations approach to start a stochastic importance sampling optimization, both for the irreducible error floor and for the general noisy operation of the receiver. Our simulations show a practical coincidence with conventional Monte Carlo results, with considerable simulation time gains.

A current assumption of most clustering methods is that the training data and future data are taken from the same distribution. However, this assumption may not hold in most real-world scenarios. In this article, we propose an information theoretical importance sampling based approach for clustering problems (ITISC), which minimizes the worst case of expected distortions under the constraint of distribution deviation. The distribution deviation constraint can be converted to the constraint over a set of weight distributions centered on the uniform distribution derived from importance sampling. The objective of the proposed approach is to minimize the loss under maximum degradation hence the resulting problem is a constrained minimax optimization problem, which can be reformulated as an unconstrained problem using the Lagrange method. The optimization problem can be solved by either an alternating optimization algorithm or a general optimization routine by commercially available software. Experiment results on synthetic datasets and a real-world load forecasting problem validate the effectiveness of the proposed model. Furthermore, we demonstrate that fuzzy c-means is a special case of ITISC with the logarithmic distortion, and this observation provides an interesting physical interpretation for fuzzy exponent <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$m$</tex-math></inline-formula>.

We are interested in estimating the bit error rate (BER) for signal transmission in digital communication systems. Since BERs tend to be extremely small, it is difficult to obtain precise estimators based on the use of crude Monte Carlo simulation techniques. In this paper, we review, expand upon, and evaluate a number of importance sampling variance reduction techniques for estimating the BER. We find that mixtures of certain "tailed" distributions with a uniform distribution produce estimators that are at least competitive with those in the literature. Our comparisons are based on analytical calculations and lay the groundwork for the evaluation of more general mixture distributions.

Weighted and unweighted power mean methods are compared against the accuracy of the simple arithmetic average to estimate an equivalent sandgrain roughness parameter ( $k_{s}$ ) for streamwise-heterogeneous rough surfaces. Specifically, these methods are conceptually and iteratively tested on roughness plates with surface characteristics following beta ( $\beta$ ), uniform or Gaussian distributions. The sandgrain roughness computed using these averaging methods, $k_{s_{eq}}$ , is then compared with true parameters, $k_{s_{eff}}$ , estimated from the fully rough asymptote model and a modified momentum integral method that accounts for the streamwise variation of skin friction over the heterogeneous surface. The weighted power mean offers significant advantages, particularly for roughness in $\beta$ distribution, which is attributed to the distribution bias towards smaller magnitudes of $k_s$ . While the advantage of using the weighted power mean over the unweighted power mean is less significant for the other surface distributions, the weighted method consistently yields the lowest discrepancy in effective drag estimates, independent of roughness configuration, and is therefore the recommended method for estimating the effective sandgrain roughness across heterogeneous surfaces.

We describe simple iterative decoders for low-density parity-check codes based on Euclidean geometries, suitable for practical very-large-scale-integration implementation in applications requiring very fast decoders. The decoders are based on shuffled and replica-shuffled versions of iterative bit-flipping (BF) and quantized weighted BF schemes. The proposed decoders converge faster and provide better ultimate performance than standard BF decoders. We present simulations that illustrate the performance versus complexity tradeoffs for these decoders. We can show in some cases through importance sampling that no significant error floor exists.

In this paper, we consider a statistical problem of learning a linear model from noisy samples. Existing work has focused on approximating the least squares solution by using leverage-based scores as an importance sampling distribution. However, no finite sample statistical guarantees and no computationally efficient optimal sampling strategies have been proposed. To evaluate the statistical properties of different sampling strategies, we propose a simple yet effective estimator, which is easy for theoretical analysis and is useful in multitask linear regression. We derive the exact mean square error of the proposed estimator for any given sampling scores. Based on minimizing the mean square error, we propose the optimal sampling scores for both estimator and predictor, and show that they are influenced by the noise-to-signal ratio. Numerical simulations match the theoretical analysis well.

One of the key weaknesses of low-density parity-check (LDPC) codes is the error floor that they typically exhibit at high signal-to-noise ratios (SNRs). Such an error floor is usually attributed to problematic structures known as trapping sets (TSs). The overwhelming majority of existing error floor estimation schemes consider the case of binary phase shift keying (BPSK) signalling. Unfortunately, these schemes are not readily extensible to estimate the error floor of high order LDPC coded modulation systems considered herein. To provide such a scheme, in this work, we use mean-shift importance sampling (MS-IS) to develop a novel error floor estimation methodology for high-order pulse amplitude modulation (PAM) and quadrature amplitude modulation (QAM) LDPC coded systems. First, a computationally efficient graphical-based approach is used to identify the TSs of a given LDPC code. Subsequently, a novel analytical approach is devised to identify the TSs that are likely to have a higher contribution in the error floor. These TSs are referred to as potentially dominant TSs (PDTSs). Finally, a new methodology for categorizing the PDTSs into equivalence classes is developed. A representative PDTS of each equivalence class is chosen and an MS-IS framework is devised to obtain the error rate corresponding to each equivalence class. To arrive at the desired MS-IS scheme, we develop an algorithm that invokes the geometry of the constellation to determine the MS value. In contrast with the conventional MS-IS method used in BPSK signalling, in the proposed MS-IS scheme, the MS value is a variable that is determined based on the TS and the transmitted codeword. The computational complexity of the three main steps of our methodology, viz. extracting the PDTSs, determining the MS values, and applying the MS-IS scheme, depends merely on the size of the constellation and the structure of the code, but not on the SNR. Numerical simulations confirm the efficacy and accuracy of the proposed technique at different SNRs.

Estimating cascade size and nodes' influence is a fundamental task in social, technological, and biological networks. Yet this task is extremely challenging due to the sheer size and the structural heterogeneity of networks. We investigate a new influence measure, termed outward influence (OI), defined as the (expected) number of nodes that a subset of nodes S will activate, excluding the nodes in S . Thus, OI equals, the de facto standard measure, influence spread of S minus | S |. OI is not only more informative for nodes with small influence, but also, critical in designing new effective sampling and statistical estimation methods. Based on OI, we propose SIEA/SOIEA, novel methods to estimate influence spread/outward influence at scale and with rigorous theoretical guarantees . The proposed methods are built on two novel components 1) IICP an important sampling method for outward influence; and 2) RSA, a robust mean estimation method that minimize the number of samples through analyzing variance and range of random variables. Compared to the state-of-the art for influence estimation, SIEA is Ω(log 4 n ) times faster in theory and up to several orders of magnitude faster in practice. For the first time, influence of nodes in the networks of billions of edges can be estimated with high accuracy within a few minutes. Our comprehensive experiments on real-world networks also give evidence against the popular practice of using a fixed number, e.g. 10K or 20K, of samples to compute the "ground truth" for influence spread.

This paper addresses the prediction of error floors of low-density parity-check codes transmitted over the additive white Gaussian noise channel. Using a linear state-space model to estimate the behavior of the sum-product algorithm (SPA) decoder in the vicinity of trapping sets (TSs), we study the performance of the SPA decoder in the log-likelihood ratio (LLR) domain as a function of the LLR saturation level. When applied to several widely studied codes, the model accurately predicts a significant decrease in the error floor as the saturation level is allowed to increase. For nonsaturating decoders, however, we find that the state-space model breaks down after a small number of iterations due to the strong correlation of LLR messages. We then revisit Richardson's importance-sampling methodology for estimating error floors due to TSs when those floors are too low for Monte Carlo simulation. We propose modifications that account for the behavior of a nonsaturating decoder and present the resulting error floor estimates for the Margulis code. These estimates are much lower, significantly steeper, and more sensitive to iteration count than those previously reported.

State estimation of a shop floor using improved resampling rules for particle filtering

Optical time-domain reflectometer (OTDR) enables simple identification and localization of a plethora of refractive and reflective events on a fiber link, including splices, connectors and breaks, and measuring insertion/return loss. Specifically, large enough OTDR dynamic range (DR) and thus high signal-to-noise-ratio (SNR) enable clear far-end visibility of longer fibers. We point out here that, under such conditions, the optical bit-error-rate (BER) floor is dominantly determined by reflective events that introduce significant return loss. This complements the OTDR legacy tests by appropriate optical BER floor estimation in the field. As high SNR implies inter-symbol interference as dominating error generating mechanism, we could apply the classical time-dispersion channel model for the optical BER floor determined by the root-mean-square (rms) delay spread of the actual fiber channel power-delay profile. However, as the high-SNR condition is not always fulfilled mostly due to insufficient DR, we propose here inserting a low-noise optical preamplifier as the OTDR front-end to reduce noise floor and amplify the backscattered signal. In order to verify the model for the exemplar test situation, we measured BER on the same fiber link to find very good matching between the measured BER floor values and the ones predicted from the OTDR trace.