Performance Analysis of Sequential Probability Ratio Test

The operating characteristic (OC) and average sample number (ASN) of the sequential probability ratio test (SPRT) and multi-hypothesis SPRT (MSPRT) are studied. We consider the case where the observation sequence is independent but not necessarily identically distributed. Also, the thresholds for the test can be time varying. Based on the governing equations for OC and ASN of the SPRT developed in our previous work, a solution for the general case is proposed. The governing equations for OC and ASN of the MSPRT are also obtained. Numerical solutions for MSPRT are developed. Basically, the solutions rely on approximating the original test by truncation, that is, truncating the test at some finite time $K$ . We show that under some mild conditions, the approximation error diminishes as $K$ increases, at the cost of increased computation. Numerical examples are provided to demonstrate our solutions by comparing with Monte Carlo simulations, Simon’s lower bound, and Dragalin’s method (if available) for ASN.

Monte Carlo operating characteristic (OC) and average sample number (ASN) functions were compared with Wald's OC and ASN equations for sequential sampling plans based on Wald's sequential probability ratio test (SPRT) using the binomial, negative binomial, normal, and Poisson distributions. This comparison showed that the errors inherent in Wald's equations as a result of "overshooting" the decision boundaries of the SPRT can be large. Relative errors increased for the OC and ASN equations as the difference between the null (θ0)) and alternative (θ1) test parameter values increased. Relative errors also increased for the ASN equation as the probabilities of type I (α) and type II (β) errors increased. For discrete distributions, the relative errors also increased as θ0 increased with θ1/θ0 fixed. Wald's equations, in general, overestimate the true error probabilities and underestimate the true ASN. For the values of θ0, θ1, α, and β used in many sequential sampling plans in forestry, Wald's equations may not be adequate. For those cases where the errors in Wald's equations are important compared with the other errors associated with the sampling plan, two alternative Monte Carlo OC and ASN functions are proposed.

A non-parametric sequential rank-sum probability ratio test method for binary hypothesis testing

Sequential sampling plans based on the negative binomial distribution were developed for egg mass density of spruce budworm, Choristoneura fumiferana (Clemens), in Michigan's Upper Peninsula. Plans developed were modifications of Wald's sequential probability ratio test (SPRT) based on Monte Carlo simulation. Parameters of the negative binomial distribution were estimated by the maximum likelihood method. Sampling models developed classify egg mass populations into low and high categories using the number of egg masses on whole branch samples of balsam fir, Abies balsamifera (L.) Voss. Plans are presented for three pairs of population density hypotheses to give forest pest managers the flexibility to select a plan that best meets management objectives. Monte Carlo estimates indicated that Wald's average sample number (ASN) and operating characteristic (OC) equations underestimated the actual ASN values, and the actual alpha and beta error probabilities were smaller than the values prescribed by Wald's equations. Errors were sufficiently large to have serious economic implications during implementation of plans developed from Wald's approximate procedure. The sequential plans developed by modifying the decision boundaries of Wald's SPRT using the Monte Carlo procedure had error probabilities, OC functions, and ASN values that were approximately equal to those desired.

Because computers (except for parallel computers) generate simulation outputs sequentially, we recommend sequential probability ratio tests (SPRTs) for the statistical analysis of these outputs. However, until now simulation analysts have ignored SPRTs. To change this situation, we review SPRTs for the simplest case; namely, the case of choosing between two hypothesized values for the mean simulation output. For this case, the classic SPRT of Wald (Wald A. Sequential tests of statistical hypotheses. Ann Math Stat 1945; 16: 117–186) allows general types of distribution, including normal distributions with known variances. A modification permits unknown variances that are estimated. Hall (Hall WJ. Some sequential analogs of Stein’s two-stage test. Biometrika 1962; 49: 367–378) developed a SPRT that assumes normal distributions with unknown variances estimated from a pilot sample. A modification uses a fully sequential variance estimator. In this paper, we quantify the performance of the various SPRTs, using several Monte Carlo experiments. In experiment #1, simulation outputs are normal. Whereas Wald’s SPRT with estimated variance gives too high error rates, Hall’s original and modified SPRTs are “conservative”; that is, the actual error rates are smaller than those prespecified (nominal). Furthermore, our experiment shows that the most efficient SPRT is Hall’s modified SPRT. In experiment #2, we estimate the robustness of these SPRTs for non-normal output. For these two experiments, we provide details on their design and analysis; these details may also be useful for simulation experiments in general.

In medical, health, and sports sciences, researchers desire a device with high reliability and validity. This article focuses on reliability and validity studies with n subjects and m ≥2 repeated measurements per subject. High statistical power can be achieved by increasing n or m, and increasing m is often easier than increasing n in practice unless m is too high to result in systematic bias. The sequential probability ratio test (SPRT) is a useful statistical method which can conclude a null hypothesis H0 or an alternative hypothesis H1 with 50% of the required sample size of a non-sequential test on average. The traditional SPRT requires the likelihood function for each observed random variable, and it can be a practical burden for evaluating the likelihood ratio after each observation of a subject. Instead, m observed random variables per subject can be transformed into a test statistic which has a known sampling distribution under H0 and under H1. This allows us to formulate a SPRT based on a sequence of test statistics. In this article, three types of study are considered: reliabilityof a device, reliability of a device relative to a criterion device, and validity of a device relative to a  criterion device. Using SPRT for testing the reliability of a device, for small m, results in an average sample size of about 50% of the fixed sample size for a non-sequential test. For comparing a device to criterion, the average sample size approaches to 60% approximately as m increases. The SPRT tolerates violation of normality assumption for validity study, but it does not for reliability study.

In this article the problem of a sequential test for the model of independent non-identically distributed observations is considered. Based on recursive calculation a new numerical approach to approximate test characteristics for a sequential probability ratio test (SPRT) and a truncated SPRT (TSPRT) is constructed. The problem of robustness evaluation is also studied when the contamination is presented by the distortion of the distributions of all increments of the log-likelihood ratio statistics. The two-side truncated functions are proposed to be used for constructing the robustified SPRT. An algorithm to choose the thresholds of these truncated functions is indicated. The results are applied for a sequential test on parameters of time series with trend. Some kinds of the contaminated models of time series with trend are used to study the robustness of the truncated SPRT. Numerical examples confirming the theoretical results mentioned above are given.

In this article, we deal with problems of testing hypotheses in the framework of sequential statistical analysis. The main concern is the optimal design and performance evaluation of sampling plans in the Kiefer-Weiss problems. For the case of observations following a discrete exponential family, we provide algorithms for optimal design in the modified Kiefer-Weiss problem and obtain formulas for evaluating their performance, calculating operating characteristic function, average sample number, and some related characteristics. These formulas cover, as a particular case, sequential probability ratio tests (SPRTs) and their truncated versions, as well as optimal finite-horizon sequential tests. On the basis of the developed algorithms we propose a method of construction of optimal tests and their performance evaluation for the original Kiefer-Weiss problem. All algorithms are implemented as functions in the R programming language and can be downloaded from https://github.com/tosinabase/Kiefer-Weiss, where the code for the binomial, Poisson, and negative binomial distributions is readily available. Finally, we make numerical comparisons of the Kiefer-Weiss solution with the SPRT and the fixed-sample-size test having the same levels of the error probabilities.

Early stopping of clinical trials either in the case of beneficial or deleterious effect of treatment on quality of life (QoL) is an important issue. QoL is usually evaluated using self-assessment questionnaires and responses to the items are combined into scores assumed to be normally distributed (which is rarely the case). An alternative is to use item response theory (IRT) models such as the partial credit model (PCM) for polytomous items which takes into account the categorical nature of the items.Sequential analysis and mixed partial credit models were combined in the context of phase II noncomparative trials. The statistical properties of the sequential probability ratio test (SPRT) and of the triangular test (TT) were compared using mixed PCM and traditional average scores methods (ASM) by means of simulations.The type I error of the sequential tests was correctly maintained for both methods, the mixed PCM being more conservative than the ASM. Although remaining a bit underpowered, the mixed PCM displayed higher power than the ASM for both sequential tests. Both methods allowed substantial reductions in average sample numbers as compared with fixed sample designs. Overlapping of item category particularly affected the ASM by inflating the type I error and power. The use of IRT models in sequential analysis of QoL endpoints is promising and should provide a more powerful method to detect therapeutic effects than the traditional ASM.

SYNOPTIC ABSTRACTKozubowski and Podgorski (2003) have discussed properties, characterizations, and estimation of parameters of skew log Laplace distribution (SLLD). In this article, classical optimum tests for scale parameter of SLLD are derived. The most powerful (MP) test is obtained for scale parameter when shape parameters are known. Wald’s sequential probability ratio test (SPRT) is obtained, and its properties are studied. The likelihood ratio tests (LRT) for scale parameter are derived when the shape parameters are known and unknown. Finally, the SPRT and LRT are illustrated to the real life data.

We consider a single-hop, random-access, wireless sensor network (WSN) performing sequential distributed detection under a bandwidth constraint. At each time slot, the sensor nodes will make their decisions on whether the observed event is happening. The sensor censoring strategy is applied such that only the local decisions equal to one (i.e., the event happens) will be sent to the fusion center (FC). Because only one transmission channel between the FC and the sensor nodes is assumed, if two or more local decisions are sent at a time slot, a packet collision happens. We design a sequential probability ratio test (SPRT) at the FC that is aware of the collisions. The performance measures of the collision-aware SPRT are analyzed based on a Markov chain. In addition, we derive the asymptotic relative efficiency of the collision-aware SPRT relative to another SPRT. Numerical results show some interesting characteristics of the collision-aware SPRT. For example, the collision-aware SPRT prefers the scenario that many local decisions are likely to be sent per time slot.

In epidemiological prospective cohort studies, exposure levels of cases with disease and disease-free control subjects can be measured by laboratory analysis of previously stored biological specimens. In such studies, a sequential t-test can be used for preliminary evaluations, at the expense of the smallest possible number of specimens, of whether a new aetiological hypothesis is worth further investigation or whether specimens should rather be spared to test other, more fruitful, hypotheses. For this purpose, we recently compared two sequential probability ratio tests (SPRTs), in which the log-likelihood ratio was either based on an approximation, or computed exactly, and which were adapted to account for various control-to-case matching ratios. The tests turned out relatively conservative, particularly in terms of the significance level achieved. In the present paper, we compare an SPRT for matched or paired data based on Rushton's approximation to the log-likelihood ratio with a profile log-likelihood method developed by Whitehead. The comparison is partly mathematical, and partly based on computerized simulations. Average sample size for a sequential test is already smaller than for the equivalent fixed sample test. Increasing the number of controls matched per case further reduces the average sample size necessary to come to a decision. We show that, irrespective of the number of controls per case, pre-specified levels of statistical power and significance are respected closely by Whitehead's method, but not by Rushton's SPRT. This last procedure can lead to a significant loss in power. Since, in addition, Whitehead's method has been implemented in a commercially available computer program (PEST), we conclude that this method can be preferred to the methods we described earlier. Moreover, compared with the method of Rushton, Whitehead's method has the advantage that it can also be applied to groupwise inspection of the data and that it can also be converted easily into a truncated procedure.

Using one class SVM to counter intelligent attacks against an SPRT defense mechanism

We classify the modulation scheme of a received signal waveform modeled by a finite state Markov chain. We compare the likelihood ratio test (LRT) known as a fixed-sample-size classifier, which uses a fixed amount of data, and the sequential probability ratio test (SPRT) known as a fixed-error-rate classifier, which uses a variable amount of data just enough to achieve a certain correct rate. The SPRT approach has several advantages, including reduced computational complexity, less decision delay, controllable classification error rate, etc. The performance of MPSK and trellis-coded modulation (TCM) classifiers are demonstrated.

We apply high-dimensional analysis of variance (HANOVA) and sequential probability ratio test (SPRT) to detect buried land mines from array ground penetrating radar (GPR) measurements. The GPR array surveys a region of interest in a progressive manner starting at a known position and moving step by step in a fixed direction. Our detection method consists of two stages. Because, at each stop of the array the path lengths are different from every transmitter/receiver pair to a mien target, there exists statistically significant difference among received signals when a mine target is presented. Thus, the first step in our processing consists of a HANOVA test to detect this statistical difference at each stop. HANOVA does not incorporate new data as the GPR array moves down-track. So secondly, we resort to a sequential probability ratio test to look for changes in the HANOVA statistics as the array proceeds down track. The SPRT allows for real-time processing as anew data are obtained by the GPR array. Finally, real sensor data are processed to verify the method.