Sequential Testing in Reliability and Validity Studies With Repeated Measurements per Subject

A non-parametric sequential rank-sum probability ratio test method for binary hypothesis testing

Because computers (except for parallel computers) generate simulation outputs sequentially, we recommend sequential probability ratio tests (SPRTs) for the statistical analysis of these outputs. However, until now simulation analysts have ignored SPRTs. To change this situation, we review SPRTs for the simplest case; namely, the case of choosing between two hypothesized values for the mean simulation output. For this case, the classic SPRT of Wald (Wald A. Sequential tests of statistical hypotheses. Ann Math Stat 1945; 16: 117–186) allows general types of distribution, including normal distributions with known variances. A modification permits unknown variances that are estimated. Hall (Hall WJ. Some sequential analogs of Stein’s two-stage test. Biometrika 1962; 49: 367–378) developed a SPRT that assumes normal distributions with unknown variances estimated from a pilot sample. A modification uses a fully sequential variance estimator. In this paper, we quantify the performance of the various SPRTs, using several Monte Carlo experiments. In experiment #1, simulation outputs are normal. Whereas Wald’s SPRT with estimated variance gives too high error rates, Hall’s original and modified SPRTs are “conservative”; that is, the actual error rates are smaller than those prespecified (nominal). Furthermore, our experiment shows that the most efficient SPRT is Hall’s modified SPRT. In experiment #2, we estimate the robustness of these SPRTs for non-normal output. For these two experiments, we provide details on their design and analysis; these details may also be useful for simulation experiments in general.

In epidemiological prospective cohort studies, exposure levels of cases with disease and disease-free control subjects can be measured by laboratory analysis of previously stored biological specimens. In such studies, a sequential t-test can be used for preliminary evaluations, at the expense of the smallest possible number of specimens, of whether a new aetiological hypothesis is worth further investigation or whether specimens should rather be spared to test other, more fruitful, hypotheses. For this purpose, we recently compared two sequential probability ratio tests (SPRTs), in which the log-likelihood ratio was either based on an approximation, or computed exactly, and which were adapted to account for various control-to-case matching ratios. The tests turned out relatively conservative, particularly in terms of the significance level achieved. In the present paper, we compare an SPRT for matched or paired data based on Rushton's approximation to the log-likelihood ratio with a profile log-likelihood method developed by Whitehead. The comparison is partly mathematical, and partly based on computerized simulations. Average sample size for a sequential test is already smaller than for the equivalent fixed sample test. Increasing the number of controls matched per case further reduces the average sample size necessary to come to a decision. We show that, irrespective of the number of controls per case, pre-specified levels of statistical power and significance are respected closely by Whitehead's method, but not by Rushton's SPRT. This last procedure can lead to a significant loss in power. Since, in addition, Whitehead's method has been implemented in a commercially available computer program (PEST), we conclude that this method can be preferred to the methods we described earlier. Moreover, compared with the method of Rushton, Whitehead's method has the advantage that it can also be applied to groupwise inspection of the data and that it can also be converted easily into a truncated procedure.

Testability plays an important role in the readiness of equipment as a good design for testability (DFT) can greatly decrease the fault detection and isolation time, which will accelerate the maintenance actions. Testability verification is a procedure to check that whether the testability indexes such as fault detection rate (FDR) and fault isolation rate (FIR) meet the requirement in the contract. Currently, standards and statistical methods used in testability verification have the problems such as large sample, long period and so on. Sequential probability ratio test (SPRT) method can decrease the test sample size with almost a same operation characteristic as classical method based on binomial distribution. SPRT method and its truncated rules are introduced and the spectrum of expected test number is proposed. Then, the sample size allocation method and failure mode selection method based on failure rate used in sequential testability verification are illustrated. Testability verification of a control system is implemented with the given method and steps. Software named testability demonstration and evaluation system (TDES) which can calculate the decision criteria, plot decision chart, select failure mode and make judgment is used in the test as assistance. The result shows that the test sample size is remarkably decreased while comparing with the classical method.

In this article the problem of a sequential test for the model of independent non-identically distributed observations is considered. Based on recursive calculation a new numerical approach to approximate test characteristics for a sequential probability ratio test (SPRT) and a truncated SPRT (TSPRT) is constructed. The problem of robustness evaluation is also studied when the contamination is presented by the distortion of the distributions of all increments of the log-likelihood ratio statistics. The two-side truncated functions are proposed to be used for constructing the robustified SPRT. An algorithm to choose the thresholds of these truncated functions is indicated. The results are applied for a sequential test on parameters of time series with trend. Some kinds of the contaminated models of time series with trend are used to study the robustness of the truncated SPRT. Numerical examples confirming the theoretical results mentioned above are given.

Conditional frequentist tests of a precise hypothesis versus a composite alternative have recently been developed, and have been shown to be equivalent to conventional Bayes tests in the very strong sense that the reported frequentist error probabilities equal the posterior probabilities of the hypotheses. These results are herein extended to sequential testing, and yield fully frequentist sequential tests that are considerably easier to use than are conventional sequential tests. Among the interesting properties of these new tests is the lack of dependence of the reported error probabilities on the stopping rule, seeming to lend frequentist support to the stopping rule principle. Keywords:Bayes factor; Composite hypothesis; Conditional test; Error probability; Likelihood ratio; Sequential probability ratio test; Sequential test.

In this paper, we will discuss the sequential probability ratio test to test the zero inflation in counting data with the sample size being a random variable with specified Type I and Type II errors. We will introduce the zero-inflated non-central negative binomial distribution, and explain how its operating characteristic and average sample number functions are derived. Simulations are studied to indicate the performances of the proposed tests with different scenarios. A real application is also discussed to explain the process of the sequential probability ratio test.

We consider a single-hop, random-access, wireless sensor network (WSN) performing sequential distributed detection under a bandwidth constraint. At each time slot, the sensor nodes will make their decisions on whether the observed event is happening. The sensor censoring strategy is applied such that only the local decisions equal to one (i.e., the event happens) will be sent to the fusion center (FC). Because only one transmission channel between the FC and the sensor nodes is assumed, if two or more local decisions are sent at a time slot, a packet collision happens. We design a sequential probability ratio test (SPRT) at the FC that is aware of the collisions. The performance measures of the collision-aware SPRT are analyzed based on a Markov chain. In addition, we derive the asymptotic relative efficiency of the collision-aware SPRT relative to another SPRT. Numerical results show some interesting characteristics of the collision-aware SPRT. For example, the collision-aware SPRT prefers the scenario that many local decisions are likely to be sent per time slot.

Because of rare but serious adverse events, pharmaceutical drugs and vaccines are sometimes withdrawn from the market, either by a government agency such as the Food and Drug Administration (FDA) in the United States or by the manufacturing pharmaceutical company. In other cases, a drug may be generally safe but increase the risk for serious adverse events for certain subpopulations such as pregnant women or people with heart problems. Due to limited sample size and selected study populations, rare adverse events are often impossible to detect during phase 3 trials conducted before the drug is approved for general use. It is then important to conduct post-approval drug safety surveillance, using, for example, health insurance claims data. In such surveillance, the goal should be to detect serious adverse events as early as possible without too many false alarms, and it is then natural to use a continuous or near-continuous sequential test procedure that reevaluates the data on a daily or weekly basis. In this article, we first show that Wald's classical sequential probability ratio test (SPRT) for continuous surveillance is very sensitive to the choice of relative risk required in the specification of the alternative hypothesis, making it difficult to use for drug and vaccine safety surveillance. We instead propose the use of a maximized sequential probability ratio test (MaxSPRT) based on a composite alternative hypothesis, which works well across a range of relative risks. We illustrate the use of this method on vaccine safety surveillance and compare it with the classical SPRT. A table of critical values for the MaxSPRT is provided, covering most parameter choices relevant for vaccine and drug safety surveillance. The critical values are based on exact numerical calculations. We also calculate the statistical power, the expected time until the null hypothesis is rejected, and the average length of surveillance.

Using one class SVM to counter intelligent attacks against an SPRT defense mechanism

In this paper, we introduce a sequential procedure for the Multi-Objective Ranking and Selection (MOR&S) problems that identifies the Pareto front with a guaranteed probability of correct selection (PCS). In particular, the proposed procedure is fully sequential using the test statistics built upon the generalized sequential probability ratio test (GSPRT). The main features of the new proposed procedure are: 1) a unified framework, the new procedure treats the multi-objective problems in the same way as the single-objective problems; 2) an indifference-zone-free formulation, the new procedure eliminates the necessity of indifference-zone parameter; 3) asymptotically optimality, the GSPRT achieves asymptotically the shortest expected sample size among all sequential tests; 4) general distribution, the procedure uses the empirical likelihood for generally distributed observation. A numerical evaluation demonstrates the efficiency of the new procedure.

The sequential probability ratio test (SPRT) is a fundamental tool for sequential analysis. It forms the basis of numerous sequential techniques for different applications; for example, the truncated SPRT and Page's cumulative sum test (CUSUM). The performance of SPRT is characterized by two important functions—operating characteristic (OC) and average sample number (ASN), and CUSUM's performance is revealed by the average run length (ARL) function. These functions have been studied extensively under the assumption of independent and identically distributed log-likelihood ratios (LLRs) with constant bounds, which is too stringent for many applications. In this article, inductive integral equations governing these functions are developed under very general settings—the bounds can be time-varying and the LLRs are assumed independent but nonstationary. These inductive equations provide a theoretical foundation for performance analysis. Unfortunately, they have nonunique solutions in the general case except for the truncated SPRT. Numerical solutions for some frequently encountered special cases are developed and are compared with the results of Monte Carlo simulations.

We apply high-dimensional analysis of variance (HANOVA) and sequential probability ratio test (SPRT) to detect buried land mines from array ground penetrating radar (GPR) measurements. The GPR array surveys a region of interest in a progressive manner starting at a known position and moving step by step in a fixed direction. Our detection method consists of two stages. Because, at each stop of the array the path lengths are different from every transmitter/receiver pair to a mien target, there exists statistically significant difference among received signals when a mine target is presented. Thus, the first step in our processing consists of a HANOVA test to detect this statistical difference at each stop. HANOVA does not incorporate new data as the GPR array moves down-track. So secondly, we resort to a sequential probability ratio test to look for changes in the HANOVA statistics as the array proceeds down track. The SPRT allows for real-time processing as anew data are obtained by the GPR array. Finally, real sensor data are processed to verify the method.

A sequential test for correlation coefficients is presented. The method consists of reducing the test of a correlation coefficient to that of a variance ratio and making use of Helmert transformations of sequential observations. The resulting test is identified as the sequential probability ratio test regarding the parameter of the Cauchy distribution. The average sample number of the test is given and it is shown that the proposed sequential test yields an appreciable saving in the required average number of observations compared to the usual fixed sample size test.

We consider methods of computing exactly the probability of “acceptance” and the “average sample size needed” for the sequential probability ratio test (SPRT) and likewise the newer “2-SPRT,” concerning the value of a Bernoulli parameter. The methods permit one to approximate, iteratively, the desired operating characteristics for the test.