Weighted statistical white-box testing with proportional-optimal stratification

Interpreting “statistical hypothesis testing” results in clinical research

Cleanroom software engineering requires statistical testing by an independent agent for the purpose of certifying software quality. Statistical software testing is a formal process that involves sampling from the intended usage environment and the precise measurement of properties of random variables inherent in such a statistical experiment. The first step in conducting a statistical test is to determine the usage distribution for the software in its intended environment. This distribution is the basis for a random generator of test sequences for the software. In order to achieve this distribution, a usage analysis is performed using the software specification and any available usage information. The usage analysis consists of a top down, structural investigation of the specification document that establishes a set of usage states and defines an ordering on this set. The paper describes in detail the usage analysis and inference procedure including various computations on the ensuing Markov chains. Stopping criteria are developed and a discrete software reliability model is presented.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

This experience report describes the successful application of statistical software testing to a large-scale realworld equipment replacement project. The Data Processing System at Dungeness B nuclear power station is obsolescent and needs to be replaced by a new system involving different technology. The new Data Processing and Control System (DPCS) involves a large amount of software code, with the potential for software failures, and therefore confidence needs to be established in the dependability of the new software-based system. Statistical software testing (ST) was used as one technique to do this. The aim was to support the claim that with high confidence the higher integrity parts of the replacement software have a failure probability of better than 10-2 failures per demand (fpd). This paper describes how this has been achieved by careful design of the statistical tests and by the development of a large number of supporting tools and procedures. A total of 395 statistical tests with no failure was achieved providing at least 98% confidence that the safety claim has been met.

Like any other science, research in natural language processing (NLP) depends on the ability to draw correct conclusions from experiments. A key tool for this is statistical significance testing: We use it to judge whether a result provides meaningful, generalizable findings or should be taken with a pinch of salt. When comparing new methods against others, performance metrics often differ by only small amounts, so researchers turn to significance tests to show that improved models are genuinely better. Unfortunately, this reasoning often fails because we choose inappropriate significance tests or carry them out incorrectly, making their outcomes meaningless. Or, the test we use may fail to indicate a significant result when a more appropriate test would find one. NLP researchers must avoid these pitfalls to ensure that their evaluations are sound and ultimately avoid wasting time and money through incorrect conclusions.This book guides NLP researchers through the whole process of significance testing, making it easy to select the right kind of test by matching canonical NLP tasks to specific significance testing procedures. As well as being a handbook for researchers, the book provides theoretical background on significance testing, includes new methods that solve problems with significance tests in the world of deep learning and multidataset benchmarks, and describes the open research problems of significance testing for NLP.The book focuses on the task of comparing one algorithm with another. At the core of this is the p-value, the probability that a difference at least as extreme as the one we observed could occur by chance. If the p-value falls below a predetermined threshold, the result is declared significant. Leaving aside the fundamental limitation of turning the validity of results into a binary question with an arbitrary threshold, to be a valid statistical significance test, the p-value must be computed in the right way. The book describes the two crucial properties of an appropriate significance test: The test must be both valid and powerful. Validity refers to the avoidance of type 1 errors, in which the result is incorrectly declared significant. Common mistakes that lead to type 1 errors include deploying tests that make incorrect assumptions, such as independence between data points. The power of a test refers to its ability to detect a significant result and therefore to avoid type 2 errors. Here, knowledge of the data and experiment must be used to choose a test that makes the correct assumptions. There is a trade-off between validity and power, but for the most common NLP tasks (language modeling, sequence labeling, translation, etc.), there are clear choices of tests that provide a good balance.Beginning with a detailed background on significance testing, the book then shows the reader how to carry out tests for specific NLP tasks. There is a mix of styles, with the first four chapters providing reference material that will be extremely useful to both new and experienced researchers. Here, it is easy to find the material related to a given NLP task. The next two chapters discuss more recent research into the application of significance tests to deep neural networks and for testing across multiple datasets. Alongside open research questions, these later chapters provide clear guidelines on how to apply the proposed methods. It is this mix of background material and reference guidelines that I believe makes this book so compelling and nicely self-contained.The introduction in Chapter 1 motivates the need for a comprehensive textbook and outlines challenges that the later chapters address more deeply. The theoretical background material begins in Chapter 2, which introduces core concepts, including hypothesis testing, type 1 and type 2 errors, validity and power, and p-values. The reader does not need to have any prior knowledge of statistical significance tests to follow this part. However, experienced readers could still benefit from reading this chapter, as concepts such as p-values are widely misunderstood and misused (Amrhein, Greenland, and McShane 2019).The significance tests themselves are introduced in Chapter 3, categorized into parametric and nonparametric tests. The chapter begins with the intuitively simple paired z-test, then builds up to more commonly-applied techniques, showing the connections and assumptions that each test makes. Step-by-step algorithms help the reader to implement each test. Although the chapter does cite uses of tests in NLP research, the main purpose is to present the theory behind each test and point out their differences.Chapter 4 provides perhaps the most handy part of the book for reference: a correspondence between common NLP tasks and statistical tests. Each task is discussed in terms of the evaluation metrics used, then a decision tree is introduced to guide the reader toward a choice between a parametric test, bootstrap or randomization test, or sampling-free nonparametric test. Section 4.3 then links each NLP evaluation measure to a specific significance test, presenting a large table that helps readers identify which test they need for a specific task. Particular considerations for each task are also pointed out to provide more detail about the appropriate options. The final part of this chapter describes the issue of p-hacking, in which dataset sizes are increased until a significance threshold is reached without consideration for biases in the data (discussed, for example, in Hofmann [2015]). The chapter proposes a simple solution to ensure robust significance testing with large datasets.Where Chapter 4 presents well-established methods, Chapter 5 introduces the current research question of how best to apply statistical significance testing to deep learning. Non-convex loss functions, stochastic optimization, random initialization, and a multitude of hyperparameters limit the conclusions we can draw from a single test run of a deep neural network (DNN). This chapter, which is based on the authors’ ACL paper (Dror, Shlomov, and Reichart 2019), explains how the comparison process can be overhauled to provide more meaningful evaluations. Beginning by explaining the difficulties of evaluating DNNs, the chapter then introduces criteria for a comparison framework, then discusses the limitations of current methods. Reimers and Gurevych (2018) have previously tackled this problem, but their approach has limited power and does not provide a confidence score. Other works, such as Clark et al. (2011), compare DNNs using a collection of statistics, such as the mean or standard deviation of performance across runs. This book shows how such an approach violates the assumptions of the significance tests. The authors propose almost stochastic dominance as the basis for a better alternative. The chapter explains how to use the proposed method, evaluates it in an empirical case study, and finally analyzes the errors made by each testing approach.Large NLP models are often tested across a range of datasets, which presents another problem for standard significance testing. Chapter 6 discusses the challenges of assessing two questions: (1) On how many datasets does algorithm A outperform algorithm B? (2) On which datasets does A outperform B? Applying standard significance tests individually to each dataset and counting the number of significant results is likely to overestimate the total number of significant results, as this chapter explains. The authors then present a new framework for replicability analysis, based on partial conjunction testing, and discuss two variants (Bonferroni and Fisher) for when the datasets are independent or dependent. They introduce a method based on Benjamini and Heller (2008) to count the number of datasets where one method outperforms another, then show how to use the Holm procedure (Holm 1979) to identify which datasets these are. Chapter 6 provides a lot of detailed background on the proposed replicability analysis framework, and the later sections again link the process to specific NLP case studies, and step-by-step summaries help the reader to apply the methodology. Extensive empirical results illustrate the very substantial differences in outcomes between the proposed approach and standard procedures.The final two chapters present open problems and conclude, showing that the topic has many interesting research questions of its own, such as problems when performing cross-validation, and the limited statistical power of replicability analysis.Overall, I highly recommend this book to a wide range of NLP researchers, from new students to seasoned experts who wish to ensure that they compare methods effectively. The book is excellent as both an introduction to the topic of significance testing and as a reference to use when evaluating your results. For anyone with further interest in the topic, it also points the way to future work. If one could level any criticism at this book at all, it is that it does not deeply discuss the basic flaws of significance testing or what the alternatives might be. For now, though, significance testing is an integral part of NLP research and this book provides a great resource for researchers who wish to perform it correctly and painlessly.

A recent article in this journal by Tabak (2006) highlighted a potentially serious source of bias that can arise when multiple statistical tests of a damages theory are performed and even one test rejecting the null hypothesis is regarded as supporting the damages theory. Repeated testing will eventually produce a “false discovery,” that is, rejection of the null hypothesis in favor of the alternative hypothesis when the null hypothesis is true, which statisticians refer to as type I error. Consequently, performing multiple tests without adjusting the critical value can be problematical because it can lead to improperly accepting statistical evidence that apparently supports rejection of the null hypothesis as reliable when it is not. Tabak (2006) recommends making the Sidak (1968, 1971) multiple-comparison adjustment to the standard statistical t-test to correct for the false-discovery bias inherent in multiple-comparison testing. In particular, he recommends making this adjustment when performing 10b-5 securities fraud event studies when more than one corrective disclosure date is involved. This article clarifies the circumstances in which a multiple-comparison adjustment is appropriate and explains why the correction is normally not needed in securities fraud event-study testing. More generally, I explain why it is not required when each of several tests is performed and its results are reported separately, as for example, where the objective is simply to test the statistical significance of the abnormal stock return on each day on which a new and distinct curative disclosure occurred. I show that the Sidak multiple-comparison adjustment is nearly as stringent as the classical Bonferroni procedure (Simes, 1986), which can increase the risk of type II error. This article discusses a more powerful alternative to the Sidak adjustment due to Benjamini and Hochberg (1993), which directly corrects for the false-discovery bias in multiple-comparison testing and reduces the risk of type II error. II. Application of Multiple-Comparison Adjustments to Securities Fraud Event Studies Multiple-comparison-false-discovery bias can arise when (a) several statistical tests are performed on subsets of the same larger data set in an effort to

An accident scenario-based statistical test method for RPS in nuclear power plant

Statistical testing based on a Markov chain usage model, as a rigorous testing method, has been around for more than two decades. Through the comprehensive application of statistical science to the testing of software, it provides audit trails of evidence to support correctness arguments for a software-intensive system as well as a decision that the system is of requisite quality for its intended use. This paper reports a real-world case study in which we applied standard statistical testing practices to the phone flag copy testing problem at the site of our industrial collaborator, and presents our solution from problem formalization, usage modeling and model analysis to test case generation and analysis. Our results helped evaluate the coverage of a heuristically generated test suite, and shed light on what other test cases to craft/generate to improve test coverage and adequacy, as well as reliability estimates both at the arc (usage event) level and at the path (system) level.

This paper argues that a Fisherian approach to statistical inference, which views statistical testing as determining the chance probability of an event, is coherent, consistent with modern statistical methods and forms a sound theoretical basis for the use of statistical tests in psychology. It is argued that Fisherian statistical tests are concerned with establishing the direction of tested effects, give rise to confidence intervals and are quite consistent with power analyses. Researchers are encouraged to report chance probabilities, and to interpret them according to the prevailing conditions rather than using a fixed decision rule. The contrasting Neyman–Pearson approach, which views statistical testing as a quality control procedure for accepting hypotheses, posits unreasonable research practices which psychologists do not and should not be expected to follow. Neyman–Pearson theory has caused confusion in the psychological literature and criticisms have been levelled at statistical testing in general that ought to have been directed specifically at Neyman–Pearson testing. Statistical inference, of any sort, is held to be insufficient to characterize the process of testing scientific hypotheses. Data should be seen as evidence to be used in psychological arguments and statistical significance is just one measure of its quality. It restrains researchers from making too much of findings which could otherwise be explained by chance.

[Context] In agile software development, requirements are frequently represented in lightweight artifacts such as user stories (USs). To support their validation, USs may be complemented with additional artifacts such as acceptance criteria and test cases. [Problem] Existing studies propose frameworks for US writing, or investigate high-level factors that contribute to productivity, such as teamwork and support from management. However, empirical evidence on the impact of requirements artifacts on team efficiency is scarce. [Method] Through a mixed-methods case study at a large Dutch organization, we investigate how the use of requirements artifacts affects efficiency. We collected data from eight agile teams: 1,345 USs from nineteen sprints, US quality indicators, the quantity of associated acceptance criteria and test cases, and cycle time. We statistically analyzed the existence of correlations between these variables, and we triangulated our results through interviews with product owners and scrum masters from the involved teams. [Findings] Our employed indicators for US quality do not seem to have an effect on on-time completion, with our interviewees highlighting the trade-off between investing on writing quality and short-term efficiency. We found, instead, that the existence of acceptance criteria may be a relevant factor for on-time completion, as confirmed both by statistical tests and the interviewees.

Safety-critical software often includes some rarely executed critical operations that are often inadequately tested in statistical testing based reliability estimation. However, it is necessary to assess the reliability of safety-critical software to a high degree of confidence before it is deployed in the field. This paper discusses how to reduce safety-critical software statistical testing cost based on importance sampling technique. When both the critical operations and the entire software are adequately tested, the method can still compute the unbiased software reliability from the test results with much less test cases. Thus, the statistical testing cost of safety-critical software can be reduced effectively The simulated annealing algorithm for calculating optimum transition probabilities of the Markov chain usage model for reducing software statistical testing cost is also presented.

The paper concerns automatic generation of usage models from real-time software UML models. Firstly, we define the reasonably constrained real-time software UML artifacts, which include use case diagrams, timed sequence diagrams and the execution probability of each sequence diagram in its associated use case. Secondly, the paper presents a method that derives the software usage model from the constrained UML artifacts. The method elicits the messages associated with the objects under testing and their occurrence probabilities to generate the usage model of each use case. Timing constraints in sequence diagrams are considered during usage model generation. Then the usage models of use cases are integrated into the software usage model by utilizing the execution sequence relations between use cases. The usage models can be used to generate real-time software statistical test cases and facilitate real-time software statistical testing.

Hand calculators have come into common use for performing calculations in high school and undergraduate courses in elementary statistics. Computers have also been used to do standard statistical computations and tests. However, computers can be used in elementary statistics courses not only to do standard statistical tests, but to perform simulations which clarify concepts and theorems of statistics. These simulations allow the nonmathematically oriented student in elementary statistics to have inductive experiences with statistical concepts in a time-efficient manner. A basic concept in statistics is that of a sample distribution. The sampling distribution of a sample statistic is defined to be the theoretical probability distribution of values for that sample statistic obtained from all possible samples of a certain fixed size from the population. Typically elementary texts illustrate this concept by using a very small finite population which has a small number of possible samples of a given size. Although large or infinite populations are of interest since they have a large or infinite number of samples of a certain size it is difficult to develop a sample distribution. Computer simulation, however, makes this easy to explore. The computer can simulate taking many samples of a certain size from a population. By taking many samples the law of large numbers guarantees that the observed frequency distribution of the sample statistic is close to the theoretical sample distribution. The sample distribution of the mean is the most important sample distribution. The computer simulations in Figure 1 explore a sample distribution of the mean. These simulations employ an elementary statistical package called MINITAB developed at Pennsylvania State University. The MINITAB command URANDOM simulates random observations from a population where numbers between 0 and 1 all have an equal probability of occurring. The first simulation consists of drawing 200 samples of size 2 and computing the means of the samples. A histogram of the results is given. Continuing with samples of size 2, drawing 750 samples is simulated and then 1500 samples. As the number of samples increases the student can see the isosceles triangle-shaped theoretical sample distribution emerging. Note that, because of the small sample size and the nonnormal population, the sample distribution should not be expected to be normal.

Introduction: Researchers from developing countries may not have access to statistical software packages. However, descriptive and inferential statistical tests are to conduct to conclude the study. In a previous article (DOI: 10.4103/ijves.ijves_116_21), we described how to conduct some of the common inferential statistical tests online. This article is the second part of the series. Aim: We aimed to provide the examples of some inferential statistical tests used in clinical studies and provide step-by-step guidelines to conduct those tests online. Methods: We prepared a set of data for each statistical test. These data were used to carry out the test online and the steps are briefly described. The result of the test is presented with screenshots and text to get an idea of how to report the result in a manuscript. Results: We described the process of conduct of the following tests online – Receiver operating characteristics curve analysis, Kaplan − Meier estimate, dose-response, logistic regression, multiple linear regression, residual analysis, odd ratio, Bland − Altman plot, Cronbach's alpha, Cohen's kappa, and intraclass correlation coefficient. In addition, a method for random allocation of subjects in groups was also described. All the tests were described with example data available in a supplementary file. Conclusion: In this article, some of the inferential statistics used for clinical studies are described with example data and a step-by-step guide. Any clinician from resource-limited settings may use this guide as a reference for statistical tests. However, the tests described in this article are not a comprehensive list.

Motivation is probably the most important factor that educators can target in order to improve learning. Numerous cross-disciplinary theories have been postulated to explain motivation. While each of these theories has some truth, no single theory seems to adequately explain all human motivation. The fact is that human beings in general and pupils in particular are complex creatures with complex needs and desires. In this paper, Analytic Hierarchy Process (AHP) has been proposed as an emerging solution to move towards too large, dynamic and complex real world multi-criteria decision making problems in selecting the most suitable motivator when choosing school for their children. Data were analyzed using SPSS 17.0 (“Statistical Package for Social Science”) software. Statistic testing used are descriptive and inferential statistic. Descriptive statistic used to identify respondent pupils and parents demographic factors. The statistical testing used to determine the pupils and parents highest motivator priorities and parents’ best priorities using AHP to determine the criteria chosen by parents such as school principals, teachers, pupils and parents. The moderating factors are selected schools based on “Standard Kualiti Pendidikan Malaysia” (SKPM) in Ampang. Inferential statistics such as One-way ANOVA used to get the significant and data used to calculate the weightage of AHP. School principals is found to be the best motivator for parents in choosing school for their pupils followed by teachers, parents and pupils.

For more than three decades, Deborah Mayo has defended severe testing, a framework for evaluating the strength of scientific evidence that provides philosophical foundations for (variants of) classical statistical methods. In her newest book, Statistical Inference as Severe Testing, Mayo (1) extends and refines her views on severe testing, (2) takes aim at researchers who claim that the reproducibility crisis is a result of the use of classical statistics, and (3) exposes some contemporary Bayesian methods as lacking philosophical foundations. The book is engaging, sometimes funny, and often insightful. After reviewing the basics of severe testing, I summarize three lessons from Mayo's book that are valuable for philosophers and scientists alike. I then discuss one way in which I think Mayo's arguments could be tightened.Mayo argues that one has evidence for a hypothesis to the extent the hypothesis has passed a severe test. More precisely, she argues that “for data to warrant a hypothesis H requires not just that (S-1) H agrees with the data (H passes the test), but also (S-2) with high probability, H would not have passed the test so well, were H false” (92). Mayo's two criteria are inspired by the Neyman-Pearsonian maxim that good statistical tests minimize Type 1 and Type 2 errors, but they are novel in at least two ways. First, Mayo's criteria are applicable outside of statistical contexts. For example, let H be the hypothesis that the Cleveland Browns played an excellent team last weekend, and suppose our data is that the Cleveland Browns lost their game. If the Cleveland Browns lose to all opponents regardless of ability, then our data fails to satisfy S-2 and so does not severely test H. Since S-1 and S-2 are applicable in both statistical and nonquantitative contexts, Mayo's criteria unify the logic of statistical testing with qualitative logics of evidence.Second, severity is a function of data, unlike size and power, the two core quantities of orthodox Neyman-Pearsonian methodology. For this reason, Mayo's two criteria closely resemble Nozick's adherence and sensitivity conditions when those conditions are relativized to “methods” (Nozick 1981).1Mayo draws several important lessons from this seemingly simple, two-pronged framework. Here, I discuss three that recur throughout the newest book. First, Mayo carefully distinguishes between statistical hypotheses (e.g., about the frequency of a trait in a population) and “substantive” scientific hypotheses about, for example, causation or the existence of a theoretical entity (84, 93). Conflating the two can lead one to mistakenly infer that a substantive scientific hypothesis has been severely tested, when in fact the evidence against the hypothesis is stronger than that for it.As a vivid example, Mayo discusses a series of psychological experiments used to defend the hypothesis H that heterosexual “men's implicit self-esteem is lower when a [female] partner succeeds than when a [female] partner fails” (101). Mayo carefully distinguishes four statistical hypotheses that are plausibly consequences of ¬ H and then argues that the data were sufficient to reject only one of the four; call it ¬ H 1. Mayo concludes that the experimental data failed to fit H, as researchers could not reject three statistical hypotheses that were compatible with ¬ H. Further, Mayo argues that the statistical evidence against ¬ H 1 (i.e., the purported evidence for ¬¬ H = H ) was perfectly compatible with the falsity of the substantive hypothesis H. So the data also failed to satisfy S-2.Second, Mayo argues that, whereas the severe tester has a unified account of testing both statistical hypotheses and modeling assumptions, likelihoodists and Bayesians lack a cogent story about when to reject a bad statistical model (298–320). Some likelihoodists, for example, argue that measures of evidential strength must be comparative and that the likelihood ratio P (E|H 0)/P (E|H 1) quantifies the degree to which evidence E fits a hypothesis H 0 better than H 1 (Edwards 1984; Royall 1997). It is easy to construct cases in which such a likelihood ratio is enormous, because the data fit H 0 much better than H 1, and yet the data fit neither statistical hypothesis well. If H 0 and H 1 are the only statistical hypotheses under investigation, one might erroneously conclude that one has good evidence for H 0. The right conclusion, according to the severe tester, is to recognize that the two hypotheses do not exhaust logical space and to reject that underlying modeling assumption instead.Naive Bayesian reasoning falls victim to the same criticism. According to some Bayesians, experimenters ought to assign prior probabilities to a fixed set of hypotheses H and then update their degrees of belief by conditionalization. Again, suppose the data fit all hypotheses in H poorly but fit one H ∈ H much better than its rivals. Then a naive Bayesian's posterior degree of belief in H might be nearly one. Again, the right conclusion, according to the severe tester, is to consider more hypotheses. But orthodox Bayesianism is silent about when to do that.Finally, Mayo argues that several novel Bayesian approaches to statistical inference lack foundations (400–415). These parts of the book will be of special interest to philosophers who are unfamiliar with the substantial differences between Bayesianism in theory and in practice. According to textbook philosophical presentations, prior and posterior probabilities represent an experimenter's beliefs. However, some contemporary Bayesians have given up on the myth that the infinitely precise densities that appear in journal articles are appropriate idealizations of any person's beliefs. Those Bayesians, however, disagree about what justifies the choice of prior (e.g., transparent communication vs. frequentist coverage properties) and the use of further Bayesian mathematical machinery.Mayo levels several different criticisms against these contemporary Bayesian approaches, but I think all share the same underlying philosophical problem. Rationality, so says standard decision theory, requires one to maximize expected utility relative to one own's degrees of belief. So some contemporary “Bayesian” methods require a new theory of rationality to justify the behavior of an experimenter whose decisions employ posterior probabilities that do not even approximately represent her (or perhaps anyone's) beliefs.There are other important lessons scattered through Mayo's engaging and fairly accessible book. That leads me to my main criticism: accessibility sometimes comes at the cost of precision. I focus on one example that is directly relevant to Mayo's analysis of the causes of the replicability crisis, a central topic of the book.Mayo repeatedly suggests that a hypothesis is not severely tested if it is the result of “data-dredging.” Data-dredging is one of several pejorative terms (including p-hacking) that some statistical reformers use, and it refers to a loosely defined set of purportedly objectionable practices for finding associations in data. Often, “data-dredging” describes the practice of subjecting a single data set to many (sometimes hundreds or thousands of) statistical tests in an attempt to find a statistically significant result that is publishable.Critics of data-dredging allege that, because there is always a chance of finding a spurious association between two variables, one should be wary of “dredged” hypotheses. For severe testers, that default skepticism is justifiable: if an experiment is not designed to test a hypothesis, it is unlikely to be a severe test by chance. However, intuitions about data-dredging depend upon how the practice is described. Searching through data for patterns sounds exactly like what scientists should do!Mayo illustrates the dangers of data-dredging via several thought experiments. For example, she imagines a pharmaceutical company that hires scientists to find the benefits of a recently developed drug (267–68). When the hired researchers find no evidence that the drug is salutary in any of ways the manufacturer had hoped, they mine their data to find some salutary effect. Sure enough, they find the drug has an “impressive benefit on factor B” and begin marketing the drug with that promise. Mayo concludes that the hired scientists fail to subject the hypothesis to a severe test because they would have found an effect no matter what.Unfortunately, Mayo never explicitly applies her two criteria for severity to this example. Which hypothesis exactly is being tested? Here are two candidates. Let H 1 be the hypothesis “The drug has some salutary effect” and let H 2 be the hypothesis “The drug has the specific salutary effect found on B.” Even if the drug manufacturer is guaranteed to find some effect (i.e., H 1 is not severely tested), it does not follow that the manufacturer lacks evidence that the drug has the specific effect found by mining the data (i.e., it does not follow that H 2 is not severely tested). When I grade student logic exams, I am guaranteed to find that some student scored highest, but I do not thereby lack evidence that Roshan scored best when I tally her score and see it is highest. Similarly, cigarette manufacturers cannot avoid liability for causing lung cancer by (1) collecting enormous amounts of data, (2) investigating the relationship between cigarette smoking and hundreds of health problems, and then (3) claiming they were destined to find some statistically significant effect or another.2Some accusations of data-dredging, I claim, are instances of a fallacy that Mayo typically distinguishes for her readers with great care: the conflation of the pre-experimental reliability of a test with the postexperimental evidence provided by a particular data set.3 The history of science is replete with examples of unlikely, happy discoveries, where the data unequivocally support a hypothesis that scientists did not intend to test. In accessible, vivid prose, Mayo illustrates why one should typically be wary of data-dredging, but readers will be disappointed if they wish to learn how Mayo's two criteria can be used to distinguish between (a) hypotheses that are illicitly “dredged” and (b) hypotheses that are supported by one's data, despite being formulated postexperiment.Quibbles aside, Mayo's newest book is a yet another deep and engaging contribution to the philosophy of statistics and philosophy of science. Most importantly, it shows how foundational philosophical debates matter to pressing issues in science policy.