Two‐Way Factorial: Distribution‐Free Methods

N‐of‐1 study designs involve the collection and analysis of repeated measures data from an individual not using an intervention and using an intervention. This study explores the use of semi‐parametric and parametric bootstrap tests in the analysis of N‐of‐1 studies under a single time series framework in the presence of autocorrelation. When the Type I error rates of bootstrap tests are compared to Wald tests, our results show that the bootstrap tests have more desirable properties. We compare the results for normally distributed errors with those for contaminated normally distributed errors and find that, except when there is relatively large autocorrelation, there is little difference between the power of the parametric and semi‐parametric bootstrap tests. We also experiment with two intervention designs: ABAB and AB, and show the ABAB design has more power. The results provide guidelines for designing N‐of‐1 studies, in the sense of how many observations and how many intervention changes are needed to achieve a certain level of power and which test should be performed.

Like any other science, research in natural language processing (NLP) depends on the ability to draw correct conclusions from experiments. A key tool for this is statistical significance testing: We use it to judge whether a result provides meaningful, generalizable findings or should be taken with a pinch of salt. When comparing new methods against others, performance metrics often differ by only small amounts, so researchers turn to significance tests to show that improved models are genuinely better. Unfortunately, this reasoning often fails because we choose inappropriate significance tests or carry them out incorrectly, making their outcomes meaningless. Or, the test we use may fail to indicate a significant result when a more appropriate test would find one. NLP researchers must avoid these pitfalls to ensure that their evaluations are sound and ultimately avoid wasting time and money through incorrect conclusions.This book guides NLP researchers through the whole process of significance testing, making it easy to select the right kind of test by matching canonical NLP tasks to specific significance testing procedures. As well as being a handbook for researchers, the book provides theoretical background on significance testing, includes new methods that solve problems with significance tests in the world of deep learning and multidataset benchmarks, and describes the open research problems of significance testing for NLP.The book focuses on the task of comparing one algorithm with another. At the core of this is the p-value, the probability that a difference at least as extreme as the one we observed could occur by chance. If the p-value falls below a predetermined threshold, the result is declared significant. Leaving aside the fundamental limitation of turning the validity of results into a binary question with an arbitrary threshold, to be a valid statistical significance test, the p-value must be computed in the right way. The book describes the two crucial properties of an appropriate significance test: The test must be both valid and powerful. Validity refers to the avoidance of type 1 errors, in which the result is incorrectly declared significant. Common mistakes that lead to type 1 errors include deploying tests that make incorrect assumptions, such as independence between data points. The power of a test refers to its ability to detect a significant result and therefore to avoid type 2 errors. Here, knowledge of the data and experiment must be used to choose a test that makes the correct assumptions. There is a trade-off between validity and power, but for the most common NLP tasks (language modeling, sequence labeling, translation, etc.), there are clear choices of tests that provide a good balance.Beginning with a detailed background on significance testing, the book then shows the reader how to carry out tests for specific NLP tasks. There is a mix of styles, with the first four chapters providing reference material that will be extremely useful to both new and experienced researchers. Here, it is easy to find the material related to a given NLP task. The next two chapters discuss more recent research into the application of significance tests to deep neural networks and for testing across multiple datasets. Alongside open research questions, these later chapters provide clear guidelines on how to apply the proposed methods. It is this mix of background material and reference guidelines that I believe makes this book so compelling and nicely self-contained.The introduction in Chapter 1 motivates the need for a comprehensive textbook and outlines challenges that the later chapters address more deeply. The theoretical background material begins in Chapter 2, which introduces core concepts, including hypothesis testing, type 1 and type 2 errors, validity and power, and p-values. The reader does not need to have any prior knowledge of statistical significance tests to follow this part. However, experienced readers could still benefit from reading this chapter, as concepts such as p-values are widely misunderstood and misused (Amrhein, Greenland, and McShane 2019).The significance tests themselves are introduced in Chapter 3, categorized into parametric and nonparametric tests. The chapter begins with the intuitively simple paired z-test, then builds up to more commonly-applied techniques, showing the connections and assumptions that each test makes. Step-by-step algorithms help the reader to implement each test. Although the chapter does cite uses of tests in NLP research, the main purpose is to present the theory behind each test and point out their differences.Chapter 4 provides perhaps the most handy part of the book for reference: a correspondence between common NLP tasks and statistical tests. Each task is discussed in terms of the evaluation metrics used, then a decision tree is introduced to guide the reader toward a choice between a parametric test, bootstrap or randomization test, or sampling-free nonparametric test. Section 4.3 then links each NLP evaluation measure to a specific significance test, presenting a large table that helps readers identify which test they need for a specific task. Particular considerations for each task are also pointed out to provide more detail about the appropriate options. The final part of this chapter describes the issue of p-hacking, in which dataset sizes are increased until a significance threshold is reached without consideration for biases in the data (discussed, for example, in Hofmann [2015]). The chapter proposes a simple solution to ensure robust significance testing with large datasets.Where Chapter 4 presents well-established methods, Chapter 5 introduces the current research question of how best to apply statistical significance testing to deep learning. Non-convex loss functions, stochastic optimization, random initialization, and a multitude of hyperparameters limit the conclusions we can draw from a single test run of a deep neural network (DNN). This chapter, which is based on the authors’ ACL paper (Dror, Shlomov, and Reichart 2019), explains how the comparison process can be overhauled to provide more meaningful evaluations. Beginning by explaining the difficulties of evaluating DNNs, the chapter then introduces criteria for a comparison framework, then discusses the limitations of current methods. Reimers and Gurevych (2018) have previously tackled this problem, but their approach has limited power and does not provide a confidence score. Other works, such as Clark et al. (2011), compare DNNs using a collection of statistics, such as the mean or standard deviation of performance across runs. This book shows how such an approach violates the assumptions of the significance tests. The authors propose almost stochastic dominance as the basis for a better alternative. The chapter explains how to use the proposed method, evaluates it in an empirical case study, and finally analyzes the errors made by each testing approach.Large NLP models are often tested across a range of datasets, which presents another problem for standard significance testing. Chapter 6 discusses the challenges of assessing two questions: (1) On how many datasets does algorithm A outperform algorithm B? (2) On which datasets does A outperform B? Applying standard significance tests individually to each dataset and counting the number of significant results is likely to overestimate the total number of significant results, as this chapter explains. The authors then present a new framework for replicability analysis, based on partial conjunction testing, and discuss two variants (Bonferroni and Fisher) for when the datasets are independent or dependent. They introduce a method based on Benjamini and Heller (2008) to count the number of datasets where one method outperforms another, then show how to use the Holm procedure (Holm 1979) to identify which datasets these are. Chapter 6 provides a lot of detailed background on the proposed replicability analysis framework, and the later sections again link the process to specific NLP case studies, and step-by-step summaries help the reader to apply the methodology. Extensive empirical results illustrate the very substantial differences in outcomes between the proposed approach and standard procedures.The final two chapters present open problems and conclude, showing that the topic has many interesting research questions of its own, such as problems when performing cross-validation, and the limited statistical power of replicability analysis.Overall, I highly recommend this book to a wide range of NLP researchers, from new students to seasoned experts who wish to ensure that they compare methods effectively. The book is excellent as both an introduction to the topic of significance testing and as a reference to use when evaluating your results. For anyone with further interest in the topic, it also points the way to future work. If one could level any criticism at this book at all, it is that it does not deeply discuss the basic flaws of significance testing or what the alternatives might be. For now, though, significance testing is an integral part of NLP research and this book provides a great resource for researchers who wish to perform it correctly and painlessly.

Interacting Biases, Non-Normal Return Distributions and the Performance of Parametric and Bootstrap Tests for Long-Horizon Event Studies

Many time series in diverse fields of application may exhibit long-memory. The class of fractionally integrated (FI) processes can be used to try to model this strong data dependence. Asymptotic tests for FI include the re-scaled range statistic test and its modified form, the frequency-domain regression-based procedure, the modified Higuchi's test and Jensen's test. De Peretti and Marimoutou (2002) finds that proper finite-sample inferences are not possible using these techniques without correcting for size distortions. Some attempt this correction through 'bootstrapping', but this method is not perfect and needs more study and improvements. In this paper, I examine and compare the finite-sample properties of parametric and nonparametric bootstrap tests by using graphical techniques of Davidson and MacKinnon (1998a) for showing whether they properly correct the distortions while retaining their power relative to the corresponding asymptotic tests. One of the tests uses a double bootstrap that provide better true power and size properties. I use a bilateral P value that permits the true power of the tests to grow when the size distortions are asymmetric. We then apply these procedures to a real time series to investigate its long memory properties.

Principal component analysis is a multivariate technique widely used in dimensionality reduction. The ideal number of principal components retained should be defined when one is dealing with high-dimensional data. Some criteria for this choice were proposed in the literature. Most of them have serious limitations, such as normality assumptions, subjective analysis, and asymptotic properties. This study aims to propose two new tests using the parametric bootstrap for determining the optimal number of principal components (PC) retained for subsequent analysis, based on the amount of the total variation accounted for by the k first principal components. The performances of these tests were compared among themselves and with those of Fujikoshi (1980) and Gebert and Ferreira (2010) through Monte Carlo simulations. Under multivariate normality the two proposed parametric bootstrap tests are recommended. Under nonnormality the test of Gebert and Ferreira (2010) is recommended. The three bootstrap tests surpass t...

Paleomagnetic statistical inference is underpinned by a family of parametric null hypothesis tests. In many cases, however, paleomagnetic data do not meet the distributional assumptions of these tests, which can lead to spurious inferences. Earlier studies have proposed the bootstrap as a nonparametric alternative for paleomagnetic analysis, which can be applied even when the distributional form of the data is unknown. Key among these approaches is the bootstrap test for a common mean direction, which relies on assessment of the overlap of estimated confidence regions. In its current form, the bootstrap test for a common mean paleomagnetic direction does not consider a null hypothesis and can yield outcomes that cannot be interpreted in terms of a statistical significance level. To resolve these issues, we use recent advances to place such bootstrap tests within a null hypothesis significance testing framework, and unify them with the existing family of paleomagnetic statistical tests. Furthermore, using numerical experiments we demonstrate the applicability of such a nonparametric approach to moderately sized paleomagnetic data sets typical of modern and legacy studies. Finally, we demonstrate how a confidence region can be estimated for the common mean of two sets of directions and how known directions, such as the expected field produced by a geocentric axial dipole, can be compared to that mean.

The problem of testing the null hypothesis of a common direction across several populations defined on the hypersphere arises frequently when we deal with directional data. We may consider the Analysis of Variance (ANOVA) for testing such hypotheses. However, for the Watson distribution, a commonly used distribution for modeling axial data, the ANOVA test is only valid for large concentrations. So we suggest to use alternative tests, such as bootstrap and permutation tests in ANOVA. Then, we investigate the performance of these tests for data from Watson populations defined on the hypersphere.

The effect of cycling parameter variation (i.e. oxidation temperature, upper and lower dwell time, humidity in test gas) of the oxidation/spallation kinetics on four alloys was investigated within the framework of an EC funded research project (COTEST). For this purpose, specimens of AISI 441, Alloy 800H, CM 247 and P91 were subjected to thermocyclic testing in dry or humidified air. It was found that a minimum of 300 h accumulated hot dwell time is required for meaningful test results. Detailed characterisation of the corrosion products was performed using OM, SEM/EDX and XRD. The net weight change curves were evaluated with regards to the characteristic quantitative parameters describing oxide growth rate, time to onset of spallation/breakaway and weight of spalled oxide. Analysis of these values was made by the ANOVA method, which allows assessment of the significance of the roles of different test parameters and parameter variations on the oxidation/spallation kinetics from a limited number of experiments.

Simulation-based calculation of p-values is an important technique in situations where it is difficult to obtain an exact or approximate distribution for a test, or when such an approximation exists but is of dubious validity, either because the conditions it requires are not met, or because it relies on questionable assumptions about the distribution of the data. But even in applications where we are fairly confident in a particular parametric model and the statistical analysis based on that model, it can still be helpful, in the spirit of robustness, to see what can be inferred from the data without particular parametric model assumptions. A substantial literature has demonstrated both theoretically and in numerical studies that the bootstrap is widely effective (Davison and Hinkley in Bootstrap methods and their applications. Cambridge University Press, Cambridge, 1997; Efron and Tibshirani in An introduction to the bootstrap. Chapman and Hall, New York, 1993). But the simplicity of the bootstrap conceals an important point: its properties depend on how resampling is done; arbitrary shuffles of the data do not necessarily accomplish desired statistical goals. Moreover, in the context of hypotheses testing, the p-value must be obtained under the hypothetical reality imposed by the null hypothesis. In this chapter, we review the general framework for statistical tests of hypotheses, and introduce the basics for Monte Carlo, permutation, and bootstrap tests.

Nonparametric Statistical Tests: A Computational Approach describes classical nonparametric tests, as well as novel and little-known methods such as the Baumgartner-Weiss-Schindler and the Cucconi tests. The book presents SAS and R programs, allowing readers to carry out the different statistical methods, such as permutation and bootstrap tests. Th

The two-sample problem for Cronbach's coefficient [Formula: see text], as an estimate of test or composite score reliability, has attracted little attention compared to the extensive treatment of the one-sample case. It is necessary to compare the reliability of a test for different subgroups, for different tests or the short and long forms of a test. In this paper, we study statistical procedures of comparing two coefficients [Formula: see text] and [Formula: see text]. The null hypothesis of interest is [Formula: see text], which we test against one-or two-sided alternatives. For this purpose, resampling-based permutation and bootstrap tests are proposed for two-group multivariate non-normal models under the general asymptotically distribution-free (ADF) setting. These statistical tests ensure a better control of the type-I error, in finite or very small sample sizes, when the state-of-affairs ADF large-sample test may fail to properly attain the nominal significance level. By proper choice of a studentized test statistic, the resampling tests are modified in order to be valid asymptotically even in non-exchangeable data frameworks. Moreover, extensions of this approach to other designs and reliability measures are discussed as well. Finally, the usefulness of the proposed resampling-based testing strategies is demonstrated in an extensive simulation study and illustrated by real data applications.

SummaryHumans show a remarkable ability to discriminate others' gaze direction, even though a given direction can be conveyed by many physically dissimilar configurations of different eye positions and head views. For example, eye contact can be signaled by a rightward glance in a left-turned head or by direct gaze in a front-facing head. Such acute gaze discrimination implies considerable perceptual invariance. Previous human research found that superior temporal sulcus (STS) responds preferentially to gaze shifts [1], but the underlying representation that supports such general responsiveness remains poorly understood. Using multivariate pattern analysis (MVPA) of human functional magnetic resonance imaging (fMRI) data, we tested whether STS contains a higher-order, head view-invariant code for gaze direction. The results revealed a finely graded gaze direction code in right anterior STS that was invariant to head view and physical image features. Further analyses revealed similar gaze effects in left anterior STS and precuneus. Our results suggest that anterior STS codes the direction of another's attention regardless of how this information is conveyed and demonstrate how high-level face areas carry out fine-grained, perceptually relevant discrimination through invariance to other face features.

While permutation tests and bootstraps have very wide‐ranging application, both share a common potential drawback: as data‐intensive resampling methods, both can be runtime prohibitive when applied to large‐ or even medium‐sized data samples drawn from large datasets. The data explosion over the past few decades has made this a common occurrence, and it highlights the increasing need for faster, and more efficient and scalable, permutation test and bootstrap algorithms. Seven bootstrap and six permutation test algorithms coded in SAS (the largest privately owned software firm globally) are compared herein. The fastest algorithms (‘OPDY’ for the bootstrap, ‘OPDN’ for permutation tests) are new, use no modules beyond Base SAS, and achieve speed increases orders of magnitude faster than the relevant ‘built‐in’ SAS procedures (OPDY is over 200× faster than Proc SurveySelect; OPDN is over 240× faster than Proc SurveySelect, over 350× faster than NPAR1WAY (which crashes on datasets less than a 10th the size OPDN can handle), and over 720× faster than Proc Multtest). OPDY also is much faster than hashing, which crashes on datasets smaller—sometimes by orders of magnitude—than OPDY can handle. OPDY is easily generalizable to multivariate regression models, and OPDN, which uses an extremely efficient draw‐by‐draw random‐sampling‐without‐replacement algorithm, can use virtually any permutation statistic, so both have a very wide range of application. And the time complexity of both OPDY and OPDN is sublinear, making them not only the fastest but also the only truly scalable bootstrap and permutation test algorithms, respectively, in SAS. WIREs Comput Stat 2013. doi: 10.1002/wics.1266This article is categorized under: Statistical and Graphical Methods of Data Analysis > Bootstrap and Resampling Statistical and Graphical Methods of Data Analysis > Nonparametric Methods Software for Computational Statistics > Software/Statistical Software

Partial least square (PLS) methods (also sometimes called projection to latent structures) relate the information present in two data tables that collect measurements on the same set of observations. PLS methods proceed by deriving latent variables which are (optimal) linear combinations of the variables of a data table. When the goal is to find the shared information between two tables, the approach is equivalent to a correlation problem and the technique is then called partial least square correlation (PLSC) (also sometimes called PLS-SVD). In this case there are two sets of latent variables (one set per table), and these latent variables are required to have maximal covariance. When the goal is to predict one data table the other one, the technique is then called partial least square regression. In this case there is one set of latent variables (derived from the predictor table) and these latent variables are required to give the best possible prediction. In this paper we present and illustrate PLSC and PLSR and show how these descriptive multivariate analysis techniques can be extended to deal with inferential questions by using cross-validation techniques such as the bootstrap and permutation tests.

Background Contrast-enhanced mammography (CEM) and abbreviated breast MRI (ABMRI) are emerging alternatives to standard MRI for supplemental breast cancer screening. Purpose To compare the diagnostic performance of CEM, ABMRI, and standard MRI. Materials and Methods This single-institution, prospective, blinded reader study included female participants referred for breast MRI from January 2018 to June 2021. CEM was performed within 14 days of standard MRI; ABMRI was produced from standard MRI images. Two readers independently interpreted each CEM and ABMRI after a washout period. Examination-level performance metrics calculated were recall rate, cancer detection, and false-positive biopsy recommendation rates per 1000 examinations and sensitivity, specificity, and positive predictive value of biopsy recommendation. Bootstrap and permutation tests were used to calculate 95% CIs and compare modalities. Results Evaluated were 492 paired CEM and ABMRI interpretations from 246 participants (median age, 51 years; IQR, 43-61 years). On 49 MRI scans with lesions recommended for biopsy, nine lesions showed malignant pathology. No differences in ABMRI and standard MRI performance were identified. Compared with standard MRI, CEM demonstrated significantly lower recall rate (14.0% vs 22.8%; difference, -8.7%; 95% CI: -14.0, -3.5), lower false-positive biopsy recommendation rate per 1000 examinations (65.0 vs 162.6; difference, -97.6; 95% CI: -146.3, -50.8), and higher specificity (87.8% vs 80.2%; difference, 7.6%; 95% CI: 2.3, 13.1). Compared with standard MRI, CEM had significantly lower cancer detection rate (22.4 vs 36.6; difference, -14.2; 95% CI: -28.5, -2.0) and sensitivity (61.1% vs 100%; difference, -38.9%; 95% CI: -66.7, -12.5). The performance differences between CEM and ABMRI were similar to those observed between CEM and standard MRI. Conclusion ABMRI had comparable performance to standard MRI and may support more efficient MRI screening. CEM had lower recall and higher specificity compared with standard MRI or ABMRI, offset by lower cancer detection rate and sensitivity compared with standard MRI. These trade-offs warrant further consideration of patient population characteristics before widespread screening with CEM. Clinical trial registration no. NCT03517813 © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Chang in this issue.