Basic Statistical Tests

Some Thoughts About Data Type, Distribution, and Statistical Significance

The relative efficiency and reliability of parametric and nonparametric sequential accuracy testing plans were investigated using Monte Carlo techniques with normal and non-normal distribution generators. For the normal distributions, zero (no equation bias) and nonzero (some equation bias) means were simulated. For normal distributions with a nonzero mean, a modified nonparametric testing plan was developed and compared with the bias-free parametric test. The parametric testing plans resulted in average sample numbers to reach a terminal decision that were 20–40% smaller than the nonparametric testing plans, when the prediction errors of a model were normally distributed. In this circumstance, both the parametric and nonparametric testing plans yielded actual probabilities of making a type I or II error close to the nominal values, with the parametric testing plans giving somewhat lower error probabilities. When model errors followed a non-normal distribution, the nonparametric testing plans still resulted in error probabilities close to the nominal values, but this was not the case for the parametric testing plans.

Statistics serves as the foundation for various domains such as businesses, policy makers, markets and other sectors involved in activities such as data collection, planning, design, analysis, interpretation and reporting of the research ndings. Moreover, the ndings obtained from a research project carry greater signicance when subjected to analysis using statistical techniques. Various methods are employed to ensure precise data interpretation within the realm of research. Considering the data parameters, we have two methodologies – Parametric tests and Nonparametric tests. Parametric test depends on the presence of normal distributions within the data, whereas Nonparametric tests are suitable for analysing continuous data in any format. This article provides an explanation of a range of Parametric and Nonparametric tests along with R code.

Different tests, different conclusions: evolutionary trees

The notion that nonparametric methods are required as a replacement of parametric statistical methods when the scale of measurement in a research study does not achieve a certain level was discussed in light of recent developments in representational measurement theory. A new approach to examining the problem via computer simulation was introduced. Some of the beliefs that have been widely held by psychologists for several decades were examined by means of a computer simulation study that mimicked measurement of an underlying empirical structure and performed two - sample Student t - tests on the resulting sample data. It was concluded that there is no need to replace parametric statistical tests by nonparametric methods when the scale of measurement is ordinal and not interval.Stevens' (1946) classic paper on the theory of scales of measurement triggered one of the longest standing debates in behavioural science methodology. The debate -- referred to as the levels of measurement controversy, or measurement - statistics debate -- is over the use of parametric and nonparametric statistics and its relation to levels of measurement. Stevens (1946; 1951; 1959; 1968), Siegel (1956), and most recently Siegel and Castellan (1988) and Conover (1980) argue that parametric statistics should be restricted to data of interval scale or higher. Furthermore, nonparametric statistics should be used on data of ordinal scale. Of course, since each scale of measurement has all of the properties of the weaker measurement, statistical methods requiring only a weaker scale may be used with the stronger scales. A detailed historical review linking Stevens' work on scales of measurement to the acceptance of psychology as a science, and a pedagogical presentation of fundamental axiomatic (i.e., representational) measurement can be found in Zumbo and Zimmerman (1991).Many modes of argumentation can be seen in the debate about levels of measurement and statistics. This paper focusses almost exclusively on an empirical form of rhetoric using experimental mathematics (Ripley, 1987). The term experimental mathematics comes from mathematical physics. It is loosely defined as the mimicking of the rules of a model of some kind via random processes. In the methodological literature this is often referred to as simulation. However, for the purpose of this paper, the terms experimental mathematics or computer simulation are preferred to because the latter is typically referred to when examining the robustness of a test in relation to particular statistical assumptions. Measurement level is not an assumption of the parametric statistical model (see Zumbo & Zimmerman, 1991 for a discussion of this issue) and to call the method used herein monte carlo would imply otherwise. The term experimental mathematics emphasizes the modelling aspect of the present approach to the debate.The purpose of this paper is to present a new paradigm using experimental mathematics to examine the claims made in the levels of measurement controversy. As Michell (1986) demonstrated, the concern over levels of measurement is inextricably tied to the differing notions of measurement and scaling. Michell further argued that fundamental axiomatic measurement or representational theory (see, for example, Narens & Luce, 1986) is the only measurement theory which implies a relation between measurement scales and statistics. Therefore, the approach advocated in this paper is linked closely to representational theory. The novelty of this approach, to the authors knowledge, is in the use of experimental mathematics to mimic representational measurement. Before describing the methodology used in this paper, we will briefly review its motivation.Admissible TransformationsRepresentational theory began in the late 1950's with Scott and Suppes (1958) and later with Suppes and Zinnes (1963), Pfanzagl (1968), and Krantz, Luce, Suppes & Tversky (1971). …

This article reviews and compares different existing statistical tests utilized in significance analysis of gene expression from DNA microarray data. Microarray gene expression study facilitates the investigation of genes that are differentially expressed with respect to different classes of samples. Many gene expression studies also aim to find groups of genes that do something together, or to find out molecular similarities among a group of samples. In all respects, it is very important to know whether there is a difference between two sets of results, and whether that difference is likely to occur due to random variations of the datasets. Hence instead of simple data analysis, statistical tests are the proper way to check statistical relevance. Basically, the statistical significance test is compulsory for yielding an ordered list of relevant genes in terms of differential expression to the investigator. This article surveys a few important statistical tests which include both parametric and nonparametric tests and compares those statistical tests with respect to different performance metrics. The statistical tests have been applied on several artificial and real-life microarray gene expression datasets. Most of the time, real-life datasets contain noise and noisy data can have undesirable effect on the results of any data mining analysis. Therefore, performance analysis on noisy datasets for different statistical tests has also been performed to facilitate the comparative study. Furthermore, correlation analysis of the resultant genes from different statistical methods has been accomplished. Keywords: Analysis, correlation analysis, eigenvector, nonparametric tests, P-value, parametric tests, principal component, statistical tests.

Like any other science, research in natural language processing (NLP) depends on the ability to draw correct conclusions from experiments. A key tool for this is statistical significance testing: We use it to judge whether a result provides meaningful, generalizable findings or should be taken with a pinch of salt. When comparing new methods against others, performance metrics often differ by only small amounts, so researchers turn to significance tests to show that improved models are genuinely better. Unfortunately, this reasoning often fails because we choose inappropriate significance tests or carry them out incorrectly, making their outcomes meaningless. Or, the test we use may fail to indicate a significant result when a more appropriate test would find one. NLP researchers must avoid these pitfalls to ensure that their evaluations are sound and ultimately avoid wasting time and money through incorrect conclusions.This book guides NLP researchers through the whole process of significance testing, making it easy to select the right kind of test by matching canonical NLP tasks to specific significance testing procedures. As well as being a handbook for researchers, the book provides theoretical background on significance testing, includes new methods that solve problems with significance tests in the world of deep learning and multidataset benchmarks, and describes the open research problems of significance testing for NLP.The book focuses on the task of comparing one algorithm with another. At the core of this is the p-value, the probability that a difference at least as extreme as the one we observed could occur by chance. If the p-value falls below a predetermined threshold, the result is declared significant. Leaving aside the fundamental limitation of turning the validity of results into a binary question with an arbitrary threshold, to be a valid statistical significance test, the p-value must be computed in the right way. The book describes the two crucial properties of an appropriate significance test: The test must be both valid and powerful. Validity refers to the avoidance of type 1 errors, in which the result is incorrectly declared significant. Common mistakes that lead to type 1 errors include deploying tests that make incorrect assumptions, such as independence between data points. The power of a test refers to its ability to detect a significant result and therefore to avoid type 2 errors. Here, knowledge of the data and experiment must be used to choose a test that makes the correct assumptions. There is a trade-off between validity and power, but for the most common NLP tasks (language modeling, sequence labeling, translation, etc.), there are clear choices of tests that provide a good balance.Beginning with a detailed background on significance testing, the book then shows the reader how to carry out tests for specific NLP tasks. There is a mix of styles, with the first four chapters providing reference material that will be extremely useful to both new and experienced researchers. Here, it is easy to find the material related to a given NLP task. The next two chapters discuss more recent research into the application of significance tests to deep neural networks and for testing across multiple datasets. Alongside open research questions, these later chapters provide clear guidelines on how to apply the proposed methods. It is this mix of background material and reference guidelines that I believe makes this book so compelling and nicely self-contained.The introduction in Chapter 1 motivates the need for a comprehensive textbook and outlines challenges that the later chapters address more deeply. The theoretical background material begins in Chapter 2, which introduces core concepts, including hypothesis testing, type 1 and type 2 errors, validity and power, and p-values. The reader does not need to have any prior knowledge of statistical significance tests to follow this part. However, experienced readers could still benefit from reading this chapter, as concepts such as p-values are widely misunderstood and misused (Amrhein, Greenland, and McShane 2019).The significance tests themselves are introduced in Chapter 3, categorized into parametric and nonparametric tests. The chapter begins with the intuitively simple paired z-test, then builds up to more commonly-applied techniques, showing the connections and assumptions that each test makes. Step-by-step algorithms help the reader to implement each test. Although the chapter does cite uses of tests in NLP research, the main purpose is to present the theory behind each test and point out their differences.Chapter 4 provides perhaps the most handy part of the book for reference: a correspondence between common NLP tasks and statistical tests. Each task is discussed in terms of the evaluation metrics used, then a decision tree is introduced to guide the reader toward a choice between a parametric test, bootstrap or randomization test, or sampling-free nonparametric test. Section 4.3 then links each NLP evaluation measure to a specific significance test, presenting a large table that helps readers identify which test they need for a specific task. Particular considerations for each task are also pointed out to provide more detail about the appropriate options. The final part of this chapter describes the issue of p-hacking, in which dataset sizes are increased until a significance threshold is reached without consideration for biases in the data (discussed, for example, in Hofmann [2015]). The chapter proposes a simple solution to ensure robust significance testing with large datasets.Where Chapter 4 presents well-established methods, Chapter 5 introduces the current research question of how best to apply statistical significance testing to deep learning. Non-convex loss functions, stochastic optimization, random initialization, and a multitude of hyperparameters limit the conclusions we can draw from a single test run of a deep neural network (DNN). This chapter, which is based on the authors’ ACL paper (Dror, Shlomov, and Reichart 2019), explains how the comparison process can be overhauled to provide more meaningful evaluations. Beginning by explaining the difficulties of evaluating DNNs, the chapter then introduces criteria for a comparison framework, then discusses the limitations of current methods. Reimers and Gurevych (2018) have previously tackled this problem, but their approach has limited power and does not provide a confidence score. Other works, such as Clark et al. (2011), compare DNNs using a collection of statistics, such as the mean or standard deviation of performance across runs. This book shows how such an approach violates the assumptions of the significance tests. The authors propose almost stochastic dominance as the basis for a better alternative. The chapter explains how to use the proposed method, evaluates it in an empirical case study, and finally analyzes the errors made by each testing approach.Large NLP models are often tested across a range of datasets, which presents another problem for standard significance testing. Chapter 6 discusses the challenges of assessing two questions: (1) On how many datasets does algorithm A outperform algorithm B? (2) On which datasets does A outperform B? Applying standard significance tests individually to each dataset and counting the number of significant results is likely to overestimate the total number of significant results, as this chapter explains. The authors then present a new framework for replicability analysis, based on partial conjunction testing, and discuss two variants (Bonferroni and Fisher) for when the datasets are independent or dependent. They introduce a method based on Benjamini and Heller (2008) to count the number of datasets where one method outperforms another, then show how to use the Holm procedure (Holm 1979) to identify which datasets these are. Chapter 6 provides a lot of detailed background on the proposed replicability analysis framework, and the later sections again link the process to specific NLP case studies, and step-by-step summaries help the reader to apply the methodology. Extensive empirical results illustrate the very substantial differences in outcomes between the proposed approach and standard procedures.The final two chapters present open problems and conclude, showing that the topic has many interesting research questions of its own, such as problems when performing cross-validation, and the limited statistical power of replicability analysis.Overall, I highly recommend this book to a wide range of NLP researchers, from new students to seasoned experts who wish to ensure that they compare methods effectively. The book is excellent as both an introduction to the topic of significance testing and as a reference to use when evaluating your results. For anyone with further interest in the topic, it also points the way to future work. If one could level any criticism at this book at all, it is that it does not deeply discuss the basic flaws of significance testing or what the alternatives might be. For now, though, significance testing is an integral part of NLP research and this book provides a great resource for researchers who wish to perform it correctly and painlessly.

Parametric and nonparametric market power tests most commonly used to assess imperfectly competitive behavior are identified. Monte Carlo experiments are used to assess the accuracy of eight nonparametric tests. The results are compared to Raper, Love, and Shumway's (1997) findings concerning three parametric market power tests in the Bresnahan-Lau tradition. Both monopolistic and monopsonistic market power tests are implemented using data from 10 known market structures. Only two of the nonparametric market power tests distinguish between market structures adequately. The parametric tests perform well, although functional form bias is not investigated in this study.

For nonnormal data we suggest a test of location based on a broader family of distributions than normality. Such a test will in a sense fall between the standard parametric and non parametric tests. We see that the Wald tests based on this family of distributions have some advantages over the score tests and that they perform well in comparison to standard parametric and nonparametric tests in a variety of situations. We also consider when and how to apply such tests in practice.

Objective: The aim of this paper is to show the application of the Non-Parametric Sign Test in problems involving the testing of central tendency values. Theoretical framework: Non-parametric methods are widely used in the study of populations that are taken in rank order (such as a movie that receives one to four star ratings). The use of non-parametric methods may also be necessary when the data has a ranking but no clear numerical interpretation, such as when accessing preferences. In terms of scale, non-parametric methods result in data that is "in order" (Thatcher et al., 2005) Method: Data was taken from a company in the South of Rio de Janeiro and a case study was carried out using the Non-Parametric Sign Test. Final Considerations: Initially it proved unfeasible to use a Parametric Test because the Anderson-Darling Test showed that the Assumption of Normality was not confirmed and finally the Non-Parametric Sign Test showed that the hypothetical Median really is the correct measure of Central Tendency. Implications of the research: The use of Non-Parametric Tests is widespread in scientific literature and has proven to be highly effective in dealing with data where the assumptions of Normality are not confirmed. Originality/value: Despite being well-known statistical tools, Non-Parametric Tests are widely used and can bring innovations to their application, as in the case of the company in question.

Hypothesis tests are a crucial statistical tool for data mining and are the workhorse of scientific research in many fields. Here we study differentially private tests of independence between a categorical and a continuous variable. We take as our starting point traditional nonparametric tests, which require no distributional assumption (e.g., normality) about the data distribution. We present private analogues of the Kruskal-Wallis, Mann-Whitney, and Wilcoxon signed-rank tests, as well as the parametric one-sample t-test. These tests use novel test statistics developed specifically for the private setting. We compare our tests to prior work, both on parametric and nonparametric tests. We find that in all cases our new nonparametric tests achieve large improvements in statistical power, even when the assumptions of parametric tests are met.

Background: Breast cancer is the most common cancer among Indian women. Although advances in treatment have improved survival rates, many patients continue to face physical and emotional difficulties that impact quality of life (QOL) even after completing therapy. This study aimed to assess the QOL in breast-cancer patients three months after treatment using the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire (EORTC QLQ-C30).Methods: A prospective observational study was conducted on 100 breast-cancer patients at Postgraduate Institute of Medical Education and Research (PGIMER), Chandigarh. Patients were assessed three months after treatment completion. Sociodemographic and clinical details were recorded. QOL was measured using the EORTC QLQ-C30. Descriptive statistics were performed. Parametric tests were applied for continuous variables with normal distribution; otherwise, non-parametric tests were used. Pearson’s and Spearman’s correlation coefficients, along with multiple regression modelling, were used wherever applicable. Results: Mean age of study population was 51.02±12.16 years. Mean Global Health Score (GHS) QOL score was 55.17± 13.75, which was lower than the EORTC reference value of 61.8 ±24.6. Among functional scales, the best score was for emotional functioning (EF) (91.00 ± 11.22) while social functioning (SF) scored the lowest (74.33± 18.26). Pain, insomnia, and fatigue were the most worrisome symptoms. Elder age negatively affected GHS, physical functioning (PF), and cognitive functioning (CF). Being married had a significant positive impact on GHS, PF, and role functioning (RF). Living in a nuclear family and having an occupation have a significant positive impact on GHS.Conclusions: There were many factors which were related to lower QOL, including age, marital status, occupation status, and type of family. This study highlights the women at risk of a poorer QOL after breast-cancer treatment in Indian society.

Naturopathic medicine is a non-invasive and evidence-based system of medicine imparting treatments using natural modalities.This study was conducted to determine the effect of revulsive compress in improving pain, disability and spinal flexibility in patients with low back pain. A total of sixty subjects were recruited after fulfilling the inclusion and exclusion criteria. They were randomly allocated to the case group which received revulsive compress for ten days and the control group, without intervention, put on wait-list. Every subject was assessed for pain, disability and spinal flexibility at baseline and following 10 days. The data were analyzed for the normal distribution using the Kolmogorov Smirnov test. Parametric tests, t-test and paired t-test were applied for sit and reach test as the data measured was normally distributed. Non-parametric tests, ie., Mann-Whitney Test and Wilcoxon signed-rank tests were used for VAS and ODI as the data measured with these were non-normally distributed.There was a significant improvement in the sit and reach test (p≤0.004), VAS (p≤0.0001) and ODI (p≤0.0001) in the case group with respect to the control group. The study result suggests that the revulsive compress group showed significance for VAS Score, ODI, & sit and reach test with respect to the control group. Hence, revulsive compress may be an effective hydrotherapeutic treatment for a reduction in pain and disability and in improving the flexibility in low back pain patients.

This study aimed at characterization of rainfall dynamics in a hot arid region of Gujarat, India by employing time-series modeling techniques and sustainability approach. Five characteristics, i.e., normality, stationarity, homogeneity, presence/absence of trend, and persistence of 34-year (1980–2013) period annual rainfall time series of ten stations were identified/detected by applying multiple parametric and non-parametric statistical tests. Furthermore, the study involves novelty of proposing sustainability concept for evaluating rainfall time series and demonstrated the concept, for the first time, by identifying the most sustainable rainfall series following reliability (R y), resilience (R e), and vulnerability (V y) approach. Box–whisker plots, normal probability plots, and histograms indicated that the annual rainfall of Mandvi and Dayapar stations is relatively more positively skewed and non-normal compared with that of other stations, which is due to the presence of severe outlier and extreme. Results of Shapiro–Wilk test and Lilliefors test revealed that annual rainfall series of all stations significantly deviated from normal distribution. Two parametric t tests and the non-parametric Mann–Whitney test indicated significant non-stationarity in annual rainfall of Rapar station, where the rainfall was also found to be non-homogeneous based on the results of four parametric homogeneity tests. Four trend tests indicated significantly increasing rainfall trends at Rapar and Gandhidham stations. The autocorrelation analysis suggested the presence of persistence of statistically significant nature in rainfall series of Bhachau (3-year time lag), Mundra (1- and 9-year time lag), Nakhatrana (9-year time lag), and Rapar (3- and 4-year time lag). Results of sustainability approach indicated that annual rainfall of Mundra and Naliya stations (R y = 0.50 and 0.44; R e = 0.47 and 0.47; V y = 0.49 and 0.46, respectively) are the most sustainable and dependable compared with that of other stations. The highest values of sustainability index at Mundra (0.120) and Naliya (0.112) stations confirmed the earlier findings of R y–R e–V y approach. In general, annual rainfall of the study area is less reliable, less resilient, and moderately vulnerable, which emphasizes the need of developing suitable strategies for managing water resources of the area on sustainable basis. Finally, it is recommended that multiple statistical tests (at least two) should be used in time-series modeling for making reliable decisions. Moreover, methodology and findings of the sustainability concept in rainfall time series can easily be adopted in other arid regions of the world.

In this study, researchers used respondents' data, such as gender, age and duration of work of respondents to be able to provide information about the characteristics of respondents. Where from the questionnaire distributed questionnaire as many as 65 respondents using census techniques. Data analysis with parametric and non parametric statistics using SEM-PLS (Structural Equation Modeling-Partial Least Square) regarding research variables, instrument test, normality test, hypothesis test, and discussion of the results of hypothesis testing and Path Path Analysis. This study uses path analysis (path analysis) to test the pattern of relationships that reveal the influence of variables or a set of variables on other variables, both direct influence and indirect influence. The results of the study are as follows: The effect of the X3 variable on X4 has a P-Values value of 0.006 0.05, so it can be stated that the effect between X3 on Y is significant. The effect of the X4 variable on Y has a P-Values value of 0.014> 0.05, so it can be stated that the effect between X4 on Y is significant. The effect of X1 on X4 has a P-Values value of 0.039 0.06, so it can be stated that the influence between X1 on Y is significant. The effect of X2 on X4 has a P-Values value of 0.003 <0.42, so it can be stated that the influence between X2 to X4 is significant. The effect of the X2 variable on Y has a P-Values value of 0.008 <0.18, so it can be stated that the effect between X2 on Y is significant.