Discovering Interesting Subpaths with Statistical Significance from Spatiotemporal Datasets

Commentary

Statistical significance tests are unnecessary even when properly done and properly interpreted: Reply to commentaries

AN ASSESSMENT OF SPATIAL VARIATION OF LAND SURFACE CHARACTERISTICS OF MINNA, NIGER STATE NIGERIA FOR SUSTAINABLE URBANIZATION USING GEOSPATIAL TECHNIQUES

Some Thoughts About Data Type, Distribution, and Statistical Significance

Long-term trend and abrupt changes of major climate variables in the Taihu Basin were investigated based on the mean, maximum and minimum air temperature, diurnal temperature range (DTR), precipitation, relative humidity and sunshine duration at six meteorological stations from 1954 to 2006. Results showed that the long-term trend for annual precipitation was not statistically significant during the past 53 years, but a wetter tendency was detected and the increasing centre for annual precipitation was located in the southeast of the study area. Mean air temperature has increased by 1.43°C, similar trends were also identified for maximum and minimum air temperature, which have increased by 1.06 and 1.54°C, respectively, while DTR exhibited a slight decreasing trend with a rate of about −0.09°C/(10 year). The annual mean relative humidity and sunshine duration exhibited a decreasing trend, with Kendall slope values of −0.99%/10 year and −7.797 h/10 year, respectively. Examination of long-range dependence showed that all climate variables exhibited strong persistence at annual scale except minimum air temperature. Detection of abrupt changes using nonparametric Mann–Kendall and Pettitt methods showed different results. Abrupt changes occurred in the 1980s and 2000s for annual precipitation using the nonparametric Mann–Kendall method, while no abrupt changes were detected using the Pettitt method. Abrupt changes of temperature and relative humidity took place in the early 1990s using the nonparametric Mann–Kendall method, which occurred in the late and mid 1980s using the Pettitt method, while abrupt changes of sunshine duration and DTR detected by two methods occurred in the similar period. The result will be helpful for local flood control and drought relief in urban planning and construction under future global climate change.

This book examines the transformations of recreational physical activity in the context of contemporary modernity, analyzing the complex challenges and opportunities it presents for individuals, communities, and policy systems. It begins by establishing the foundational concepts and theoretical frameworks of physical activity and recreation, then systematically explores its sociological, psychological, cultural, technological, urban, economic, and environmental dimensions. The book investigates how modern technological developments — including wearable devices, smart fitness applications, social media platforms, virtual reality, augmented reality, and electronic gaming — have fundamentally altered the nature of recreational practices and the concept of leisure itself. The book also sheds light on the escalating global phenomenon of physical inactivity, its multifaceted causes, and its comprehensive consequences for health and society. The pivotal role of the built environment and urban planning in either facilitating or impeding movement and active living is thoroughly examined. National and international policies and programs supporting recreational physical activity are reviewed, and the book concludes by linking physical activity to the United Nations Sustainable Development Goals (SDGs), presenting forward-looking perspectives on sports tourism, the role of physical activity in the workplace, and practical recommendations for researchers, policymakers, and practitioners. This work adopts a Multi-Dimensional Analytical Approach, integrating theoretical frameworks from multiple disciplines: social science — including Pierre Bourdieu's theory of Distinction, Anthony Giddens's theory of Late Modernity and the Reflexive Self, and Zygmunt Bauman's Liquid Modernity; psychology — including Albert Bandura's Social Cognitive Theory (SCT), Edward Deci and Richard Ryan's Self-Determination Theory (SDT), and Driver's Leisure Motivation Model; and environmental science — including the Socio-Ecological Model and the concept of the Built Environment. This integrated approach enables precise analysis of data and statistics drawn from reliable global sources, including the World Health Organization (WHO), the Centers for Disease Control and Prevention (CDC), and recent academic studies published in peer-reviewed scientific journals. This book is intended as a comprehensive academic reference for university students, researchers, and specialists in the fields of physical education, sports sciences, public health, social sciences, urban planning, and sports psychology. It is equally a valuable resource for policymakers and practitioners in the domains of health promotion and physical fitness, offering a rigorous research outlook and practical recommendations directed at enhancing recreational physical activity in contemporary societies, in accordance with international best practices and standards. Keywords: Recreational physical activity, modernity, digital revolution, physical inactivity, urban planning, social media, wearable devices, health policies, sustainable development, sports tourism

Tests of statistical significance do not provide information about the clinical significance of research results. It is of utmost importance for a researcher to not only know the statistical significance of the difference, but also the magnitude of the difference and whether it is clinically significant or not. Results might be statistically significant yet it may not have some clinical relevance and vice versa. The concept of clinical significance has received little discussion or debate in the dental literature. The purpose of this paper was to compare statistical and clinical significance, to provide an overview of the two, and to discuss some implications of these concepts for dental research with the help of some suitable examples.

Chapter 3 provides a brief overview of null hypothesis significance testing and points out its primary defects. It then outlines the neo-Fisherian account of tests of statistical significance, along with a second option contained in the philosophy of statistics known as the error-statistical philosophy, both of which are defensible. Tests of statistical significance are the most widely used means for evaluating hypotheses and theories in psychology. A massive critical literature has developed in psychology, and the behavioral sciences more generally, regarding the worth of these tests. The chapter provides a list of important lessons learned from the ongoing debates about tests of significance.

The goal of the article is to determine which components of sustainable and smart development of urban areas are the most important for the economy of a city. For this, regression, cluster and discriminant analysis are applied, using the data of the ranking positions of 180 cities of the world according to the Cities in Motion Index (CIMI) and its components for 2022. The Stata and Statgraphics 19 software packages are used for the calculations. The statistical significance of the input data is confirmed using descriptive statistics, and the normality of the data distribution was determined according to the Shapiro-Wilk test. A regression analysis (based on the least squares method) of the influence of the integral value of CIMI and its components (Human capital, Social cohesion, Environment, Governance, Urban planning, International profile, Technology, Mobility and Transportation) on its first component – Economy, is carried out. It testifies that only four indicators have a statistically significant impact: Cities in Motion, Environment, Urban planning, and International profile. Multiple regression, constructed using the strict screening procedure, confirms these findings; and discriminant analysis proves that the regression equation coefficients is used to predict the Economy variable. Analysis of Spearman’s and Kendall’s correlation matrices prove a close relationship between the Economy, Human capital, Governance, and Cities in motion; direct dependence between Cities in motion and such indicators as Technology, Urban planning, and International profile; average direct connection between Economy, Social cohesion and Mobility and transportation. Cluster analysis using the k-means method in the R Studio software environment made it possible to distinguish eight clusters of cities according to their ranking positions in relation to various parameters of the CIMI index (their number was calculated according to the Sturgess formula, and the optimality of their number is confirmed by the agglomeration scheme according to the Ward method). For the cities of the first cluster (17 cities, 9.44% of the total number analyzed, mostly world capitals), Cities in motion has the greatest impact on the Economy component, while Mobility and Transportation has a lesser impact; for the cities of the second cluster (23 cities, 12.78%, mostly large cities of the United States and China) it is Technology that has the greatest impact; for cities of the third cluster (35 cities, 19.44%, primarily powerful regional centers)it is Cities in motion, International profile, Mobility and transportation, Social cohesion, and Urban planning; for clusters four (9 cities, 5%) and five (6 cities, 3.33%), the regressions are not significant, so these clusters require further study for each city separately; for the cities of the sixth cluster (33 cities, 18, 33%, mostly developed European cities) the most important are Cities In motion, Environment, Governance, Mobility and transportation, Social cohesion, and Urban planning; for the cities of the seventh cluster (10 cities, 5.56%) – Human capital, Social cohesion, and Technology; for cities in the eighth cluster (47 cities, 26.11%, mostly cities facing economic obstacles to their development) – Cities in motion, Environment, Technology, and Urban planning. The discriminant analysis shows that the Environment indicator has the greatest impact on the division of clusters into groups.

Like any other science, research in natural language processing (NLP) depends on the ability to draw correct conclusions from experiments. A key tool for this is statistical significance testing: We use it to judge whether a result provides meaningful, generalizable findings or should be taken with a pinch of salt. When comparing new methods against others, performance metrics often differ by only small amounts, so researchers turn to significance tests to show that improved models are genuinely better. Unfortunately, this reasoning often fails because we choose inappropriate significance tests or carry them out incorrectly, making their outcomes meaningless. Or, the test we use may fail to indicate a significant result when a more appropriate test would find one. NLP researchers must avoid these pitfalls to ensure that their evaluations are sound and ultimately avoid wasting time and money through incorrect conclusions.This book guides NLP researchers through the whole process of significance testing, making it easy to select the right kind of test by matching canonical NLP tasks to specific significance testing procedures. As well as being a handbook for researchers, the book provides theoretical background on significance testing, includes new methods that solve problems with significance tests in the world of deep learning and multidataset benchmarks, and describes the open research problems of significance testing for NLP.The book focuses on the task of comparing one algorithm with another. At the core of this is the p-value, the probability that a difference at least as extreme as the one we observed could occur by chance. If the p-value falls below a predetermined threshold, the result is declared significant. Leaving aside the fundamental limitation of turning the validity of results into a binary question with an arbitrary threshold, to be a valid statistical significance test, the p-value must be computed in the right way. The book describes the two crucial properties of an appropriate significance test: The test must be both valid and powerful. Validity refers to the avoidance of type 1 errors, in which the result is incorrectly declared significant. Common mistakes that lead to type 1 errors include deploying tests that make incorrect assumptions, such as independence between data points. The power of a test refers to its ability to detect a significant result and therefore to avoid type 2 errors. Here, knowledge of the data and experiment must be used to choose a test that makes the correct assumptions. There is a trade-off between validity and power, but for the most common NLP tasks (language modeling, sequence labeling, translation, etc.), there are clear choices of tests that provide a good balance.Beginning with a detailed background on significance testing, the book then shows the reader how to carry out tests for specific NLP tasks. There is a mix of styles, with the first four chapters providing reference material that will be extremely useful to both new and experienced researchers. Here, it is easy to find the material related to a given NLP task. The next two chapters discuss more recent research into the application of significance tests to deep neural networks and for testing across multiple datasets. Alongside open research questions, these later chapters provide clear guidelines on how to apply the proposed methods. It is this mix of background material and reference guidelines that I believe makes this book so compelling and nicely self-contained.The introduction in Chapter 1 motivates the need for a comprehensive textbook and outlines challenges that the later chapters address more deeply. The theoretical background material begins in Chapter 2, which introduces core concepts, including hypothesis testing, type 1 and type 2 errors, validity and power, and p-values. The reader does not need to have any prior knowledge of statistical significance tests to follow this part. However, experienced readers could still benefit from reading this chapter, as concepts such as p-values are widely misunderstood and misused (Amrhein, Greenland, and McShane 2019).The significance tests themselves are introduced in Chapter 3, categorized into parametric and nonparametric tests. The chapter begins with the intuitively simple paired z-test, then builds up to more commonly-applied techniques, showing the connections and assumptions that each test makes. Step-by-step algorithms help the reader to implement each test. Although the chapter does cite uses of tests in NLP research, the main purpose is to present the theory behind each test and point out their differences.Chapter 4 provides perhaps the most handy part of the book for reference: a correspondence between common NLP tasks and statistical tests. Each task is discussed in terms of the evaluation metrics used, then a decision tree is introduced to guide the reader toward a choice between a parametric test, bootstrap or randomization test, or sampling-free nonparametric test. Section 4.3 then links each NLP evaluation measure to a specific significance test, presenting a large table that helps readers identify which test they need for a specific task. Particular considerations for each task are also pointed out to provide more detail about the appropriate options. The final part of this chapter describes the issue of p-hacking, in which dataset sizes are increased until a significance threshold is reached without consideration for biases in the data (discussed, for example, in Hofmann [2015]). The chapter proposes a simple solution to ensure robust significance testing with large datasets.Where Chapter 4 presents well-established methods, Chapter 5 introduces the current research question of how best to apply statistical significance testing to deep learning. Non-convex loss functions, stochastic optimization, random initialization, and a multitude of hyperparameters limit the conclusions we can draw from a single test run of a deep neural network (DNN). This chapter, which is based on the authors’ ACL paper (Dror, Shlomov, and Reichart 2019), explains how the comparison process can be overhauled to provide more meaningful evaluations. Beginning by explaining the difficulties of evaluating DNNs, the chapter then introduces criteria for a comparison framework, then discusses the limitations of current methods. Reimers and Gurevych (2018) have previously tackled this problem, but their approach has limited power and does not provide a confidence score. Other works, such as Clark et al. (2011), compare DNNs using a collection of statistics, such as the mean or standard deviation of performance across runs. This book shows how such an approach violates the assumptions of the significance tests. The authors propose almost stochastic dominance as the basis for a better alternative. The chapter explains how to use the proposed method, evaluates it in an empirical case study, and finally analyzes the errors made by each testing approach.Large NLP models are often tested across a range of datasets, which presents another problem for standard significance testing. Chapter 6 discusses the challenges of assessing two questions: (1) On how many datasets does algorithm A outperform algorithm B? (2) On which datasets does A outperform B? Applying standard significance tests individually to each dataset and counting the number of significant results is likely to overestimate the total number of significant results, as this chapter explains. The authors then present a new framework for replicability analysis, based on partial conjunction testing, and discuss two variants (Bonferroni and Fisher) for when the datasets are independent or dependent. They introduce a method based on Benjamini and Heller (2008) to count the number of datasets where one method outperforms another, then show how to use the Holm procedure (Holm 1979) to identify which datasets these are. Chapter 6 provides a lot of detailed background on the proposed replicability analysis framework, and the later sections again link the process to specific NLP case studies, and step-by-step summaries help the reader to apply the methodology. Extensive empirical results illustrate the very substantial differences in outcomes between the proposed approach and standard procedures.The final two chapters present open problems and conclude, showing that the topic has many interesting research questions of its own, such as problems when performing cross-validation, and the limited statistical power of replicability analysis.Overall, I highly recommend this book to a wide range of NLP researchers, from new students to seasoned experts who wish to ensure that they compare methods effectively. The book is excellent as both an introduction to the topic of significance testing and as a reference to use when evaluating your results. For anyone with further interest in the topic, it also points the way to future work. If one could level any criticism at this book at all, it is that it does not deeply discuss the basic flaws of significance testing or what the alternatives might be. For now, though, significance testing is an integral part of NLP research and this book provides a great resource for researchers who wish to perform it correctly and painlessly.

It is well documented that academics and practitioners focus on statistical significance (typically represented by P tests) and statistical hypothesis testing to determine if their non-statistical analytical hypothesis is correct or likely to be correct. Moreover, statistical significance is relied upon to determine which variables should be used in their models or analyses. In spite of ongoing criticism, this practice continues to the detriment of robust scientific analysis. I discuss the significant limitations of statistical significance in scientific analysis, irrespective of discipline, with some focus on economics. I place statistical significance in a broader analytical context, discussing other analytical procedures that need to be followed and emphasized for one’s analysis to be scientifically robust. This relates the development of models, assumptions underlying the models, the data collected and constructed, the relationship between statistical significance and causality and the importance of non-statistical theory to the identification of pertinent modelling variables. Analytical significance (size effects and variability) is core to any robust scientific analysis, but only in the context of all of the other prior steps in the applied research project being in place. In this broader analytical framework, the statistical significance becomes relatively insignificant. I also address why statistical significance and statistical hypothesis testing dominates the applied analytical landscape even though this dominance is not best practice. Of critical importance are the mental models of best practice and the worldview and preferences of decision makers determining what gets published and who is successful in securing grants and employment.

A recent article in this journal by Tabak (2006) highlighted a potentially serious source of bias that can arise when multiple statistical tests of a damages theory are performed and even one test rejecting the null hypothesis is regarded as supporting the damages theory. Repeated testing will eventually produce a “false discovery,” that is, rejection of the null hypothesis in favor of the alternative hypothesis when the null hypothesis is true, which statisticians refer to as type I error. Consequently, performing multiple tests without adjusting the critical value can be problematical because it can lead to improperly accepting statistical evidence that apparently supports rejection of the null hypothesis as reliable when it is not. Tabak (2006) recommends making the Sidak (1968, 1971) multiple-comparison adjustment to the standard statistical t-test to correct for the false-discovery bias inherent in multiple-comparison testing. In particular, he recommends making this adjustment when performing 10b-5 securities fraud event studies when more than one corrective disclosure date is involved. This article clarifies the circumstances in which a multiple-comparison adjustment is appropriate and explains why the correction is normally not needed in securities fraud event-study testing. More generally, I explain why it is not required when each of several tests is performed and its results are reported separately, as for example, where the objective is simply to test the statistical significance of the abnormal stock return on each day on which a new and distinct curative disclosure occurred. I show that the Sidak multiple-comparison adjustment is nearly as stringent as the classical Bonferroni procedure (Simes, 1986), which can increase the risk of type II error. This article discusses a more powerful alternative to the Sidak adjustment due to Benjamini and Hochberg (1993), which directly corrects for the false-discovery bias in multiple-comparison testing and reduces the risk of type II error. II. Application of Multiple-Comparison Adjustments to Securities Fraud Event Studies Multiple-comparison-false-discovery bias can arise when (a) several statistical tests are performed on subsets of the same larger data set in an effort to

The interpretation and reporting of p values and significance testing in biomedical research are fraught with misconceptions and inaccuracies. Publications of peer-reviewed research in orthopaedics are not immune to such problems. The American Statistical Association (ASA) recently published an official statement on the use, misuse, and misinterpretation of statistical testing and p values in applied research. The ASA statement discussed 6 principles: (1) "P-values can indicate how incompatible the data are with a specified statistical model." (2) "P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone." (3) "Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold." (4) "Proper inference requires full reporting and transparency." (5) "A p-value, or statistical significance, does not measure the size of an effect or the importance of a result." (6) "By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis." The purpose of this article was to discuss these principles. We make several recommendations for moving forward: (1) Authors should avoid statements such as "statistically significant" or "statistically nonsignificant." (2) Investigators should report the magnitude of effect of all outcomes together with the appropriate measure of precision or variation. (3) Orthopaedic residents and surgeons must be educated in biostatistics, the ASA principles, and clinical epidemiology. (4) Journal editors and reviewers need to be familiar with and enforce the ASA principles.

Are there benefits from NHST?

Despite more than half a century of criticism, significance testing continues to be used commonly by ecologists. Significance tests are widely misused and misunderstood, and even when properly used, they are not very informative for most ecological data. Problems of misuse and misinterpretation include: (i) invalid logic; (ii) rote use; (iii) equating statistical significance with biological importance; (iv) regarding the P‐value as the probability that the null hypothesis is true; (v) regarding the P‐value as a measure of effect size; and (vi) regarding the P‐value as a measure of evidence. Significance tests are poorly suited for inference because they pose the wrong question. In addition, most null hypotheses in ecology are point hypotheses already known to be false, so whether they are rejected or not provides little additional understanding. Ecological data rarely fit the controlled experimental setting for which significance tests were developed. More satisfactory methods of inference assess the degree of support which data provide for hypotheses, measured in terms of information theory (model‐based inference), likelihood ratios (likelihood inference) or probability (Bayesian inference). Modern statistical methods allow multiple data sets to be combined into a single likelihood framework, avoiding the loss of information that can occur when data are analyzed in separate steps. Inference based on significance testing is compared with model‐based, likelihood and Bayesian inference using data on an endangered porpoise,Phocoena sinus. All of the alternatives lead to greater understanding and improved inference than provided by a P‐value and the associated statement of statistical significance.