The origins of testing scientific models with statistical techniques go back to 18th century mathematics. However, the modern theory of statistical testing was primarily developed through the work of Sir R.A. Fisher, Jerzy Neyman, and Egon Pearson in the inter-war period. Some of Fisher's papers on testing were published in economics journals (Fisher, 1923, 1935) and exerted a notable influence on the discipline. The development of econometrics and the rise of quantitative economic models in the mid-20th century made (...) statistical significance testing a commonplace, albeit controversial tool within economics. -/- In the debate about significance testing, methodological controversies intertwine with epistemological issues and sociological developments. Our aim in this chapter is to expound these connections and to show how the use of, and the debate about, significance testing in economics differs from other social sciences, such as psychology. (shrink)

While the common procedure of statistical significance testing and its accompanying concept of p-values have long been surrounded by controversy, renewed concern has been triggered by the replication crisis in science. Many blame statistical significance tests themselves, and some regard them as sufficiently damaging to scientific practice as to warrant being abandoned. We take a contrary position, arguing that the central criticisms arise from misunderstanding and misusing the statistical tools, and that in fact the purported (...) remedies themselves risk damaging science. We argue that banning the use of p-value thresholds in interpreting data does not diminish but rather exacerbates data-dredging and biasing selection effects. If an account cannot specify outcomes that will not be allowed to count as evidence for a claim—if all thresholds are abandoned—then there is no test of that claim. The contributions of this paper are: To explain the rival statistical philosophies underlying the ongoing controversy; To elucidate and reinterpret statistical significance tests, and explain how this reinterpretation ameliorates common misuses and misinterpretations; To argue why recent recommendations to replace, abandon, or retire statistical significance undermine a central function of statistics in science: to test whether observed patterns in the data are genuine or due to background variability. (shrink)

Chow sets his version of statistical significance testing in an impoverished context of “theory corroboration” that explicitly excludes well-posed theories admitting of strong support by precise empirical evidence. He demonstrates no scientific usefulness for the problematic procedure he recommends instead. The important role played by significance testing in today's behavioral and brain sciences is wholly inconsistent with the rhetoric he would enforce.

Birch’s formulation is persuasive but not nuanced enough to capture at least one situation where it is reasonable to invoke the precautionary principle (PP): when we have multiple, weak, but convergent, lines of evidence that a species is sentient, but no statistically significant evidence of a single credible indicator of sentience within the order as required by BAR. I respond to the worry that if we include such cases in our framework for applying the PP, we open ourselves to the (...) charge of being “unscientific.”. (shrink)

Whether in meta-analysis or single experiments, selecting results based on statistical significance leads to overestimated effect sizes, impeding falsification. We critique a quantitative synthesis that used significance to score and select previously published effects for situation awareness-performance associations. How much does selection using statistical significance quantitatively impact results in a meta-analytic context? We evaluate and compare results using significance-filtered effects versus analyses with all effects as-reported. Endsley reported high predictiveness scores and large positive mean (...) correlations but used atypical methods: the hypothesis was used to select papers and effects. Papers were assigned the maximum predictiveness scores if they contained at-least-one significant effect, yet most papers reported multiple effects, and the number of non-significant effects did not impact the score. Thus, the predictiveness score was rarely less than the maximum. In addition, only significant effects were included in Endsley’s quantitative synthesis. Filtering excluded half of all reported effects, with guaranteed minimum effect sizes based on sample size. Results for filtered compared to as-reported effects clearly diverged. Compared to the mean of as-reported effects, the filtered mean was overestimated by 56%. Furthermore, 92% of the as-reported effects were below the mean of filtered effects. We conclude that outcome-dependent selection of effects is circular, predetermining results and running contrary to the purpose of meta-analysis. Instead of using significance to score and filter effects, meta-analyses should follow established research practices. (shrink)

From a statistician's viewpoint, the concepts discussed by Chow relating to “statistical” significance bear little resemblance to the concept developed in statistics. Whether or not “statistical significance” has a place in psychological research is a decision for psychologists, not statisticians, to make, but the decision should be based on a less flawed version of what is being considered.

The null-hypothesis significance-test procedure (NHSTP) is defended in the context of the theory-corroboration experiment, as well as the following contrasts: (a) substantive hypotheses versus statistical hypotheses, (b) theory corroboration versus statistical hypothesis testing, (c) theoretical inference versus statistical decision, (d) experiments versus nonexperimental studies, and (e) theory corroboration versus treatment assessment. The null hypothesis can be true because it is the hypothesis that errors are randomly distributed in data. Moreover, the null hypothesis is never used as (...) a categorical proposition. Statistical significance means only that chance influences can be excluded as an explanation of data; it does not identify the nonchance factor responsible. The experimental conclusion is drawn with the inductive principle underlying the experimental design. A chain of deductive arguments gives rise to the theoretical conclusion via the experimental conclusion. The anomalous relationship between statistical significance and the effect size often used to criticize NHSTP is more apparent than real. The absolute size of the effect is not an index of evidential support for the substantive hypothesis. Nor is the effect size, by itself, informative as to the practical importance of the research result. Being a conditional probability, statistical power cannot be the apriori probability of statistical significance. The validity of statistical power is debatable because statistical significance is determined with a single sampling distribution of the test statistic based on H0, whereas it takes two distributions to represent statistical power or effect size. Sample size should not be determined in the mechanical manner envisaged in power analysis. It is inappropriate to criticize NHSTP for nonstatistical reasons. At the same time, neither effect size, nor confidence interval estimate, nor posterior probability can be used to exclude chance as an explanation of data. Neither can any of them fulfill the nonstatistical functions expected of them by critics. (shrink)

Chow's endorsement of a limited role for null hypothesis significance testing is a needed corrective of research malpractice, but his decision to place this procedure in a hypothetico-deductive framework of Popperian cast is unwise. Various failures of this version of the hypothetico-deductive method have negative implications for Chow's treatment of significance testing, meta-analysis, and theory evaluation.

Although various statistical measures may have other valid uses, the single purpose served by statistical significance testing in the epistemology of experimental science is as a peremptory rebuttal of one potential alternative interpretation of the data.

The soundness of Chow's (1996; 1998a) argument depends on the soundness of his assertion that statistical significance may be understood to signify that chance may be excluded as the reason for results. The examples and arguments provided here show that statistical significance signifies no such thing.

Chow's book provides a thorough analysis of the confusing array of issues surrounding conventional tests of statistical significance. This book should be required reading for behavioral and social scientists. Chow concludes that the null-hypothesis significance-testing procedure (NHSTP) plays a limited, but necessary, role in the experimental sciences. Another possibility is that – owing in part to its metaphorical underpinnings and convoluted logic – the NHSTP is declining in importance in those few sciences in which it ever played (...) a role. (shrink)

While correctly describing the differences in current practices between experimental psychologists and economists, Hertwig and Ortmann do not provide a compelling explanation for these differences. Our explanation focuses on the fact that psychologists view the world as composed of categories and types. This discrete organizational scheme results in merely testing nulls and wider variation in observed practices in experimental psychology.

That Haig and Sohn find the hypothetico-deductive approach wanting in different ways shows that multiple conditional syllogisms are being used in different stages of theory corroboration in the Popperian approach. The issues raised in the two commentaries assume a different complexion when certain distinctions are made.

Chow's book makes a provocative contribution to the debate on the role of statistical significance, but it involves some important misconceptions in the presentation of the Fisher and Neyman/Pearson's theories. Moreover, the author's caricature-like considerations about “Bayesianism” are completely irrelevant for discarding the Bayesian statistical theory. These facts call into question the objectivity of his contribution.

Chow pays lip service (but not much more!) to Type I errors and thus opts for a hard (all-or-none) .05 level of significance (Superego of Neyman/Pearson theory; Gigerenzer 1993). Most working scientists disregard Type I errors and thus utilize a soft .05 level (Ego of Fisher; Gigerenzer 1993), which lets them report gradations of significance (e.g., p.

Statistical significance is almost universally equated with the attribution to some population of nonchance influences as the source of structure in the data. But statistical significance can be divorced from both parameter estimation and probability as, instead, a statement about the atypicality or lack of exchangeability over some distinction of the data relative to some set. From this perspective, the criticisms of significance tests evaporate.

In a recent note Roger Carlson presented a rather negative appraisal of my treatment of the logic of Fisherian significance testing in [10]. The main issue between us involves Carlson's thesis that, within the limits set by Fisher, standard significance tests are valuable tools of data analysis as they stand, i.e., without modification of the structure of the reasoning they employ. Call this the adequacy thesis. In my paper I argued that the pattern of reasoning employed by tests (...) of significance needs to be justified in spite of its unquestioned acceptance by most researchers, The best justification offered to date—by R. A. Fisher—is seriously defective. Therefore the adequacy thesis is not justified. I proposed some alterations of the pattern of reasoning employed by tests of significance and argued that they guarantee that tests are adequate epistemological tools. (shrink)

Los estudios de ciencia, tecnología y sociedad revelan las interrelaciones entre la ciencia y la tecnología como procesos sociales. Este artículo persigue como objetivo: valorar la significación social de un entorno virtual en la enseñanza aprendizaje de la Estadística descriptiva en la carrera de Medicina. El diagnóstico preliminar mediante de la observación, la encuesta y el análisis documental, mostró que existen insuficiencias en el uso de las tecnologías de la información y las comunicaciones en el proceso de enseñanza aprendizaje de (...) la Estadística. Se diseñó un entorno virtual constituido por cinco módulos que incluyen contenidos de Estadística descriptiva, ejercicios, bibliografía y ayuda. Con la validación a partir del criterio de especialistas y la sistematización con la introducción en la práctica, se evaluó el mismo como muy adecuado, fue considerado útil, necesario y aplicable al contexto objeto de estudio, lo que contribuye a la formación científica de los estudiantes desde una posición materialista. Science, technology and society studies reveal the interrelations between science and technology as social processes. This article pursues the following objectives: to assess the social significance of a virtual environment in the teaching and learning of descriptive Statistics in Medicine degree course. The preliminary diagnosis through the observation, opinion poll and documentary analysis, showed that there are insufficiencies in the use of Information and Communication technologies in teaching and learning Statistics. It was designed a virtual environment with five modules that include contents of descriptive Statistics, exercises, bibliography and help. With the validation of specialists and the systematization after putting it in practice, the virtual environment was assessed as adequate, useful, necessary and applicable to the context being studied, which contributes to the scientific training of students from a materialistic position. (shrink)

To estimate causal relationships, time series econometricians must be aware of spurious correlation, a problem first mentioned by Yule (1926). To deal with this problem, one can work either with differenced series or multivariate models: VAR (VEC or VECM) models. These models usually include at least one cointegration relation. Although the Bayesian literature on VAR/VEC is quite advanced, Bauwens et al. (1999) highlighted that “the topic of selecting the cointegrating rank has not yet given very useful and convincing results”. The (...) present article applies the Full Bayesian Significance Test (FBST), especially designed to deal with sharp hypotheses, to cointegration rank selection tests in VECM time series models. It shows the FBST implementation using both simulated and available (in the literature) data sets. As illustration, standard non informative priors are used. (shrink)

The computations involved in statistical learning have long been debated. Here, we build on work suggesting that a basic memory process, chunking, may account for the processing of statistical regularities into larger units. Drawing on methods from the memory literature, we developed a novel paradigm to test statistical learning by leveraging a robust phenomenon observed in serial recall tasks: that short‐term memory is fundamentally shaped by long‐term distributional learning. In the statistically induced chunking recall (SICR) task, participants (...) are exposed to an artificial language, using a standard statistical learning exposure phase. Afterward, they recall strings of syllables that either follow the statistics of the artificial language or comprise the same syllables presented in a random order. We hypothesized that if individuals had chunked the artificial language into word‐like units, then the statistically structured items would be more accurately recalled relative to the random controls. Our results demonstrate that SICR effectively captures learning in both the auditory and visual modalities, with participants displaying significantly improved recall of the statistically structured items, and even recall specific trigram chunks from the input. SICR also exhibits greater test–retest reliability in the auditory modality and sensitivity to individual differences in both modalities than the standard two‐alternative forced‐choice task. These results thereby provide key empirical support to the chunking account of statistical learning and contribute a valuable new tool to the literature. (shrink)

There is little empirical evidence showing a direct link between a capacity for statistical learning (SL) and proficiency with natural language. Moreover, discussion of the role of SL in language acquisition has seldom focused on literacy development. Our study addressed these issues by investigating the relationship between SL and reading ability in typically developing children and healthy adults. We tested SL using visually presented stimuli within a triplet learning paradigm and examined reading ability by administering the Wide Range Achievement (...) Test (WRAT-4; Wilkinson & Robertson, 2006). A total of 38 typically developing children (mean age of 9;5 years, range 6;4–12;5) and 37 healthy adults (mean age of 21 years, range 18–34) were assessed. In children, SL was significantly related to reading ability. Importantly, this relationship was independent of grade and also age. The adult data, too, revealed that SL was significantly related to reading ability. A regression analysis of the combined child and adult data revealed that SL accounted for a unique amount of variance in reading ability, after age and attention had been taken into consideration. For the first time, this study provides empirical evidence that a capacity for more effective SL is related to higher reading ability in the general population. (shrink)

The advance of science and human knowledge is impeded by misunderstandings of various statistics, insufficient reporting of findings, and the use of numerous standardized and non-standardized presentations of essentially identical information. Communication with journalists and the public is hindered by the failure to present statistics that are easy for non-scientists to interpret as well as by use of the word significant, which in scientific English does not carry the meaning of "important" or "large." This article promotes a new standard method (...) for reporting two-group and two-variable statistics that can enhance the presentation of relevant information, increase understanding of findings, and replace the current presentations of two-group ANOVA, t-tests, correlations, chi-squares, and z-tests of proportions. A brief call to highly restrict the publication of risk ratios, odds ratios, and relative increase in risk percentages is also made, since these statistics appear to provide no useful scientific information regarding the magnitude of findings. (shrink)

It is one thing to hold that merely statistical evidence is _sometimes_ insufficient for rational belief, as in typical lottery and profiling cases. It is another thing to hold that merely statistical evidence is _always_ insufficient for rational belief. Indeed, there are cases where statistical evidence plainly does justify belief. This project develops a dispositional account of the normativity of statistical evidence, where the dispositions that ground justifying statistical evidence are connected to the goals (= (...) proper function) of objects. There are strong intuitive motivations for doing this. For we can turn almost any case of _non-justifying_ merely statistical evidence into a case of _justifying_ merely statistical evidence by adding information about the dispositions and goals of the objects involved. The resulting view not only helps us understand when and why merely statistical evidence is normatively significant, but it also helps us understand how statistical evidence relates to more standard forms of evidence (perceptual, testimonial). The emerging view also has surprising applications, as it imposes limitations on the epistemic value of fine-tuning arguments for theism as well as undermines a standard class of case-based arguments for moral encroachment. (shrink)

In this paper I propose an interpretation of classical statistical mechanics that centers on taking seriously the idea that probability measures represent complete states of statistical mechanical systems. I show how this leads naturally to the idea that the stochasticity of statistical mechanics is associated directly with the observables of the theory rather than with the microstates (as traditional accounts would have it). The usual assumption that microstates are representationally significant in the theory is therefore dispensable, a (...) consequence which suggests interesting possibilities for developing non-equilibrium statistical mechanics and investigating inter-theoretic answers to the foundational questions of statistical mechanics. (shrink)

After some general remarks about the interrelation between philosophical and statistical thinking, the discussion centres largely on significance tests. These are defined as the calculation of p-values rather than as formal procedures for ‘acceptance‘ and ‘rejection‘. A number of types of null hypothesis are described and a principle for evidential interpretation set out governing the implications of p- values in the specific circumstances of each application, as contrasted with a long-run interpretation. A number of more complicated situ- ations (...) are discussed in which modification of the simple p-value may be essential. (shrink)

This study investigated visual statistical learning (VSL) in 24 deaf signers and 24 hearing non‐signers. Previous research with hearing individuals suggests that SL mechanisms support literacy. Our first goal was to assess whether VSL was associated with reading ability in deaf individuals, and whether this relation was sustained by a link between VSL and sign language skill. Our second goal was to test the Auditory Scaffolding Hypothesis, which makes the prediction that deaf people should be impaired in sequential processing (...) tasks. For the VSL task, we adopted a modified version of the triplet learning paradigm, with stimuli presented sequentially across space and time. Results revealed that measures of sign language skill (sentence comprehension/repetition) did not correlate with VSL scores, possibly due to the sequential nature of our VSL task. Reading comprehension scores (PIAT‐R) were a significant predictor of VSL accuracy in hearing but not deaf people. This finding might be due to the sequential nature of the VSL task and to a less salient role of the sequential orthography‐to‐phonology mapping in deaf readers compared to hearing readers. The two groups did not differ in VSL scores. However, when reading ability was taken into account, VSL scores were higher for the deaf group than the hearing group. Overall, this evidence is inconsistent with the Auditory Scaffolding Hypothesis, suggesting that humans can develop efficient sequencing abilities even in the absence of sound. (shrink)

Statistical evidence is pervasive in medicine. In this chapter we will focus on the reliability of randomized clinical trials (RCTs) conducted to test the safety and efficacy of medical treatments. RCTs are scientific experiments and, as such, we expect them to be replicable: if we repeat the same experiment time and again, we should obtain the same outcome (Norton 2015). The statistical design of the test should guarantee that the observed outcome is not a random event, but rather (...) a real effect of the treatments administered. However, for more than a decade now, we have been discussing a replicability crisis across different experimental disciplines including medicine: the outcomes of trials published in very prestigious journals often disappear when the experiment is repeated (see, e.g., Lehrer 2010, Begley and Ellis 2012, Horton 2015). -/- There are different accounts of the reason for this replicability crisis, ranging from scientific fraud to lack of institutional incentives to double-check someone else’s results. In this chapter we will use the replicability crisis as a thread to introduce some central issues in the design of scientific experiments in medicine. First, in section 1, we will see how replicability and statistical significance are connected: we can only make sense of the p-value of a trial outcome within a series of replications of the test. But in order to conduct these replications properly, we need to agree on the proper design of the experiment we are going to repeat. In particular, we need to prevent the preferences of the experimenters from biasing the outcome of the experiment. If there is such a bias, when the experiment is replicated by a third party, the observed outcome will vanish. In section 2, we will argue that trialists need to agree on the debiasing procedures and the statistical quality controls that feature in the trial protocol, if they want the outcome to be replicable. In section 3, we will make two complementary points. On the one hand, replicability per se is not everything: we need trial outcomes that are not only statistically significant but also clinically relevant. On the other hand, trials are not everything: the experts analyzing the evidence can improve the reliability of statistical evidence, although they sometimes fail; we need to further study how they make their decisions. In section 4, we will use a controversy about the overprescription of statins to show how non-replicable effects are obtained in trials and how experts may fail at detecting such flaws, if the commercial interests are big enough. (shrink)

Statistical evidence is crucial throughout disparate impact’s three-stage analysis: during (1) the plaintiff’s prima facie demonstration of a policy’s disparate impact; (2) the defendant’s job-related business necessity defense of the discriminatory policy; and (3) the plaintiff’s demonstration of an alternative policy without the same discriminatory impact. The circuit courts are split on a vital question about the “practical significance” of statistics at Stage 1: Are “small” impacts legally insignificant? For example, is an employment policy that causes a one (...) percent disparate impact an appropriate policy for redress through disparate impact litigation? This circuit split calls for a comprehensive analysis of practical significance testing across disparate impact’s stages. Importantly, courts and commentators use “practical significance” ambiguously between two aspects of practical significance: the magnitude of an effect and confidence in statistical evidence. For example, at Stage 1 courts might ask whether statistical evidence supports a disparate impact (a confidence inquiry) and whether such an impact is large enough to be legally relevant (a magnitude inquiry). Disparate impact’s texts, purposes, and controlling interpretations are consistent with confidence inquires at all three stages, but not magnitude inquiries. Specifically, magnitude inquiries are inappropriate at Stages 1 and 3—there is no discriminatory impact or reduction too small or subtle for the purposes of the disparate impact analysis. Magnitude inquiries are appropriate at Stage 2, when an employer defends a discriminatory policy on the basis of its job-related business necessity. (shrink)

The application of statistical methods and models both in the natural and social sciences is nowadays a trivial fact which nobody would deny. Bold analogies even suggest the application of the same statistical models to fields as different as statistical mechanics and economics, among them the case of the young and controversial discipline of Econophysics . Less trivial, however, is the answer to the philosophical question, which has been raised ever since the possibility of “commuting” statistical (...) thinking and models between natural and social sciences emerged: whether such a methodological kinship would imply some kind of more profound unity of the natural and the social domain. Starting with Adolphe Quetelet (1796-1874) and ending with the Vienna Circle (from the late 1920s until the 1940s), this paper offers a brief historical and philosophical reconstruction of some important stages in the development of statistics as “commuting” between the natural and the social sciences. This reconstruction is meant to highlight (with respect to the authors under consideration): (1) the existence of a significant correlation between the readiness to “transfer” statistical thinking from natural to social sciences and vice versa, on the one hand, and the standpoints on the issue of the unity/disunity of science, on the other; (2) the historical roots and the fortunes of the analogy between statistical models of society and statistical models of gases. (shrink)

In this paper epistemological, ontological and sociological questions concerning the statistical significance of sharp hypotheses in scientific research are investigated within the framework provided by Cognitive Constructivism and the FBST (Full Bayesian Significance Test). The constructivist framework is contrasted with the traditional epistemological settings for orthodox Bayesian and frequentist statistics provided by Decision Theory and Falsificationism.

Some philosphers of science of an empiricist and pragmatist bent have proposed models of statistical explanation, but have then become sceptical of the adequacy of these models. It is argued that general considerations concerning the purpose of function of explanation in science which are usually appealed to by such philosophers show that their scepticism is not well taken; for such considerations provide much the same rationale for the search for statistical explanations, as these philosophers have characterized them, as (...) they do for lawlike explanations. But, it is further argued, a significant piece of what is frequently offered as an explanation of well known phenomena in statistical mechanics, fails to meet this general "pragmatic rationale" for statistical, or indeed any kind of, explanation. The question then arises whether the physicists have misconstrued the value of this piece of physical theorizing, ergodic theory, taking it to be explanatory when it is actually not; or whether, instead, the philosopher's account of just what is genuinely explanatory is too narrow. (shrink)

Statistical mechanics is the name of the ongoing attempt to explain and predict certain phenomena, above all those described by thermodynamics on the basis of the fundamental theories of physics, in particular mechanics, together with certain auxiliary assumptions. In another paper in this journal, Foundations of statistical mechanics: Mechanics by itself, I have shown that some of the thermodynamic regularities, including the probabilistic ones, can be described in terms of mechanics by itself. But in order to prove those (...) regularities, in particular the time asymmetric ones, it is necessary to add to mechanics assumptions of three kinds, all of which are under debate in contemporary literature. One kind of assumptions concerns measure and probability, and here, a major debate is around the notion of “typicality.” A second assumption concerns initial conditions, and here, the debate is about the nature and status of the so-called past hypothesis. The third kind of assumptions concerns the dynamics, and here, the possibility and significance of “Maxwell's Demon” is the main topic of discussions. This article describes these assumptions and examines the justification for introducing them, emphasizing the contemporary debates around them. (shrink)

Statistics in public life .................................................................................................... .....5 Things and numbers............................................................................................. ...................8 Representative samples............................................................................................. ..........8 Averages: meaning and relevance .....................................................................................9 Correlations........................................................................................ ................................10 Applied statistics .................................................................................................... ................13 Relative risks .................................................................................................... ..................14 Relative risk versus absolute risk.....................................................................................16 Problems of classification and confounding factors....................................................17 Epidemiological research............................................................................................ ..........19 Publication bias................................................................................................ ..................20 Statistical significance versus scientific relevance................................................................24 Relative risk again............................................................................................... ...............24 P-values............................................................................................ ...................................25 Confidence intervals .................................................................................................... .....26 Correlation is not causation .............................................................................................26 An infamous episode .................................................................................................... ....27 Terror, utopianism and power .............................................................................................29 Faith and science .................................................................................................... ...........29 Fear and power: the precautionary principle.................................................................30 Utopian salvation........................................................................................... ....................32....

No one has a well developed solution to Duhem's problem, the problem of how experimental evidence warrants revision of our theories. Deborah Mayo proposes a solution to Duhem's problem in route to her more ambitious program of providing a philosophical account of inductive inference and experimental knowledge. This paper is a response to Mayo's Error Statistics (ES) program, paying particular attention to her response to Duhem's problem. It turns out that Mayo's purported solution to Duhem's problem is very significant to (...) her project, for the epistemic license claimed by ES and the philosophical underpinnings to her account of experimental knowledge depend on this solution. By introducing the partition problem, I argue that ES fails to solve Duhem's problem and therefore fails to provide an adequate account of experimental knowledge. (shrink)