Précis of statistical significance: Rationale, validity, and utility

Behavioral and Brain Sciences 21 (2):169-194 (1998)

Abstract

The null-hypothesis significance-test procedure (NHSTP) is defended in the context of the theory-corroboration experiment, as well as the following contrasts: (a) substantive hypotheses versus statistical hypotheses, (b) theory corroboration versus statistical hypothesis testing, (c) theoretical inference versus statistical decision, (d) experiments versus nonexperimental studies, and (e) theory corroboration versus treatment assessment. The null hypothesis can be true because it is the hypothesis that errors are randomly distributed in data. Moreover, the null hypothesis is never used as a categorical proposition. Statistical significance means only that chance influences can be excluded as an explanation of data; it does not identify the nonchance factor responsible. The experimental conclusion is drawn with the inductive principle underlying the experimental design. A chain of deductive arguments gives rise to the theoretical conclusion via the experimental conclusion. The anomalous relationship between statistical significance and the effect size often used to criticize NHSTP is more apparent than real. The absolute size of the effect is not an index of evidential support for the substantive hypothesis. Nor is the effect size, by itself, informative as to the practical importance of the research result. Being a conditional probability, statistical power cannot be the apriori probability of statistical significance. The validity of statistical power is debatable because statistical significance is determined with a single sampling distribution of the test statistic based on H0, whereas it takes two distributions to represent statistical power or effect size. Sample size should not be determined in the mechanical manner envisaged in power analysis. It is inappropriate to criticize NHSTP for nonstatistical reasons. At the same time, neither effect size, nor confidence interval estimate, nor posterior probability can be used to exclude chance as an explanation of data. Neither can any of them fulfill the nonstatistical functions expected of them by critics

Download options

PhilArchive



    Upload a copy of this work     Papers currently archived: 72,766

External links

Setup an account with your affiliations in order to access resources via your University's proxy server

Through your library

Analytics

Added to PP
2009-01-28

Downloads
44 (#261,436)

6 months
1 (#386,499)

Historical graph of downloads
How can I increase my downloads?

References found in this work

No references found.

Add more references

Similar books and articles

Science, Ecological Validity and Experimentation.Siu L. Chow - 1987 - Journal for the Theory of Social Behaviour 17 (2):181–194.
Statistical Significance: A Statistician's View.Helena Chmura Kraemer - 1998 - Behavioral and Brain Sciences 21 (2):206-207.
Significance Tests and Deduction: Reply to Folger (1989).Siu L. Chow - 1989 - Psychological Bulletin 106 (1):161-165.
The Logic of Tests of Significance.Stephen Spielman - 1974 - Philosophy of Science 41 (3):211-226.
Significance Testing – Does It Need This Defence?Günther Palm - 1998 - Behavioral and Brain Sciences 21 (2):214-215.
The Null-Hypothesis Significance-Test Procedure is Still Warranted.Siu L. Chow - 1998 - Behavioral and Brain Sciences 21 (2):228-235.
Costs and Benefits of Statistical Significance Tests.Michael G. Shafto - 1998 - Behavioral and Brain Sciences 21 (2):218-219.