Unfortunately, reading Chow's work is likely to leave the reader more confused than enlightened. My preferred solutions to the “controversy” about null- hypothesis testing are: (1) recognize that we really want to test the hypothesis that an effect is “small,” not null, and (2) use Bayesian methods, which are much more in keeping with the way humans naturally think than are classical statistical methods.

Chow's defense of NHSTP ignores the fact that in psychology it is used to test substantive hypotheses in theory-corroborating research. In this role, NHSTP is not only inadequate, but damaging to the progress of psychology as a science. NHSTP does not fulfill the Popperian requirement that theories be tested severely. It also encourages nonspecific predictions and feeble theoretical formulations.

This commentary advocates an alternative to null-hypothesis testing that was originally represented by Rozeboom over three decades ago yet is not considered by Chow (1996). The central distinguishing feature of this approach is that it allows the scientist to conclude that the data are much better fit by those hypotheses whose values fall inside the interval than by those outside.

I disagree with several of Chow's traditional descriptions and justifications of null hypothesis testing: (1) accepting the null hypothesis whenever p > .05; (2) random sampling from a population; (3) the frequentist interpretation of probability; (4) having the null hypothesis generate both a probability distribution and a complement of the desired conclusion; (5) assuming that researchers must fix their sample size before performing their study.

In this commentary, I agree with Chow's treatment of null hypothesis significance testing as a noninferential procedure. However, I dispute his reconstruction of the logic of theory corroboration. I also challenge recent criticisms of NHSTP based on power analysis and meta-analysis.

I agree with Gibbs that the message of the base rate literature reads differently depending on which null hypothesis is used to frame the issue. But I argue that the normative null hypothesis, H0: “People use base rates in a Bayesian manner,” is no longer appropriate. I also challenge Adler's distinction between unused and ignored base rates, and criticize Goodie's reluctance to shift research attention to the field. Macchi's arguments about textual ambiguities in traditional base rate (...) problems suggest that empirical testing is needed to tease apart the effects of problem clarification and problem framing. Macdonald's, Fletcher's and Snow's skepticism about the value of Bayesian methods in real world judgment tasks is treated as a challenge for the next generation of empirical base rate studies. (shrink)

Entertaining diverse assumptions about empirical research, commentators give a wide range of verdicts on the NHSTP defence in Statistical significance. The null-hypothesis significance- test procedure is defended in a framework in which deductive and inductive rules are deployed in theory corroboration in the spirit of Popper's Conjectures and refutations. The defensible hypothetico-deductive structure of the framework is used to make explicit the distinctions between substantive and statistical hypotheses, statistical alternative and conceptual alternative hypotheses, and making statistical decisions and (...) drawing theoretical conclusions. These distinctions make it easier to show that H0 can be true, the effect size is irrelevant to theory corroboration, and “strong” hypotheses make no difference to NHSTP. Reservations about statistical power, meta-analysis, and the Bayesian approach are still warranted. (shrink)

If the NHSTP procedure is essential for controlling for chance, why is there little, if any, discussion of the nature of chance by Chow and other advocates of the procedure. Also, many criticisms that Chow takes to be aimed against the NHSTP procedure are actually directed against the kind of theory that is tested by the procedure.

Several of Krueger & Funder's (K&F's) suggestions may promote more balanced social cognition research, but reconsidered null hypothesis statistical testing (NHST) is not one of them. Although NHST has primarily supported negative conclusions, this is simply because most conclusions have been negative. NHST can support positive, negative, and even balanced conclusions. Better NHST practices would benefit psychology, but would not alter the balance between positive and negative approaches.

The interpretation of tests of a point null hypothesis against an unspecified alternative is a classical and yet unresolved issue in statistical methodology. This paper approaches the problem from the perspective of Lindley's Paradox: the divergence of Bayesian and frequentist inference in hypothesis tests with large sample size. I contend that the standard approaches in both frameworks fail to resolve the paradox. As an alternative, I suggest the Bayesian Reference Criterion: it targets the predictive performance of the (...) null hypothesis in future experiments; it provides a proper decision-theoretic model for testing a point null hypothesis and it convincingly accounts for Lindley's Paradox. (shrink)

The first of 3 objectives in this study was to address the major problem with Null Hypothesis Significance Testing (NHST) and 2 common misconceptions related to NHST that cause confusion for students and researchers. The misconcep- tions are (a) a smaller p indicates a stronger relationship and (b) statistical signifi- cance indicates practical importance. The second objective was to determine how this problem and the misconceptions were treated in 12 recent textbooks used in edu- cation research methods (...) and statistics classes. The third objective was to examine how the textbooks’ presentations relate to current best practices and how much help they provide for students. The results show that almost all of the textbooks fail to acknowledge that there is controversy surrounding NHST. Most of the textbooks dealt, at least minimally, with the alleged misconceptions of interest, but they pro- vided relatively little help for students. (shrink)

We examine the claim that the methodology of psychology leads to a bias in animal cognition research against attributing “anthropomorphic” properties to animals . This charge is examined in light of a debate on the role of folk psychology between primatologists who emphasize similarities between humans and other apes, and those who emphasize differences. We argue that while in practice there is sometimes bias, either in the formulation of the null hypothesis or in the preference of Type-II errors (...) over Type-I errors, the bias is not the result of proper use of the Neyman and Pearson hypothesis testing method. Psychologists’ preference for false negatives over false positives cannot justify a preference for avoiding anthropomorphic errors over anthropectic errors. (shrink)

We argue that Chow's defense of hypothesis-testing procedures attempts to restore an aura of objectivity to the core procedures, allowing these to take on the role of judgment that should be reserved for the researcher. We provide a brief overview of what we call the historical case against hypothesis testing and argue that the latter has led to a constrained and simplified conception of what passes for theory in psychology.

In this article I give a critical evaluation of the use and limitations of null-model-based hypothesis testing as a research strategy in the biological sciences. According to this strategy, the null model based on a randomization procedure provides an appropriate null hypothesis stating that the existence of a pattern is the result of random processes or can be expected by chance alone, and proponents of other hypotheses should first try to reject this null (...) hypothesis in order to demonstrate their own hypotheses. Using as an example the controversy over the use of null hypotheses and null models in species co-occurrence studies, I argue that null-model-based hypothesis testing fails to work as a proper analog to traditional statistical null-hypothesis testing as used in well-controlled experimental research, and that the random process hypothesis should not be privileged as a null hypothesis. Instead, the possible use of the null model resides in its role of providing a way to challenge scientists’ commonsense judgments about how a seemingly unusual pattern could have come to be. Despite this possible use, null-model-based hypothesis testing still carries certain limitations, and it should not be regarded as an obligation for biologists who are interested in explaining patterns in nature to first conduct such a test before pursuing their own hypotheses. (shrink)

Conventional discussion of research methodology contrast two approaches, the quantitative and the qualitative, presented as collectively exhaustive. But if qualitative is taken as the understanding of lifeworlds, the two approaches between them cover only a tiny fraction of research methodologies; and the quantitative, taken as the routine application to controlled experiments of frequentist statistics by way of the Null Hypothesis Significance Testing Procedure, is seriously flawed. It is contrary to the advice both of Fisher and of Neyman (...) and Pearson, the two sources from which it is drawn. (shrink)

Conventional discussion of research methodology contrast two approaches, the quantitative and the qualitative, presented as collectively exhaustive. But if qualitative is taken as the understanding of lifeworlds, the two approaches between them cover only a tiny fraction of research methodologies; and the quantitative, taken as the routine application to controlled experiments of frequentist statistics by way of the Null Hypothesis Significance Testing Procedure, is seriously flawed. It is contrary to the advice both of Fisher and of Neyman (...) and Pearson, the two sources from which it is drawn. (shrink)

Chow's defense of the null-hypothesis significance- test procedure is thoughtful and compelling in many respects. Nevertheless, techniques such as meta-analysis, power analysis, effect size estimation, and confidence intervals can be useful supplements to NHSTP in furthering the cumulative nature of behavioral research, as illustrated by the history of research on the spontaneous recovery of verbal learning.

(2011). Korn and Freidlin's Misunderstanding of the Null Hypothesis Significance Testing Procedure. The American Journal of Bioethics: Vol. 11, No. 3, pp. 15-16.

Reexamining William Rozeboom’s recommendations for the future direction of disciplines such as psychology and philosophy is imminent due to the pressing issues in null hypothesis significance testing. An overreliance on NHST forms the basis of the replication crisis in psychology. Likewise, the discipline’s stringent guidelines on significance levels convey a pressure to publish, which is also significantly contributing to the replication crisis. As researchers’ careers are staked on the extent to which they publish, reassessing the fundamental issues (...) with NHST within the context of Rozeboom’s ideas is paramount. In this brief review, I focus specifically on Rozeboom’s critiques of NHST and on a proposed alternative for evaluating the depth of findings within the framework of theoretical psychology. Thereafter, I compare his foreboding concerns with modern issues facing the validity of psychology and posit Rozeboom prophesized the failings of the discipline years ahead of his time. Keywords: Theoretical Psychology; NHST; William Rozeboom; P-Hacking; Explanatory Induction. (shrink)

The marker approach is taken as best practice for answering the distribution question: Which animals are conscious? However, the methodology can be used to increase confidence in animals many presume to be unconscious, including C. elegans, leading to a trilemma: accept the worms as conscious; reject the specific markers; or reject the marker methodology for answering the distribution question. I defend the third option and argue that answering the distribution question requires a secure theory of consciousness. Accepting the hypothesis (...) all animals are conscious will promote research leading to secure theory, which is needed to create reliable consciousness tests for animals and AIs. Rather than asking the distribution question, we should shift to the dimensions question: How are animals conscious? (shrink)

Professor Meehl [2] has pointed out a very significant problem in the methodology of psychological research, indicating that statistical tests of psychological hypotheses against a null hypothesis are loaded in favor of eventual success at rejecting the null hypothesis. In my opinion this is not, however, a contrast between physics and psychology, but rather between the method of parameter estimation and that of the null hypothesis in the tradition of Fisher. A physicist could use (...) the null hypothesis method as well as the psychologist. The fact that he doesn't is probably related to the more advanced state of his measurement techniques and theoretical constructs. (shrink)

A common strategy in comparative cognition is to require that one reject associative learning as an explanation for behavior before concluding that an organism is capable of causal reasoning. In this paper, I argue that standard causal-reasoning tasks can be explained by a powerful form of associative learning: unlimited associative learning (UAL). The lesson, however, is not that researchers should conduct more studies to reject UAL, but that they should instead focus on 1) enriching the cognitive hypothesis space and (...) 2) testing a broader range of information processing patterns—errors, biases and limits, rather than successful problem solving alone. (shrink)

Neyman-Pearson methods in statistics distinguish between Type I and Type II errors. Through rigid control of Type I error, the "null" hypothesis typically receives the benefit of the doubt. I compare philosophers' interpretations of this feature of Neyman-Pearson tests with interpretations given in statistics textbooks. The pragmatic view of the tests advocated by Neyman, largely rejected by philosophers, lives on in many textbooks. Birnbaum thought the pragmatic view has a useful "heuristic" role in understanding testing. I suggest (...) that it may have the opposite effect. (shrink)

Chow's one-tailed null-hypothesis significance-test procedure, with its rationale based on the elimination of chance influences, is not appropriate for theory-corroboration experiments. Estimated effect sizes and their associated standard errors or confidence limits will always suffice.

Setting optimal significance levels that minimize Type I and Type II errors allows for more transparent and well‐considered statistical decision making compared to the traditional α = 0.05 significance level. We use the optimal α approach to re‐assess conclusions reached by three recently published tests of the pace‐of‐life syndrome hypothesis, which attempts to unify occurrences of different physiological, behavioral, and life history characteristics under one theory, over different scales of biological organization. While some of the conclusions reached using optimal (...) α were consistent to those previously reported using the traditional α = 0.05 threshold, opposing conclusions were also frequently reached. The optimal α approach reduced probabilities of Type I and Type II errors, and ensured statistical significance was associated with biological relevance. Biologists should seriously consider their choice of α when conducting null hypothesis significance tests, as there are serious disadvantages with consistent reliance on the traditional but arbitrary α = 0.05 significance level. (shrink)

The full Bayesian signi/cance test (FBST) for precise hypotheses is presented, with some illustrative applications. In the FBST we compute the evidence against the precise hypothesis. We discuss some of the theoretical properties of the FBST, and provide an invariant formulation for coordinate transformations, provided a reference density has been established. This evidence is the probability of the highest relative surprise set, “tangential” to the sub-manifold (of the parameter space) that defines the null hypothesis.

Chow's (1996) Statistical significance is a defence of null-hypothesis significance testing (NHSTP). The most common and straightforward use of significance testing is for the statistical corroboration of general hypotheses. In this case, criticisms of NHSTP, at least those mentioned in the book, are unfounded or misdirected. This point is driven home by the author a bit too forcefully and meticulously. The awkward and cumbersome organisation and argumentation of the book makes it even harder to read.

The ecologist today finds scarce ground safe from controversy. Decisions must be made about what combination of data, goals, methods, and theories offers them the foundations and tools they need to construct and defend their research. When push comes to shove, ecologists often turn to philosophy to justify why it is their approach that is scientific. Karl Popper’s image of science as bold conjectures and heroic refutations is routinely enlisted to justify testing hypotheses over merely confirming them. One of (...) the most controversial theories in contemporary science is the Neutral Theory of Ecology. Its chief developer and proponent, Stephen Hubbell, presents the neutral theory as a bold conjecture that has so far escaped refutation. Critics of the neutral theory claim that it already stands refuted, despite what the dogmatic neutralists say. We see the controversy through a Popperian lens. But Popper’s is an impoverished philosophy of science that distorts contemporary ecology. -/- The controversy surrounding the neutral theory actually rests on a methodological fault. There is a strong but messy historical link between the concepts of being neutral and being null in biology, and Hubbell perpetuates this when he claims that the neutral theory is supplies the appropriate null for testing alternative theories. What method is being followed here? There are three contenders: -/- Null hypothesis testing tests for whether a there is a pattern to be explained. Null modeling tests for whether a process is causally relevant to a pattern. Baseline modeling apportions relative responsibility to multiple processes each relevant to a pattern. -/- Whether the neutral theory supplies an appropriate “null” depends upon whether null hypothesis, null modeling, or baseline model is intended. These methods prescribe distinct inference patterns. If they are null hypothesis testing or null modeling, the neutralists’s reasoning is invalid. If they are baseline modeling, the justification of a crucial assumption remains opaque. Either way, the neutral-null connection is being exploited rhetorically to privilege the neutral theory over its rivals. Clarifying the reasoning immunizes us against the rhetoric and foregrounds the underlying virtues of the neutralist approach to ecology. -/- The Popperian lens distorts theoretical development as dogmatism. Lakatos’s view of science as the development of research programmes clarifies the epistemology of the neutral theory. Focusing philosophical attention on the neutralist research programme illuminates (1) the synchronic uses of the neutral theory to make predictions and give descriptions and explanations; (2) its diachronic development in response to theoretical innovation and confrontation with data; (3) its complex relationships to alternative theories. For example, baseline modeling is now seen to be its primary explanatory heuristic. The justification for baseline modeling with the neutral theory, previously hidden from view, is seen in the logic of in the neutralist research programme. (shrink)

Abstract: The Pull Bayesian Significance Test (FBST) for precise hy- potheses is applied to a Multivariate Normal Structure (MNS) model. In the FBST we compute the evidence against the precise hypothesis. This evi- dence is the probability of the Highest Relative Surprise Set (HRSS) tangent to the sub-manifold (of the parameter space) that defines the null hypothesis. The MNS model we present appears when testing equivalence conditions for genetic expression measurements, using micro-array technology.

The unit root problem plays a central role in empirical applications in the time series econometric literature. However, significance tests developed under the frequentist tradition present various conceptual problems that jeopardize the power of these tests, especially for small samples. Bayesian alternatives, although having interesting interpretations and being precisely defined, experience problems due to the fact that that the hypothesis of interest in this case is sharp or precise. The Bayesian significance test used in this article, for the unit (...) root hypothesis, is based solely on the posterior density function, without the need of imposing positive probabilities to sets of zero Lebesgue measure. Furthermore, it is conducted under strict observance of the likelihood principle. It was designed mainly for testing sharp null hypotheses and it is called FBST for Full Bayesian Significance Test. (shrink)

The Full Bayesian Significance Test (FBST) for precise hypotheses is presented, with some applications relevant to reliability theory. The FBST is an alternative to significance tests or, equivalently, to p-ualue.s. In the FBST we compute the evidence of the precise hypothesis. This evidence is the probability of the complement of a credible set "tangent" to the sub-manifold (of the para,rreter space) that defines the null hypothesis. We use the FBST in an application requiring a quality control of (...) used components, based on remaining life statistics. (shrink)

Scientists often adjust their significance threshold during null hypothesis significance testing in order to take into account multiple testing and multiple comparisons. This alpha adjustment has become particularly relevant in the context of the replication crisis in science. The present article considers the conditions in which this alpha adjustment is appropriate and the conditions in which it is inappropriate. A distinction is drawn between three types of multiple testing: disjunction testing, conjunction testing, and (...) individual testing. It is argued that alpha adjustment is only appropriate in the case of disjunction testing, in which at least one test result must be significant in order to reject the associated joint null hypothesis. Alpha adjustment is inappropriate in the case of conjunction testing, in which all relevant results must be significant in order to reject the joint null hypothesis. Alpha adjustment is also inappropriate in the case of individual testing, in which each individual result must be significant in order to reject each associated individual null hypothesis. The conditions under which each of these three types of multiple testing is warranted are examined. It is concluded that researchers should not automatically assume that alpha adjustment is necessary during multiple testing. Illustrations are provided in relation to joint studywise hypotheses and joint multiway ANOVAwise hypotheses. (shrink)

This paper reports on a survey sent to financial executives at 405 small corporations. A cover letter assured recipients all survey responses would be anonymous and that the enclosed $5 check was to be considered payment for completing and returning the survey. The letter requested the check be returned or destroyed if the survey was not going to be completed and returned.In a quantitative test of financial ethics, the proportion of cancelled checks and checks returned with a completed survey is (...) compared to the proportion of completed and returned surveys. Based on the Z test statistic, the null hypothesis of equal proportions was accepted, implying ethical behavior by the group of executives in this test. Suggestions are given for future research. (shrink)

Chow's endorsement of a limited role for null hypothesis significance testing is a needed corrective of research malpractice, but his decision to place this procedure in a hypothetico-deductive framework of Popperian cast is unwise. Various failures of this version of the hypothetico-deductive method have negative implications for Chow's treatment of significance testing, meta-analysis, and theory evaluation.

It is generally accepted among philosophers of science that hypothesis testing is a key methodological feature of science. As far as philosophical theories of confirmation are con...

The null-hypothesis significance-test procedure (NHSTP) is defended in the context of the theory-corroboration experiment, as well as the following contrasts: (a) substantive hypotheses versus statistical hypotheses, (b) theory corroboration versus statistical hypothesis testing, (c) theoretical inference versus statistical decision, (d) experiments versus nonexperimental studies, and (e) theory corroboration versus treatment assessment. The null hypothesis can be true because it is the hypothesis that errors are randomly distributed in data. Moreover, the null (...) class='Hi'>hypothesis is never used as a categorical proposition. Statistical significance means only that chance influences can be excluded as an explanation of data; it does not identify the nonchance factor responsible. The experimental conclusion is drawn with the inductive principle underlying the experimental design. A chain of deductive arguments gives rise to the theoretical conclusion via the experimental conclusion. The anomalous relationship between statistical significance and the effect size often used to criticize NHSTP is more apparent than real. The absolute size of the effect is not an index of evidential support for the substantive hypothesis. Nor is the effect size, by itself, informative as to the practical importance of the research result. Being a conditional probability, statistical power cannot be the apriori probability of statistical significance. The validity of statistical power is debatable because statistical significance is determined with a single sampling distribution of the test statistic based on H0, whereas it takes two distributions to represent statistical power or effect size. Sample size should not be determined in the mechanical manner envisaged in power analysis. It is inappropriate to criticize NHSTP for nonstatistical reasons. At the same time, neither effect size, nor confidence interval estimate, nor posterior probability can be used to exclude chance as an explanation of data. Neither can any of them fulfill the nonstatistical functions expected of them by critics. (shrink)

My aim in this paper is to show how the problem of inflated effect sizes corrupts the severity measure of evidence. This has never been done. In fact, the Winner’s Curse is barely mentioned in the philosophical literature. Since the severity score is the predominant measure of evidence for frequentist tests in the philosophical literature, it is important to underscore its flaws. It is also crucial to bring the philosophical literature up to speed with the limits of classical testing. (...) The Winner’s Curse is one of them. The problem is that when a significant result is obtained by using an underpowered test, the severity score becomes particularly high for large discrepancies from the null-hypothesis. This means that such discrepancies are very well supported by the evidence according to that measure. However, it is now well documented that significant tests with low power display inflated effect sizes. They systematically show departures from the null hypothesis H0 that are much greater than they really are. From an epistemological point of view this means that a significant result produced by an underpowered test does not provide evidence for large discrepancies from H0. Therefore, the severity score is an inadequate measure of evidence. Given that we are now aware of the phenomenon of inflated effect sizes, it would be irresponsible to rely on the severity score to measure the strength of the evidence against the null. Instead, one must take appropriate measures to try and avoid using underpowered tests by setting a threshold for the sample size or by replicating the results of the experiment. (shrink)

A Bayesian measure of evidence for precise hypotheses is presented. The intention is to give a Bayesian alternative to significance tests or, equivalently, to p-values. In fact, a set is defined in the parameter space and the posterior probability, its credibility, is evaluated. This set is the “Highest Posterior Density Region” that is “tangent” to the set that defines the null hypothesis. Our measure of evidence is the complement of the credibility of the “tangent” region.

Chow's book provides a thorough analysis of the confusing array of issues surrounding conventional tests of statistical significance. This book should be required reading for behavioral and social scientists. Chow concludes that the null-hypothesis significance-testing procedure (NHSTP) plays a limited, but necessary, role in the experimental sciences. Another possibility is that – owing in part to its metaphorical underpinnings and convoluted logic – the NHSTP is declining in importance in those few sciences in which it ever played (...) a role. (shrink)