My aim in this paper is to show how the problem of inflated effect sizes corrupts the severity measure of evidence. This has never been done. In fact, the Winner’s Curse is barely mentioned in the philosophical literature. Since the severity score is the predominant measure of evidence for frequentist tests in the philosophical literature, it is important to underscore its flaws. It is also crucial to bring the philosophical literature up to speed with the limits of classical (...) testing. The Winner’s Curse is one of them. The problem is that when a significant result is obtained by using an underpowered test, the severity score becomes particularly high for large discrepancies from the null-hypothesis. This means that such discrepancies are very well supported by the evidence according to that measure. However, it is now well documented that significant tests with low power display inflated effect sizes. They systematically show departures from the null hypothesis H0 that are much greater than they really are. From an epistemological point of view this means that a significant result produced by an underpowered test does not provide evidence for large discrepancies from H0. Therefore, the severity score is an inadequate measure of evidence. Given that we are now aware of the phenomenon of inflated effect sizes, it would be irresponsible to rely on the severity score to measure the strength of the evidence against the null. Instead, one must take appropriate measures to try and avoid using underpowered tests by setting a threshold for the sample size or by replicating the results of the experiment. (shrink)

DOI: 10.1080/00031305.2018.1564697 When the editors of Basic and Applied Social Psychology effectively banned the use of null hypothesis significance testing (NHST) from articles published in their journal, it set off a fire-storm of discussions both supporting the decision and defending the utility of NHST in scientific research. At the heart of NHST is the p-value which is the probability of obtaining an effect equal to or more extreme than the one observed in the sample data, given the null hypothesis (...) and other model assumptions. Although this is conceptually different from the probability of the null hypothesis being true, given the sample, p-values nonetheless can provide evidential information, toward making an inference about a parameter. Applying a 10,000-case simulation described in this article, the authors found that p-values’ inferential signals to either reject or not reject a null hypothesis about the mean (α = 0.05) were consistent for almost 70% of the cases with the parameter’s true location for the sampled-from population. Success increases if a hybrid decision criterion, minimum effect size plus p-value (MESP), is used. Here, rejecting the null also requires the difference of the observed statistic from the exact null to be meaningfully large or practically significant, in the researcher’s judgment and experience. The simulation compares performances of several methods: from p-value and/or effect size-based, to confidence-interval based, under various conditions of true location of the mean, test power, and comparative sizes of the meaningful distance and population variability. For any inference procedure that outputs a binary indicator, like flagging whether a p-value is significant, the output of one single experiment is not sufficient evidence for a definitive conclusion. Yet, if a tool like MESP generates a relatively reliable signal and is used knowledgeably as part of a research process, it can provide useful information. (shrink)

This is a brief response to David Wright’s commentary on my paper, “Critical Thinking Instruction: A Realistic Evaluation: The Dream vs. Reality.” Wright claims that if one looks more closely at the literature on critical thinking assessment that the reported effect sizes for CT instruction are quite respectable and my standards are too high. My comments will focus is on whether effect size is both problematic and an adequate measure for assessment.

Sex dimorphism occurs when group means differ by four or more standard deviations. However, the average size of the corpus callosum is greater in males by about one standard deviation in rats, 0.2 standard deviation in humans, and virtually zero in mice. Furthermore, variations in corpus callosum size are related to brain size and are not sex specific.

Absolute and relative outcome measures measure a treatment’s effect size, purporting to inform treatment choices. I argue that absolute measures are at least as good as, if not better than, relative ones for informing rational decisions across choice scenarios. Specifically, this dominance of absolute measures holds for choices between a treatment and a control group treatment from a trial and for ones between treatments tested in different trials. This distinction has hitherto been neglected, just like the role of absolute (...) and baseline risks in rational decision-making that my analysis reveals. Recognizing both aspects advances the discussion on reporting outcome measures. (shrink)

Alternative displays of effect size statistics can enhance the understandability and impact of validity evidence in a variety of applied settings. Arguably, the proliferation of alternative effect size statistics has been limited due to the lack of user-friendly tools to create them. Common statistical packages do not readily produce these alternative effect sizes and existing tools are outdated and inaccessible. In this paper, I introduce a free-to-use web-based calculator (https://dczhang.shinyapps.io/expectancyApp/) for generating alternative effect size displays (...) from empirical data. This calculator requires no mathematical or programming expertise, and therefore, is ideal for academics and practitioners. I also present results from an empirical study that demonstrates the benefits of alternative effect size displays for enhancing lay people’s perceived understandability of validity information and attitudes toward the use of standardized testing for college admissions. (shrink)

The usefulness of effect-size differs in utilitarian experiments from its use in theory corroborations. Chow introduces the question of the relationship of effect-size to practical validity and the role of the assessment of “importance” in this. This review develops this question and suggests the actuarial table as a replacement for effect-size in practical decision-making.

The variance effective population size (NeV\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{eV}$$\end{document}) is frequently used to quantify the expected rate at which a population’s allele frequencies change over time. The purpose of this paper is to find expressions for the global NeV\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{eV}$$\end{document} of a spatially structured population that are of interest for conservation of species. Since NeV\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{eV}$$\end{document} depends on allele (...) frequency change, we start by dividing the cause of allele frequency change into genetic drift within subpopulations (I) and a second component mainly due to migration between subpopulations (II). We investigate in detail how these two components depend on the way in which subpopulations are weighted as well as their dependence on parameters of the model such a migration rates, and local effective and census sizes. It is shown that under certain conditions the impact of II is eliminated, and NeV\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{eV}$$\end{document} of the metapopulation is maximized, when subpopulations are weighted proportionally to their long term reproductive contributions. This maximal NeV\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{eV}$$\end{document} is the sought for global effective size, since it approximates the gene diversity effective size NeGD\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{eGD}$$\end{document}, a quantifier of the rate of loss of genetic diversity that is relevant for conservation of species and populations. We also propose two novel versions of NeV\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{eV}$$\end{document}, one of which (the backward version of NeV\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{eV}$$\end{document}) is most stable, exists for most populations, and is closer to NeGD\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{eGD}$$\end{document} than the classical notion of NeV\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{eV}$$\end{document}. Expressions for the optimal length of the time interval for measuring genetic change are developed, that make it possible to estimate any version of NeV\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{eV}$$\end{document} with maximal accuracy. (shrink)

Recent downward trends in corporate giving have renewed interest in the factors that shape corporate philanthropy. This paper examines the relationships between charitable contributions, firm size and industry. Improvements over previous studies include an IRS data base that covers a much broader range of firm sizes and industries as compared to previous studies and estimation using an instrumental variable technique that explicitly addresses potential simultaneity between charitable contributions and profitability. Important findings provide evidence of a cubic relationship between charitable (...) giving and firm size and evidence of strong industry effects. The plus-minus-plus regression coefficient sign pattern for the cubic firm size model suggests that small and large firms give more relative to total receipts with lower giving ratios among medium size firms. One interpretation for this finding is that small firms are close to the communities they serve while high visibility creates a need for large firm philanthropy. Strong industry effects provide evidence of inter-industry differences in giving culture and/or different public relations requirements across industries. (shrink)

Difficulties understanding speech form one of the most prevalent complaints among older adults. Successful speech perception depends on top-down linguistic and cognitive processes that interact with the bottom-up sensory processing of the incoming acoustic information. The relative roles of these processes in age-related difficulties in speech perception, especially when listening conditions are not ideal, are still unclear. In the current study, we asked whether older adults with a larger working memory capacity process speech more efficiently than peers with lower capacity (...) when speech is presented in noise, with another task performed in tandem. Using the Eye-tracking of Word Identification in Noise Under Memory Increased Load an adapted version of the “visual world” paradigm, 36 older listeners were asked to follow spoken instructions presented in background noise, while retaining digits for later recall under low or high memory load. In critical trials, instructions directed listeners’ gaze to pictures of objects whose names shared onset or offset sounds with the name of a competitor that was displayed on the screen at the same time. We compared listeners with different memory capacities on the time course for spoken word recognition under the two memory loads by testing eye-fixations on a named object, relative to fixations on an object whose name shared phonology with the named object. Results indicated two trends. For older adults with lower working memory capacity, increased memory load did not affect online speech processing, however, it impaired offline word recognition accuracy. The reverse pattern was observed for older adults with higher working memory capacity: increased task difficulty significantly decreases online speech processing efficiency but had no effect on offline word recognition accuracy. Results suggest that in older adults, adaptation to adverse listening conditions is at least partially supported by cognitive reserve. Therefore, additional cognitive capacity may lead to greater resilience of older listeners to adverse listening conditions. The differential effects documented by eye movements and accuracy highlight the importance of using both online and offline measures of speech processing to explore age-related changes in speech perception. (shrink)

Business ethics journals have appeared on a few ranked lists that are specific to this niche discipline. As with more traditional academic disciplines, these rankings are used for academic rewards such as faculty tenure and promotion, along with department and school ratings. Journal ranking has been subject to considerable criticism even as its administrative use persists. Among the criticisms are that journal quality is a poor proxy for article quality, citation rate is an imperfect reflection of article influence, and bias (...) may be introduced into rankings by visibility characteristics such as journal age, size, circulation, and experience of the rater with a journal. This research note studies the effect of journal age and size on the rankings of business ethics journals compiled by Beets, Lewis, and Brower, by Albrecht, Thompson, Hoopes, and Rodrigo, and by Serenko and Bontis. Significant correlation was found for journal age with the administratively derived Beets et al. ranking. No significant correlation was found for size in any ranking study. Results were not significant for the Albrecht et al. and the Serenko and Bontis rankings representing the perspectives from surveys of active researchers or citation analysis. Perhaps sometimes a journal’s reputation precedes it, as perception of journal quality may be biased by journal visibility, either because it has been published and available for a number of years, or because it is well known and likely to be cited. (shrink)

The visual system adapts to a wide range of visual features, from lower-level features like color and motion to higher-level features like causality and, perhaps, number. According to some, adaptation is a strictly perceptual phenomenon, such that the presence of adaptation licenses the claim that a feature is truly perceptual in nature. Given the theoretical importance of claims about adaptation, then, it is important to understand exactly when the visual system does and does not exhibit adaptation. Here, we take as (...) a case study one specific kind of adaptation: visual adaptation to size. Supported by evidence from four experiments, we argue that, despite robust effects of size adaptation in the lab, (1) size adaptation effects are phenomenologically underwhelming (in some cases, hardly appreciable at all), (2) some effects of size adaptation appear contradictory, and difficult to explain given current theories of size adaptation, and (3) prior studies on size adaptation may have failed to isolate size as the adapted dimension. Ultimately, we argue that while there is evidence to license the claim that size adaptation is genuine, size adaptation is a puzzling and poorly understood phenomenon. (shrink)