The generalizability crisis

Behavioral and Brain Sciences 45:1-37 (2022)
  Copy   BIBTEX

Abstract

Most theories and hypotheses in psychology are verbal in nature, yet their evaluation overwhelmingly relies on inferential statistical procedures. The validity of the move from qualitative to quantitative analysis depends on the verbal and statistical expressions of a hypothesis being closely aligned – that is, that the two must refer to roughly the same set of hypothetical observations. Here, I argue that many applications of statistical inference in psychology fail to meet this basic condition. Focusing on the most widely used class of model in psychology – the linear mixed model – I explore the consequences of failing to statistically operationalize verbal hypotheses in a way that respects researchers' actual generalization intentions. I demonstrate that although the “random effect” formalism is used pervasively in psychology to model intersubject variability, few researchers accord the same treatment to other variables they clearly intend to generalize over. The under-specification of random effects imposes far stronger constraints on the generalizability of results than most researchers appreciate. Ignoring these constraints can dramatically inflate false-positive rates, and often leads researchers to draw sweeping verbal generalizations that lack a meaningful connection to the statistical quantities they are putatively based on. I argue that failure to take the alignment between verbal and statistical expressions seriously lies at the heart of many of psychology's ongoing problems, and conclude with a discussion of several potential avenues for improvement.

Links

PhilArchive



    Upload a copy of this work     Papers currently archived: 90,616

External links

Setup an account with your affiliations in order to access resources via your University's proxy server

Through your library

Similar books and articles

Four reasons why the science of psychology is still in trouble.Kim J. Vicente - 1998 - Behavioral and Brain Sciences 21 (2):224-225.
Statistical Inference and the Replication Crisis.Lincoln J. Colling & Dénes Szűcs - 2018 - Review of Philosophy and Psychology 12 (1):121-147.
Associative factors in verbal transfer.Jarvis Bastian - 1961 - Journal of Experimental Psychology 62 (1):70.
Hypothesis Testing as a Moral Choice.David J. Pittenger - 2001 - Ethics and Behavior 11 (2):151-162.

Analytics

Added to PP
2020-12-22

Downloads
46 (#304,991)

6 months
11 (#128,523)

Historical graph of downloads
How can I increase my downloads?

Author's Profile

Tal Yarkoni
University of Texas at Austin

References found in this work

Studies of interference in serial verbal reactions.J. R. Stroop - 1935 - Journal of Experimental Psychology 18 (6):643.
Novel evidence and severe tests.Deborah G. Mayo - 1991 - Philosophy of Science 58 (4):523-552.

View all 7 references / Add more references