Most theories and hypotheses in psychology are verbal in nature, yet their evaluation overwhelmingly relies on inferential statistical procedures. The validity of the move from qualitative to quantitative analysis depends on the verbal and statistical expressions of a hypothesis being closely aligned – that is, that the two must refer to roughly the same set of hypothetical observations. Here, I argue that many applications of statistical inference in psychology fail to meet this basic condition. Focusing on the most widely used (...) class of model in psychology – the linear mixed model – I explore the consequences of failing to statistically operationalize verbal hypotheses in a way that respects researchers' actual generalization intentions. I demonstrate that although the “random effect” formalism is used pervasively in psychology to model intersubject variability, few researchers accord the same treatment to other variables they clearly intend to generalize over. The under-specification of random effects imposes far stronger constraints on the generalizability of results than most researchers appreciate. Ignoring these constraints can dramatically inflate false-positive rates, and often leads researchers to draw sweeping verbal generalizations that lack a meaningful connection to the statistical quantities they are putatively based on. I argue that failure to take the alignment between verbal and statistical expressions seriously lies at the heart of many of psychology's ongoing problems, and conclude with a discussion of several potential avenues for improvement. (shrink)

Yarkoni highlights patterns of overgeneralization in psychology research. In this comment, we note that such challenges also pertain to applied psychological and organizational research and practice. We use two examples – cross-cultural generalizability and implicit bias training – to illustrate common practices of overgeneralization from narrow research samples to broader operational populations. We conclude with recommendations for research and practice.

Traditional statistical model evaluation typically relies on goodness-of-fit testing and quantifying model complexity by counting parameters. Both of these practices may result in overfitting and have thereby contributed to the generalizability crisis. The information-theoretic principle of minimum description length addresses both of these concerns by filtering noise from the observed data and consequently increasing generalizability to unseen data.

Part of the generalizability issues that haunt controlled lab experiment designs in psychology, and more particularly in psycholinguistics, can be alleviated by adopting corpus linguistic methods. These work with natural data. This advantage comes at a cost: in corpus studies, lexemes and language users can show different kinds of skew. We discuss a number of solutions to bolster the control.

The question of how we apply knowledge from biomedical science to medical and public health practice has been the subject of heated debates about generalizability and related concepts, such as applicability and inductive inference. In this essay, I interpret the term from the perspective of two causal models: determinism and indeterminism. I suggest that theories of generalizability can be formulated on the basis of both models and take the form of testable but unverifiable hypotheses, an attribute that is (...) common to all scientific theories. Nonetheless, there is one noteworthy difference between the two models: determinism allows one to rationalize a decision to treat a certain kind of patient but only indirectly a decision to treat any particular patient, whereas indeterminism accommodates both types of decisions. (shrink)

The explanatory/pragmatic-trial distinction enjoys a burgeoning philosophical and medical literature and a significant contingent of support among philosophers and healthcare stakeholders as an important way to assess the design and results of randomized controlled trials. A major motivation has been the need to provide relevant, generalizable data to drive healthcare decisions. While talk of pragmatic and explanatory trials could be seen as convenient shorthand, the distinction can also be seen as harboring deeper issues related to inferential strategies used to evaluate (...) causal claims regarding medical treatments. A comprehensive, critical analysis of the distinction and underlying epistemological framework upon which the distinction is based, particularly with respect to treatment effectiveness, has yet to be forthcoming. I provide this, analyzing the distinction’s relationship to generalizability and cognate distinctions between ideal conditions and real-world practice, internal and external validity, and efficacy and effectiveness. I also analyze recent philosophical work that relies on the explanatory/pragmatic-trial distinction and that advocates for more pragmatic trials. I conclude that as an organizing principle for trial-design decisions and trial evaluation, the explanatory/pragmatic-trial distinction is conceptually problematic and not as useful as its proponents seem to think. Since some pragmatic-trial features can be inimical to establishing treatment effectiveness, pragmatic-trial features should not be conflated with pragmatic trials’ avowed goals. If the distinction is to be useful, it and some associated concepts, including generalizability, should be reformulated, lest they continue to underlie a medical epistemology that could contribute to methodologically flawed and potentially unethical advice for the design and interpretation of trials. (shrink)

In this paper I convey a recurring problem and possible solution that arose during my doctoral research on the topic of cross-cultural Holocaust curriculum development for Latvian schools. Specifically, as I devised the methodology for my research, I experienced a number of frustrations concerning the issue of transferability and the limitations of both reader generalizability and grounded theory. Ultimately, I found a more appropriate goal for the external applicability of this and other highly contextual research studies in the form (...) of "grounded understandings," which are tentative apprehensions of the importance or significance of phenomena and conceptualizations that hold meaning and explanatory power, but are only embryonic in their potential to generate theory. (shrink)

Emphasizing the predictive success and practical utility of psychological science is an admirable goal but it will require a substantive shift in how we design research. Applied research often assumes that findings are transferable to all practices, insensitive to variation between implementations. We describe efforts to quantify and close this practice-to-practice gap in education research.

Falsificationist and confirmationist approaches provide two well-established ways of evaluating generalizability. Yarkoni rejects both and invents a third approach we call neo-operationalism. His proposal cannot work for the hypothetical concepts psychologists use, because the universe of operationalizations is impossible to define, and hypothetical concepts cannot be reduced to their operationalizations. We conclude that he is wrong in his generalizability-crisis diagnosis.

A child’s age in comparison to the age of her or his classmates (relative age) has been found to be an influential factor on academic achievement, particularly but not exclusively at the beginning of formal schooling. However, few studies have focused on the generalizability of relative age effects. To close this gap, the present study analyzes the generalizability across students with and without immigrant backgrounds, across three student cohorts that entered school under a changing law of school enrollment, (...) and across classes. To this end, we capitalized on representative large-scale data sets from three student cohorts attending public schools in Berlin, the capital of Germany. We analyzed the data using a multilevel framework. Our results for the overall student sample indicate relative age effects for reading and mathematics in favor of the relatively older students in Grade 2 that become somewhat smaller in size in Grade 3. By Grade 8, relative age effects had vanished in reading and had even reversed in favor of the relatively young in mathematics. Furthermore, relative age effects were not found to be systematically different among students with and without immigrant backgrounds, student cohorts, or across classes. Taken together, these results empirically underscore the broad generalizability of the findings as found for the overall student population and replicate the pattern of findings on relative effects as identified by the majority of previous studies. (shrink)

Because of the misspecification of models and specificity of operationalizations, many studies produce claims of limited utility. We suggest a path forward that requires taking a few steps back. Researchers can retool large-scale replications to conduct the descriptive research which assesses the generalizability of constructs. Large-scale construct validation is feasible and a necessary next step in addressing the generalizability crisis.

Certain researchers in the field of moral psychology, following Turiel, argue that children and adults in different cultures make a distinction between moral and conventional transgressions. One interpretation of the theory holds that moral transgressions elicit a signature moral response pattern while conventional transgressions elicit a signature conventional response pattern. Four dimensions distinguish the moral response pattern from the conventional response pattern. 1. HARM/JUSTICE/RIGHTS – Subjects justify the wrongness of moral transgressions by stating that they involve a victim that is (...) harmed, whose rights have been violated or who has been subject to an injustice. Conventional transgressions do not involve a victim that is harmed, whose rights have been violated or who has been subject to an injustice. 2. AUTHORITY – Subjects judge moral transgressions as wrong independent of structures of authority while the wrongness of conventional transgressions can be changed by an authority. 3. GENERALIZABILITY – Subjects judge moral transgressions as generalizably wrong, i.e., independent of time and place, while conventional transgressions’ wrongness depends on time and place. 4. SERIOUSNESS – Subjects judge moral transgressions as more seriously wrong than conventional transgressions. Others have criticized this view for a diversity of reasons. Relevant for our purposes is that, first, there appear to be cultural differences in what constitutes a moral transgression and second, it is unclear what the exact hypotheses are, surrounding this supposed moral/conventional distinction. I will present planned and ongoing experimental research that investigates two specific problems we encountered in the moral-conventional literature. First of all, we cannot draw reliable conclusions from previous work about the generalizability of the wrongness of different kinds of transgressions. In previous experiments, differences in time and place are often but not always confounded with a variety of other differences. For example, Huebner et al. ask participants if the depicted act would be OK for someone who lived elsewhere where everyone else did this. Moreover, when varying time and/or place, participants are likely to assume that other things differ as well. In our study, we vary time and/or place in a variety of scenarios in order to investigate what assumptions participants make when confronted with the generalizability question. Second, it is an open question as to what extent any transgression will universally elicit one of the two signature response patterns. In our study, we make use of existing differences in participants’ value hierarchy to test this. For one and the same scenario, we compare the response of participants for whom authority is an important value with the results of participants for whom authority is not an important value, in order to see if there are differences in the two groups’ response patterns. References: Haidt J., Koller S. & Dias M. 1993. Affect, culture and morality, or is it wrong to eat your dog? Journal of Personality and Social Psychology 65:613-628. Huebner B., Lee, J.L. & Hauser, M.D. 2010. The Moral-Conventional Distinction in Mature Moral Competence. Journal of Cognition and Culture 10: 1-26. Kelly D., Stich S., Haley K.J., Eng S.J. & Fessler D.M.T. 2007. Harm, Affect, and the Moral/Conventional Distinction. Mind & Language 22:117-131. Nichols S. 2004. Sentimental Rules: on the Natural Foundations of Moral Judgment. Oxford University Press. Stich S., Fessler, D.M.T. & Kelly D. 2009. On the Morality of Harm: A response to Sousa, Holbrook and Piazza. Cognition 113:93-97. Turiel E. 1983. The Development of Social Knowledge. Morality & Convention. Cambridge University Press. (shrink)

Yarkoni's analysis clearly articulates a number of concerns limiting the generalizability and explanatory power of psychological findings, many of which are compounded in infancy research. ManyBabies addresses these concerns via a radically collaborative, large-scale and open approach to research that is grounded in theory-building, committed to diversification, and focused on understanding sources of variation.

Psychology's tendency to focus on confirmatory analyses before ensuring constructs are clearly defined and accurately measured is exacerbating the generalizability crisis. Our growing use of digital behaviors as predictors has revealed the fragility of subjective measures and the latent constructs they scaffold. However, new technologies can provide opportunities to improve conceptualizations, theories, and measurement practices.

The influence on the global evaluation of a person based on the perception of a single trait is a phenomenon widely investigated in social psychology. Widely regarded as Halo effect, this phenomenon has been studied for more than 100 years now, and findings such as the relationship between aesthetic perception and other personality traits—such as competence and trustworthiness—have since been uncovered. Trustworthiness plays an especially crucial role in individuals' social interactions. Despite the large body of literature published on the Halo (...) effect, and especially on the relationship between aesthetic appearance and perceived trustworthiness, little is known about the overall generalizability of the effect, as almost all of the studies have been conducted on adult participants from Western countries. Moreover, little is known about the stability of the effect over time, in the event of major destabilization, such as the outbreak of a pandemic. In this work, the cross-cultural generalizability of the Halo effect is investigated before and during the first few months of the COVID-19 pandemic. An analysis of the generalizability and stability over time of the Halo effect is presented. Participants have been asked to rate the aesthetic appearance and perceived trustworthiness of a set of human faces of different ages, gender, and ethnicity. Result of our analysis demonstrated that the Halo effect is influenced by the age of presented faces, but not by their gender or ethnicity. Moreover, our results show that the strength of the effect can be affected by external events and that the volatility is higher for adults' than children's faces. (shrink)

Pain continues to be one of the most controversial subjects in neurophilosophy. One focus of current debates is the apparent absence of an ideal brain‐based biomarker that could function as a coherent and distinct indicator for pain. One prominent reaction to this in the philosophical literature is scientific pain eliminativism. In this article, I argue for a non‐eliminative alternative that builds on family resemblances and provides a useful heuristic in the tradeoff between the idiosyncrasy of the neural processes corresponding to (...) different pain cases and the demand for generalizability in pain research. (shrink)

The aim of the social and behavioral sciences is to understand human behavior across a wide array of contexts. Our theories often make sweeping claims about human nature, assuming that our ancestors or offspring will be prone to the same biases and preferences. Yet we gloss over the fact that our research is often based in a single temporal context with a limited set of stimuli. Political and moral psychology are domains in which the context and stimuli are likely to (...) matter a great deal (Van Bavel, Mende-Siedlecki, Brady, & Reinero, 2016). In response to Yarkoni (see BBS issue), we delve into topics related to political and moral psychology that likely depend on features of the research. These topics include understanding differences between liberals and conservatives, when people are willing to sacrifice someone to save others, the behavior of political leaders, and the dynamics of intergroup conflict. (shrink)

Cowan's experimental techniques cannot constrain subject's recall of presented information to distinct independent chunks in short-term memory (STM). The encoding of associations in long-term memory contaminates recall of pure STM capacity. Even in task environments where the functional independence of chunks is convincingly demonstrated, individuals can increase the storage of independent chunks with deliberate practice – well above the magical number four.

Teaching is hypothesized to be a species-typical behavior in humans that contributed to the emergence of cumulative culture. Several within-culture studies indicate that foragers depend heavily on social learning to acquire practical skills and knowledge, but it is unknown whether teaching is universal across forager populations. Teaching can be defined ethologically as the modification of behavior by an expert in the presence of a novice, such that the expert incurs a cost and the novice acquires skills/knowledge more efficiently or that (...) it would not acquire otherwise. One behavioral modification hypothesized to be an adaptation for teaching is ostensive communication—exaggerations of prosody and gesture that signal intent to transmit generalizable knowledge and indicate the intended receiver. On this view, the use of ostensive communication in conjunction with the transmission of generalizable knowledge constitutes evidence of teaching. Oral storytelling appears to meet these criteria: Indigenous peoples regard their traditions as important sources of ecological and social knowledge, and oral storytelling is widely reported to employ paralinguistic communication. To test this hypothesis, descriptions of performed narrative in forager societies were coded for the use of 14 ostensive-communicative behaviors and the presence of generalizable knowledge. Although biased toward North America, the study sample comprised 53 forager cultures spanning five continents, 34 language families, and diverse biomes. All cultures evinced the predicted behaviors. Results suggest that foragers use storytelling as a mode of instruction, thus providing cross-cultural evidence of teaching in forager populations. (shrink)

We applaud the effort to draw attention to generalizability concerns in twenty-first-century psychological research. Yet we do not feel that a pessimistic perspective is warranted. We outline a continuum of available methodological tools and perspectives, including incremental steps and meta-analytic approaches that can be readily and easily deployed by researchers to advance generalizability claims in a forward-looking manner.

Psychologists wish to identify and study the mechanisms and implications of nomothetic constructs that reveal truths about human nature and span across operationalizations. To achieve this goal, psychologists should spend more time carefully describing and measuring constructs across a wide range of methods and measures, and less time rushing to explain and predict.

The fact that causal laws in the social sciences are most realistically expressed as both multivariate and stochastic has a number of very important implications for indirect measurement and generalizability. It becomes difficult to link theoretical definitions of general constructs in a one-to-one relationship to research operations, with the result that there is conceptual slippage in both experimental and nonexperimental research. It is argued that problems of this nature can be approached by developing specific multivariate causal models that incorporate (...) sources of measurement bias, along with the theoretical variables of interest. Many general concepts are defined in such a way that causal assumptions are built into the definitions themselves. Additionally, in any given piece of research it is necessary to omit many variables from consideration, and this is often done without careful consideration of the assumptions required to justify such omissions. Finally, generalization to more inclusive populations or a diversity of settings ordinarily requires one to replace constants by variables. It is concluded that the criteria of parsimony, generalizability, and precision are incompatible, given the multivariate nature of social causation, and the author expresses his own preference for sacrificing parsimony in favor of the objectives of achieving increased precision and generalizability of social science laws. (shrink)

This article provides an overview of Edmund Husserl’s lesser known account of high-level imaginative empathy. The author discusses Husserl’s solution to what we might call the ‘generalizability problem’; if empathy is conceived as a relation whereby the understanding I have of my own mind allows me to understand your mind, then how does empathy account for potential differences between us? The author also discusses some features that make empathy more generalizable than might be initially thought, as well as its (...) limits. A second major aim is to use this exegesis of Husserl to show a variety of overlaps between his theory and high-level simulation theory. The author also shows how Husserl’s phenomenological theory provides a compelling response to critiques of high-level simulation from authors that utilize a hybrid cognitive science/phenomenological approach. (shrink)

I draw lessons from experimental economics. I argue that the lack of mathematical formalism cannot be usefully thought as the cause of the underappreciation of contextual and generalizability considerations. Instead, this lack is problematic because it hinders a clear relationship between theory and quantitative predictions. I also advocate a pragmatic policy-focused approach as a partial remedy to the generalizability problem.

Artificial intelligence (AI) shares many generalizability challenges with psychology. But the fields publish differently. AI publishes fast, through rapid preprint sharing and conference publications. Psychology publishes more slowly, but creates integrative reviews and meta-analyses. We discuss the complementary advantages of each strategy, and suggest that incorporating both types of strategies could lead to more generalizable research in both fields.

Improving generalization in psychology will require more expansive data collection to fuel more expansive statistical models, beyond the scale of traditional lab research. We argue that citizen science is uniquely positioned to scale up data collection and, that in spite of certain limitations, can help to alleviate the generalizability crisis.

The 38 commentaries on the target article span a broad range of disciplines and perspectives. I have organized my response to the commentaries around three broad questions: First, how serious are the problems discussed in the target article? Second, are there are other, potentially more productive, ways to think about the issues that the target article framed in terms of generalizability? And third, what, if anything, should we collectively do about these problems?

The fact that causal laws in the social sciences are most realistically expressed as both multivariate and stochastic has a number of very important implications for indirect measurement and generalizability. It becomes difficult to link theoretical definitions of general constructs in a one-to-one relationship to research operations, with the result that there is conceptual slippage in both experimental and nonexperimental research. It is argued that problems of this nature can be approached by developing specific multivariate causal models that incorporate (...) sources of measurement bias, along with the theoretical variables of interest. Many general concepts are defined in such a way that causal assumptions are built into the definitions themselves. Additionally, in any given piece of research it is necessary to omit many variables from consideration, and this is often done without careful consideration of the assumptions required to justify such omissions. Finally, generalization to more inclusive populations or a diversity of settings ordinarily requires one to replace "constants" by variables. It is concluded that the criteria of parsimony, generalizability, and precision are incompatible, given the multivariate nature of social causation, and the author expresses his own preference for sacrificing parsimony in favor of the objectives of achieving increased precision and generalizability of social science laws. (shrink)

For decades, psychological research has heavily favored quantitative over qualitative methods. One reason for this imbalance is the perception that quantitative methods follow from a post-positivist paradigm, which guides mainstream psychology, whereas qualitative methods follow from a constructivist paradigm. However, methods and paradigms are independent, and embracing qualitative methods within mainstream psychology is one way of addressing the generalizability crisis.

Overgeneralizations by psychologists of the research evidence on memory and eyewitness testimony, such as “memory decays with time” or “memories are fluid and malleable,” are beginning to appear in legal judgements and guidance documents, accompanied by unwarranted disparagement of lay beliefs about memory. These overgeneralizations could have significant adverse consequences for the conduct of civil and criminal law.

Yarkoni argues that researchers making broad inferences often use impoverished statistical models that fail to include important sources of variation as random effects. We argue, however, that for many common study designs, random effects are inappropriate and insufficient to draw general inferences, as the source of variation is not random, but systematic.

ObjectiveEfforts to understand why some marriages thrive while others falter are not well integrated conceptually and rely heavily on data collected from White middle-class samples. The Vulnerability-Stress-Adaptation Model is used here to integrate prior efforts and is tested using data collected from couples living with low incomes.BackgroundThe VSA Model assumes that enduring vulnerabilities, stress, and couple communication account for unique variance in relationship satisfaction, that communication mediates the effects of vulnerabilities and stress on satisfaction, and that the predictors of satisfaction (...) generalize across socioeconomic levels. To date, these assumptions remain untested.Materials and methodsWith 388 couples from diverse backgrounds, we used latent variable structural equation models to examine enduring vulnerabilities, chronic stress, and observed communication as predictors of 4-wave, 27-month satisfaction trajectories, first as main effects and then interacting with a validated 10-item index of sociodemographic risk.Results The three variable sets independently predict satisfaction trajectories; couple communication does not mediate the effects of enduring vulnerabilities or stress on satisfaction; and in 19% of tests, effects were stronger among couples with higher sociodemographic risk.ConclusionEffects of established predictor domains on satisfaction replicate in a diverse sample of newlywed couples, and most findings generalize across levels of sociodemographic risk. The failure of couple communication to mediate effects of enduring personal vulnerabilities and stress raises new questions about how these two domains undermine committed partnerships. (shrink)

Volume 20, Issue 2, February 2020, Page 27-30.