The overreliance on verbal models and theories in psychology has been criticized for hindering the development of reliable research programs (Harris, 1976; Yarkoni, 2020). We demonstrate how the conceptual space framework can be used to formalize verbal theories and improve their precision and testability. In the framework, scientific concepts are represented by means of geometric objects. As a case study, we present a formalization of an existing three-dimensional theory of emotion which was developed with a spatial metaphor in mind. Wundt (...) posits that the range of human emotion can be represented along three axes of basic emotions (pleasure/displeasure, excitement/inhibition, tension/relaxation), just as color vision can be represented using three basic colors. We use dimensions to represent basic emotions, points to represent emotional states, and regions to represent broader emotional concepts. We then compare our formalization to an existing structuralist formalization of Wundt’s theory. Further, we discuss the empirical predictions that our formalization generates, such as comparisons of similarity and intensity. We conclude by demonstrating how the tools developed in the conceptual space framework can be used to formulate a theory of emotion based on empirical observation. (shrink)

The overreliance on verbal models and theories in psychology has been criticized for hindering the development of reliable research programs (Harris, 1976; Yarkoni, 2020). We demonstrate how the conceptual space framework can be used to formalize verbal theories and improve their precision and testability. In the framework, scientific concepts are represented by means of geometric objects. As a case study, we present a formalization of an existing three-dimensional theory of emotion which was developed with a spatial metaphor in mind. Wundt (...) posits that the range of human emotion can be represented along three axes of basic emotions (pleasure/displeasure, excitement/inhibition, tension/relaxation), just as color vision can be represented using three basic colors. We use dimensions to represent basic emotions, points to represent emotional states, and regions to represent broader emotional concepts. We then compare our formalization to an existing structuralist formalization of Wundt’s theory. Further, we discuss the empirical predictions that our formalization generates, such as comparisons of similarity and intensity. We conclude by demonstrating how the tools developed in the conceptual space framework can be used to formulate a theory of emotion based on empirical observation. (shrink)

Social cognition differs from general cognition in its focus on understanding, perceiving, and interpreting social information. However, we argue that the significance of domain‐general processes for controlling cognition has been historically undervalued in social cognition and social neuroscience research. We suggest much of social cognition can be characterized as specialized feature representations supported by domain‐general cognitive control systems. To test this proposal, we develop a comprehensive working model, based on an interactive activation and competition architecture and applied to the control (...) of action. As such, we label the model “ICON” (interactive activation and competition model for the control of action). We used the ICON model to simulate human performance across various laboratory tasks. Our simulations emphasize that many laboratory‐based social tasks do not require socially specific control systems, such as those that are argued to rely on neural networks associated with theory‐of‐mind. Moreover, our model clarifies that perceived disruptions in social cognition, even in what appears to be disruption to the control of social cognition, can stem from deficits in social representation instead. We advocate for a “default stance” in social cognition, where control is usually general, but representation is specific. This study underscores the importance of integrating social cognition within the broader realm of domain‐general control processing, offering a unified perspective on task processing. (shrink)

Scientists often respond to failures to replicate by citing differences between the experimental components of an original study and those of its attempted replication. In this paper, we investigate these purported mismatch explanations. We assess a body of failures to replicate in neuroscience studies on spinal cord injury. We argue that a defensible mismatch explanation is one where a mismatch of components is a difference maker for a mismatch of outcomes, and the components are relevantly different in the follow-up study, (...) given the scope of the original study. With this account, we argue that not all differences between studies are meaningful, even if they are difference makers. As our examples show, focusing only on these differences results in disregarding the representativeness of the original experiment’s components and the scope of its outcomes, undercutting other epistemic aims, such as translation, in the process. (shrink)

Behavior genetics often emphasizes methods over the underlying quality of the psychological information to which the methods are applied. A core aspect of this quality is the demographic diversity of the samples. Building causal genetic models based only on European-ancestry samples compromises their generalizability. Behavior genetics researchers must spend additional time and resources diversifying their samples before emphasizing causation.

This paper argues that some of the discussion around meta-scientific issues can be viewed as an argument over different “meta-hypotheses” – assumptions made about how different hypotheses in a scientific literature relate to each other. I argue that, currently, such meta-hypotheses are typically left unstated except in methodological papers and that the consequence of this practice is that it is hard to determine what can be learned from a direct replication study. I argue in favor of a procedure dubbed the (...) “limited homogeneity assumption” – assuming very little heterogeneity of effect sizes when a literature is initiated but switching to an assumption of heterogeneity once an initial finding has been successfully replicated in a direct replication study. Until that has happened, we do not allow the literature to proceed to a mature stage. This procedure will elevate the scientific status of direct replication studies in science. Following this procedure, a well-designed direct replication study is a means of falsifying an overall claim in an early phase of a literature and thus sets up a hurdle against the canonization of false facts in the behavioral sciences. (shrink)

The results of forensic science are believed to be reliable, and are widely used in support of verdicts around the world. However, due to the lack of suitable empirical studies, we actually know very little about the reliability of such results. In this paper, I argue that phenomena analogous to the main culprits for the replication crisis in psychology are also present in forensic science. Therefore forensic results are significantly less reliable than is commonly believed. I conclude that in order (...) to obtain reliable estimates for the reliability of forensic results, we need to conduct studies analogous to the large-scale replication projects in psychology. Additionally, I point to some ways for improving the reliability of forensic science, inspired by the reforms proposed in response to the replicability crisis. (shrink)

This study investigates the role of sensory processing sensitivity as a predictor of employees’ proactive work behavior. SPS is a multidimensional concept that depicts differences in people’s sensory awareness, processing, and reactivity to internal and external influences. Based on research on SPS as grounded in a heightened sensitivity of the behavioral inhibition and activation systems, it was argued that the relationships with task proactivity and personal initiative as indicators of proactive work behavior differ for the three SPS dimensions. Furthermore, based (...) on the person–environment fit perspective, SPS was assumed to moderate the relationship between employees’ job complexity and proactivity. The hypotheses were tested in two two-wave studies. Across both studies, ease of excitation was unrelated to proactivity. Low sensory threshold was negatively related to personal initiative, only in Study 2, but it did not predict task proactivity. Meanwhile, aesthetic sensitivity was positively related to proactivity, but in Study 2, this relationship could only be established for personal initiative. Moreover, job complexity was positively related to proactivity for those employees high but not for those low in AES. EOE and LST did not act as moderators. This study offers evidence of positive behavioral implications among highly sensitive persons when dealing with job complexity. Overall, the study presents an interesting point of departure for the role of SPS in employee proactivity that calls for more research. (shrink)

The rise of behavioral economics since the 1980s led to richer mutual influence between economic and psychological theory and experimentation. However, as behavioral economics has become increasingly integrated into the main stream in economics, and as psychology has remained damagingly methodologically conservative, this convergence has recently gone into reverse. At the same time, growing appreciation among economists of the limitations of atomistic individualism, along with advantages in econometric modeling flexibility by comparison with psychometrics, is leading economists to become more pluralistic (...) than psychologists about the ontology of behavioral causation and structures. This, combined with economists’ growing interest in network models, is drawing economists closer in theory and practice to sociologists who use quantitative or mixed methods. (shrink)

Interactions with social robots are symbolic experiences guided by the pretense that robots depict real people. But they can also be natural experiences that are direct, automatic, and independent of any thoughtful mapping between what is real and depicted. Both experiences are important, both may apply within the same interaction, and they may vary within a person over time.

Many scientists routinely generalize from study samples to larger populations. It is commonly assumed that this cognitive process of scientific induction is a voluntary inference in which researchers assess the generalizability of their data and then draw conclusions accordingly. We challenge this view and argue for a novel account. The account describes scientific induction as involving by default a generalization bias that operates automatically and frequently leads researchers to unintentionally generalize their findings without sufficient evidence. The result is unwarranted, overgeneralized (...) conclusions. We support this account of scientific induction by integrating a range of disparate findings from across the cognitive sciences that have until now not been connected to research on the nature of scientific induction. The view that scientific induction involves by default a generalization bias calls for a revision of the current thinking about scientific induction and highlights an overlooked cause of the replication crisis in the sciences. Commonly proposed interventions to tackle scientific overgeneralizations that may feed into this crisis need to be supplemented with cognitive debiasing strategies against generalization bias to most effectively improve science. (shrink)

In experimental cognitive psychology, objects of inquiry are typically operationalized with psychological tasks. When interpreting results from such tasks, we focus primarily on behavioral measures such as reaction times and accuracy rather than experiences – i.e., phenomenology – associated with the task, and posit that the tasks elicit the desired cognitive phenomenon. Evaluating whether the tasks indeed elicit the desired phenomenon can be facilitated by understanding the experience during task performance. In this paper we explore the breadth of experiences that (...) are elicited by and accompany task performance using in-depth phenomenological and qualitative methodology to gather subjective reports during the performance of a visuo-spatial change detection task. Thirty-one participants were asked to remember either colors, orientations or positions of the presented stimuli and recall them after a short delay. Qualitative reports revealed rich experiential landscapes associated with the task-performance, suggesting a distinction between two broad classes of experience: phenomena at the front of consciousness and background feelings. The former includes cognitive strategies and aspects of metacognition, whereas the latter include more difficult-to-detect aspects of experience that comprise the overall sense of experience. We focus primarily on the background feelings, since strategies of task-performance to a large extent map onto previously identified cognitive processes and discuss the methodological implications of our findings. (shrink)

Generalizations strengthen in traditional sciences, but in psychology (and social and behavioral sciences, more generally) they decay. This is usually viewed as a problem requiring solution. It could be viewed instead as a law-like phenomenon. Generalization decay cannot be squelched because human behavior is metastable and all behavioral data collected thus far have resulted from a thin sliver of human time.

We received 23 spirited commentaries on our target article from across the disciplines of philosophy, economics, evolutionary genetics, molecular biology, criminology, epidemiology, and law. We organize our reply around three overarching questions: (1) What is a cause? (2) How are randomized controlled trials (RCTs) and within-family genome-wide association studies (GWASs) alike and unalike? (3) Is behavior genetics a qualitatively different enterprise? Throughout our discussion of these questions, we advocate for the idea that behavior genetics shares many of the same pitfalls (...) and promises as environmentally oriented research, medical genetics, and other arenas of the social and behavioral sciences. (shrink)

The target article argues researchers should be more ambitious, designing studies that systematically and comprehensively explore the space of possible experiments in one fell swoop. We argue that while “systematic” is rarely achievable, “comprehensive” is often enough. Critically, the recent popularization of massive online experiments shows that comprehensive studies are achievable for most cognitive and behavioral research questions.

The integrative experiment design proposal currently only relates to group results, but downplays individual differences between participants, which may nevertheless be substantial enough to constitute a relevant dimension in the design space. Excluding the individual participant in the integrative design will not solve all problems mentioned in the target article, because averaging results may obscure the underlying mechanisms.

While the term “reactivity” has come to be associated with specific phenomena in the social sciences, having to do with subjects’ awareness of being studied, this paper takes a broader stance on this concept. I argue that reactivity is a ubiquitous feature of the psychological subject matter and that this fact is a precondition of experimental research, while also posing potential problems for the experimenter. The latter are connected to the worry about distorted data and experimental artifacts. But what are (...) experimental artifacts and what is the most productive way of dealing with them? In this paper, I approach these questions by exploring the ways in which experimenters in psychology simultaneously exploit and suppress the reactivity of their subject matter in order to produce experimental data that speak to the question or subject matter at hand. Highlighting the artificiality of experimental data, I raise the question of what distinguishes a genuine experimental result from an experimental artifact. My analysis construes experimental results as the outcomes of inferences from the data that take material background assumptions as auxiliary premises. Artifacts occur when one or more of these background assumptions are false, such that the data do not reliably serve the purposes they were generated for. I conclude by laying out the ways in which my analysis of data quality is relevant to, and informed by, recent debates about the replicability of experimental results. (shrink)

Affective theory of mind (aToM) impairments associated with the youth antisocial phenotype callous-unemotional (CU) traits predict antisocial behaviour above CU traits alone. Importantly, CU traits associate with decrements in complex but not basic aToM. aToM is modulated by cognitive control and CU traits associate with cognitive control impairments; thus, cognitive control is a plausible mechanism underlying aToM impairments in CU traits. Because cognitive control is dependent on the availability of cognitive resources, youth with CU traits may have difficulty with allocating (...) cognitive resources under greater demands that impact complex aToM. To test this, 81 participants (ages 12–14, Female = 51.8%, Male = 48.2%) were recruited to complete a behavioural paradigm that involved an initial aToM task with complex and basic emotions followed by placing additional demands on cognitive control and a final repeat of the same aToM task. Results indicate adolescents higher in CU traits had intact basic aToM but less accuracy in complex aToM that worsened after taxing cognitive control; and this load only required a short duration to account for ToM decrements (200 ms [range 150–1600 ms]). These results demonstrate CU traits association with cognitive control limitations that impact complex aToM. This may partially explain antisocial behaviour associated with CU traits. (shrink)

Computational models of child language development can help us understand the cognitive underpinnings of the language learning process, which occurs along several linguistic levels at once (e.g., prosodic and phonological). However, in light of the replication crisis, modelers face the challenge of selecting representative and consolidated infant data. Thus, it is desirable to have evaluation methodologies that could account for robust empirical reference data, across multiple infant capabilities. Moreover, there is a need for practices that can compare developmental trajectories of (...) infants to those of models as a function of language experience and development. The present study aims to take concrete steps to address these needs by introducing the concept of comparing models with large-scale cumulative empirical data from infants, as quantified by meta-analyses conducted across a large number of individual behavioral studies. We formalize the connection between measurable model and human behavior, and then present a conceptual framework for meta-analytic evaluation of computational models. We exemplify the meta-analytic model evaluation approach with two modeling experiments on infant-directed speech preference and native/non-native vowel discrimination. (shrink)

Nowadays, almost everyone seems to agree that science is facing an epistemological crisis – namely the replicability crisis – and that we need to take action. But as to precisely what to do or how...

The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is (...) at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. Researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm – and with far greater efficiency. (shrink)

In light of the recent credibility crisis in psychology, this paper argues for a greater emphasis on theorizing in scientific research. Although reliable experimental evidence, preregistration, methodological rigor, and new computational frameworks for modeling are important, scientific progress also relies on properly functioning theories. However, the current understanding of the role of theorizing in psychology is lacking, which may lead to future crises. Theories should not be viewed as mere speculations or simple inductive generalizations. To address this issue, the author (...) introduces a framework called “cognitive metascience,” which studies the processes and results of evaluating scientific practice. This study should proceed both qualitatively, as in traditional science and technology studies and cognitive science, and quantitatively, by analyzing scientific discourse using language technology. By analyzing theories as cognitive artifacts that support cognitive tasks, this paper aims to shed more light on their nature. This perspective reveals that multiple distinct theories serve entirely different roles, and studying these roles, along with their epistemic vices and virtues, can provide insight into how theorizing should proceed. The author urges a change in research culture to appreciate the variety of distinct theories and to systematically advance scientific progress. . (shrink)

The replicability of a research claim is often positioned as an important step in establishing the credibility of scientific research. This expectation persists despite ongoing disagreements over how to characterise replication practices in various contexts. Rather than attempt to explain or resolve these disagreements, we propose that there is value in exploring the variable uses of the replicability concept. To this end, we treat the replicability concept as a goal-directed tool for studying scientific practices. This approach extends scholarship on the (...) goal-directed uses of concepts within investigative practices to explore the value of reflexively interrogating those concepts used to study the sciences as practiced. In doing so, we highlight the importance of considering how and when a given concept is an appropriate tool for a given goal when studying scientific practices. For instance, in the case of the replicability concept, our examination suggests that there is value in clarifying how a given characterisation of replicability is appropriate to a given context-specific analytic and/or evaluative goal. In addition, we hope to draw attention to how Metaresearch and HPS contexts provide complementary insights for those seeking to understand when and how replication practices can help in assessing a given research claim. (shrink)