Commentaries on the target article offer diverse perspectives on integrative experiment design. Our responses engage three themes: (1) Disputes of our characterization of the problem, (2) skepticism toward our proposed solution, and (3) endorsement of the solution, with accompanying discussions of its implementation in existing work and its potential for other domains. Collectively, the commentaries enhance our confidence in the promise and viability of integrative experiment design, while highlighting important considerations about how it is used.

Creating an integrated design space can be successful only if researchers agree on how to define and measure a certain phenomenon of interest. Adversarial collaborations and mathematical modeling can aid in reaching the necessary level of agreement when researchers depart from different theoretical perspectives.

Glowacki argues that the human capacity for peace emerged 100,000 years ago, and that the logic of peace is such that the traits and technologies that enable peace are the same that are used to wage war. In my commentary I raise some concerns about these points as well as about Glowacki's understanding of peace.

Integrative experiments, as described, seem blindly empirical, as if the question of generality of effects could not be understood through controlled one-at-a-time experiments. But current research using such experiments, especially applied research, can resolve issues and make progress through understanding of cause–effect pathways, leaving to engineers the task of translating this understanding into practice.

Glowacki's work meshes well with our view of human nature as having evolved to use culture to improve survival and reproduction. Peace is a cultural achievement, requiring advances in social organization and control, including leaders who can implement policies to benefit the group, third-party mediation, and intergroup cooperation. Cultural advances shift intergroup interactions from negative-sum (war) to positive-sum (trade).

Economic games provide models of real-world contexts in which researchers can probe dispositional and structural determinants of intergroup relations. Most intergroup games focus on determinants of aggression between groups and constrain the possibilities for peace. However, paradigms such as the intergroup parochial and universal cooperation game allow for peaceful intergroup relations and can be adapted for the study of peace.

Glowacki offers many new directions for understanding and even eliminating the problem of war, especially creating positive interdependencies with out-group members. We develop Glowacki's intriguing proposition that in-group dynamics provide a route to peace by describing a prosocial motivational system, the caregiving system, that aligns individual interests and eliminates the need to use coercion to achieve mutually beneficial outcomes.

We propose a friendly amendment to integrative experiment design (IED), adversarial-collaboration IED, that incentivizes research teams from competing theoretical perspectives to identify zones of the design space where they possess an explanatory edge. This amendment is especially critical in debates that have high policy stakes and carry a strong normative-political charge that might otherwise prevent free exchange of ideas.

Glowacki explored the conditions required for peace and argued its preconditions arose only within the last 100,000 years. The present commentary addresses some major brain changes that occurred only in Homo sapiens within that period of time and the verbal and nonverbal cognitive sequelae of those neurological changes that may have aided the diplomatic negotiations required for peaceful solutions.

Demonstrating the limitations of the one-at-a-time approach, crowd initiatives reveal the surprisingly powerful role of analytic and design choices in shaping scientific results. At the same time, cross-cultural variability in effects is far below the levels initially expected. This highlights the value of “medium” science, leveraging diverse stimulus sets and extensive robustness checks to achieve integrative tests of competing theories.

Almaatouq et al. claim that the integrative experiment design can help “develop a reliable, cohesive, and cumulative theoretical understanding.” I will contest this claim by challenging three underlying assumptions about the nature of scientific theories. I propose that the integrative experiment design should be viewed as an exploratory framework rather than a means to build or evaluate theories.

Integrative experiment design should be extended to thought-experiments. Thought-experiments are closely connected to “real” experiments. They are involved in devising the design space of theories and possible experiments. The latter may be partitioned into experiments to be really performed and mere thought-experiments. The proposed extension of integrative experiment design lends guidance to a more methodical performance of thought-experiments.

In many areas of the social and behavioral sciences, the nature of the experiments and theories that best capture the underlying constructs are themselves areas of active inquiry. Integrative experiment design risks being prematurely exploitative, hindering exploration of experimental paradigms and of diverse theoretical accounts for target phenomena.

The authors rightly critique existing social sciences approaches. However, they are too quick to dismiss the criticism that their proposed paradigm is atheoretical. Social and cognitive theories are indeed incommensurate, often due to the lack of a unifying framework. Without proper integration with theoretical frameworks, their proposal may merely produce a resource-intensive veneer of thoroughness without substantive improvements to understanding.

Here we revise Glowacki's model by proposing a simple and empirically tested mechanism that is applicable to a comprehensive set of social interactions. This parsimonious mechanism accounts for the choice of both cooperative and peaceful alternatives and explains when each choice benefits the interacting parties. It is proposed that this mechanism is key to the evolution of both peace and conflict.

The capacities required for both peace and war predate 100,000 years ago in the genus Homo are deeply entangled in the modes by which humans physically and perceptually construct their worlds and communities, and may not be sufficiently captured by economic models.

Almaatouq et al. view the purpose of research is to map variable-to-variable relationships (e.g., the effect of X on Y). They also view theory as this mapping of variable-to-variable relationships rather than an explanation of why the relationships occur. However, it is theory as explanation that allows us to reconcile disparate findings and that should guide application.

Almaatouq et al. propose an “integrative approach” to increase the generalisability and commensurability of experiments. Yet their metascientific approach has one glaring omission (and misinterpretation of) – the role of sample diversity in generalisability. In this commentary, we challenge false notions of subsumed duality between contexts, population, and diversity, and propose modifications to their design space to accommodate sample diversity.

The integrative experiment design proposal currently only relates to group results, but downplays individual differences between participants, which may nevertheless be substantial enough to constitute a relevant dimension in the design space. Excluding the individual participant in the integrative design will not solve all problems mentioned in the target article, because averaging results may obscure the underlying mechanisms.

Almaatouq et al. argue that an “integrative experiment design” approach can help generating cumulative empirical and theoretical knowledge. Here, we discuss the novelty of their approach and scrutinize its promises and pitfalls. We argue that setting up a “design space” may turn out to be theoretically uninformative, inefficient, and even impossible. Designing truly diagnostic experiments provides a better alternative.

Glowacki asserts that interdependent relationships beyond group boundaries are exceptionally rare among nonhuman mammals. However, rudimentary forms of interdependence can be seen in primate species that form multilevel societies, that is, core social units embedded within higher-level grouping categories. Studies of primate multilevel societies can enrich discussions about the evolutionary origins of peaceful between-group interactions in humans.

We elaborate on Glowacki's claim that humans are more capable of establishing peace than other mammals. We present three aspects suggesting caution. First, the social capabilities of nonhuman primates should not be underestimated. Second, the effect of these capabilities on peace establishment is nonmonotonous. Third, defining peace by human-centered values introduces a fallacy.

Almaatouq et al. propose an integrative experiment design space combined with large samples for scientific advancement. We argue recent innovative designs combining closed-loop experiment designs and Bayesian optimisation allow for integrative experiments at an individual level during a single session, circumventing the necessity for large samples. This method can be applied across disciplines, including developmental and clinical research.

While effective institutional practices are critical for the evolution of peace certain factors deter their effectiveness. In-group and out-group dynamics may make peace difficult between culturally distinct groups. Critical ecological conditions often lead to intractable conflict over resources. And within group conflicts of interest most prominently between generations may inhibit effective peace making.

Almaatouq et al. propose a novel integrative approach to experiments. We provide three examples of how unaddressed measurement issues threaten the feasibility of the approach and its promise of promoting commensurability and knowledge integration.

Almaatouq et al.'s prescription for more integrative experimental designs is welcome but does not address an equally important problem: Lack of adequate theories. We highlight two features theories ought to satisfy: “Well-specified” and “grounded.” We discuss the importance of these features, some positive exemplars, and the complementarity between the target article's prescriptions and improved theorizing.

We disagree with Almaatouq et al. that no realistic alternative exists to the “one-at-a-time” paradigm. Seventy years ago, Egon Brunswik introduced representative design, which offers a clear path to commensurability and generality. Almaatouq et al.'s integrative design cannot guarantee the external validity and generalizability of results which is sorely needed, while representative design tackles the problem head on.

I challenge the idea by Glowacki that “strong sanctions” such as fines, physical punishment, or execution are more effective in promoting peace than “weak punishments” like social rejection. Reviewing evidence that social rejection can have significant social and psychological costs for norm violators, I propose that social rejection can serve as a powerful reputational sanction in fostering peace in society.

Integrative experimentation will improve on the status quo in empirical behavioral science. However, the results integrative experiments produce will remain conditional on the various assumptions used to produce them. Without a theory of interpretability, it remains unclear how viable it is to address the crud factor without sacrificing explainability.

The target article offers a game-theoretical analysis of primitive intergroup aggression (i.e., raiding) and discusses difficulties in achieving peace. We argue the analysis does not capture the actual strategy space, missing out “do-nothing.” Experimental evidence robustly shows people prefer doing nothing against out-group members over cooperating with/attacking them. Thus, the target article overestimates the likelihood of intergroup aggression.

In this commentary, we set out the specifics of how Glowacki's game theoretical framework for the evolution of peace could be incorporated within broader cultural evolutionary approaches. We outline a formal proposal for prisoner's dilemma games investigating raid-based conflict. We also centre an ethnographic lens to understand the norms surrounding war and peace in intergroup interactions in small-scale communities.

To succeed, we posit that research cartography will require high-throughput natural description to identify unknown unknowns in a particular design space. High-throughput natural description, the systematic collection and annotation of representative corpora of real-world stimuli, faces logistical challenges, but these can be overcome by solutions that are deployed in the later stages of integrative experiment design.

This commentary argues against the indictment of current experimental practices such as piecemeal testing, and the proposed integrated experiment design (IED) approach, which we see as yet another attempt at automating scientific thinking. We identify a number of undesirable features of IED that lead us to believe that its broad application will hinder scientific progress.

Integrative experiment design is a needed improvement over ad hoc experiments, but the specific proposed method has limitations. We urge a further break with tradition through the use of an enormous untapped resource: Decades of causal discovery artificial intelligence (AI) literature on optimizing the design of systematic experimentation.

The target article argues researchers should be more ambitious, designing studies that systematically and comprehensively explore the space of possible experiments in one fell swoop. We argue that while “systematic” is rarely achievable, “comprehensive” is often enough. Critically, the recent popularization of massive online experiments shows that comprehensive studies are achievable for most cognitive and behavioral research questions.

Glowacki discusses how humans regularly face collective action problems that may result in either peaceful or aggressive between-group interactions. Peace and war probably coevolved in humans. Using a gene–culture evolutionary framework is a powerful way to analyse why, when, and how humans have the capacity to build and maintain long-term peaceful interactions between groups and also to wage deadly wars.

Generalizations strengthen in traditional sciences, but in psychology (and social and behavioral sciences, more generally) they decay. This is usually viewed as a problem requiring solution. It could be viewed instead as a law-like phenomenon. Generalization decay cannot be squelched because human behavior is metastable and all behavioral data collected thus far have resulted from a thin sliver of human time.

We comment on the limits of relying on prior literature when constructing the design space for an integrative experiment; the adaptive nature of social and behavioral phenomena and the implications for the use of theory and modeling when constructing the design space; and on the challenges of measuring random errors and lab-related biases in measurement without replication.

Glowacki defines peace as harmonious relationships between groups maintained without the threat of violence, where groups can be anything from families to nation states. However, defining such contentious concepts like “peace” and “groups” is a difficult task, and we discuss the implications of Glowacki's definitions for understanding intergroup relationships and their evolutionary history.

We expect that consensus meetings, where researchers come together to discuss their theoretical viewpoints, prioritize the factors they agree are important to study, standardize their measures, and determine a smallest effect size of interest, will prove to be a more efficient solution to the lack of coordination and integration of claims in science than integrative experiments.

Peace is a hallmark of human societies. However, certain ant species engage in long-term intergroup resource sharing, which is remarkably similar to peace among human groups. We discuss how individual and group payoff distributions are affected by kinship, dispersal, and age structure; the challenges of diagnosing peace; and the benefits of comparing convergent complex behaviours in disparate taxa.

Dominant paradigms in science foster integration of research findings, but at what cost? Forcing convergence requires centralizing decision-making authority, and risks reducing the diversity of methods and contributors, both of which are essential for the breakthrough ideas that advance science.

Peace, the article shows, is achieved by culturally evolved institutions that incentivize positive-sum relationships. We propose that this insight has important consequences for the design of human social cognition. Cues that signal the existence of such institutions should play a prominent role in detecting group membership. We show how this accounts for previous findings and suggest avenues for future research.

The target article ignores the crucial role of correlational methods in the behavioral and social sciences. Yet such methods are often mandated by the greater complexity of the phenomena investigated. This necessity is especially conspicuous in psychological research where its position in the hierarchy of the sciences implies the need for both experimental and correlational investigations, each featuring distinct assets.

We affirm the utility of integrative modeling, according to which it is advantageous to move beyond “one-at-a-time binary paradigms” through studies that position themselves within realistic multidimensional design spaces. We extend the integrative modeling approach to a target domain with which we are familiar, the consequences of bilingualism on mind and brain, often referred to as the “bilingual advantage.” In doing so, we highlight work from our group consistent with integrative modeling.

Integrative experiment design promises to foster cumulative knowledge by changing how we design experiments, build theories, and conduct research. I support the push to increase commensurability across experimental research but raise several reservations regarding results-driven and large-team-based research. I argue that it is vital to preserve academic diversity and adversarial debate via independent efforts.

Integrative experiment design assumes that we can effectively design a space of factors that cause contextual variation. However, this is impossible to do so in a sufficiently objective way, resulting inevitably in observations laden with surrogate models. Consequently, integrative experiment design may even deepen the problem of incommensurability. In comparison, one-at-a-time approaches make much more tentative assumptions about the factors excluded from experiment design, hence still seem better suited to deal with incommensurability.

I provide a personal perspective on metastudies and emphasize lesser-known benefits. I stress the need for integrative theories to establish commensurability between experiments. I argue that mathematical social scientists should be engaged to develop integrative theories, and that likelihood functions provide a common mathematical framework across experiments. The development of quantitative theories promotes commensurability engineering on a larger scale.

This commentary analyzes the extent to which the incommensurability problem can be resolved through the proposed alternative method of integrative experiment design. We suggest that, although one aspect of incommensurability is successfully addressed (dimensional incommensurability), the proposed design space method does not yet alleviate another major source of discontinuity, which we call conceptual incommensurability.

The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is (...) at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. Researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm – and with far greater efficiency. (shrink)

While some species have affiliative and even cooperative interactions between individuals of different social groups, humans are alone in having durable, positive-sum, interdependent relationships across unrelated social groups. Our capacity to have harmonious relationships that cross group boundaries is an important aspect of our species' success, allowing for the exchange of ideas, materials, and ultimately enabling cumulative cultural evolution. Knowledge about the conditions required for peaceful intergroup relationships is critical for understanding the success of our species and building a more (...) peaceful world. How do humans create harmonious relationships across group boundaries and when did this capacity emerge in the human lineage? Answering these questions involves considering the costs and benefits of intergroup cooperation and aggression, for oneself, one's group, and one's neighbor. Taking a game theoretical perspective provides new insights into the difficulties of removing the threat of war and reveals an ironic logic to peace – the factors that enable peace also facilitate the increased scale and destructiveness of conflict. In what follows, I explore the conditions required for peace, why they are so difficult to achieve, and when we expect peace to have emerged in the human lineage. I argue that intergroup cooperation was an important component of human relationships and a selective force in our species history beginning at least 300 thousand years. But the preconditions for peace only emerged in the past 100 thousand years and likely coexisted with intermittent intergroup violence which would have also been an important and selective force in our species' history. (shrink)