This target article is concerned with the implications of the surprisingly different experimental practices in economics and in areas of psychology relevant to both economists and psychologists, such as behavioral decision making. We consider four features of experimentation in economics, namely, script enactment, repeated trials, performance-based monetary payments, and the proscription against deception, and compare them to experimental practices in psychology, primarily in the area of behavioral decision making. Whereas economists bring a precisely defined “script” to experiments for participants to (...) enact, psychologists often do not provide such a script, leaving participants to infer what choices the situation affords. By often using repeated experimental trials, economists allow participants to learn about the task and the environment; psychologists typically do not. Economists generally pay participants on the basis of clearly defined performance criteria; psychologists usually pay a flat fee or grant a fixed amount of course credit. Economists virtually never deceive participants; psychologists, especially in some areas of inquiry, often do. We argue that experimental standards in economics are regulatory in that they allow for little variation between the experimental practices of individual researchers. The experimental standards in psychology, by contrast, are comparatively laissez-faire. We believe that the wider range of experimental practices in psychology reflects a lack of procedural regularity that may contribute to the variability of empirical findings in the research fields under consideration. We conclude with a call for more research on the consequences of methodological preferences, such as the use on monetary payments, and propose a “do-it-both-ways” rule regarding the enactment of scripts, repetition of trials, and performance-based monetary payments. We also argue, on pragmatic grounds, that the default practice should be not to deceive participants. Key Words: behavioral decision making; cognitive illusions; deception; experimental design; experimental economics; experimental practices; financial incentives; learning; role playing. (shrink)

In psychology, deception is commonly used to increase experimental control. Yet, its use has provoked concerns that it raises participants' suspicions, prompts second-guessing of experimenters' true intentions, and ultimately distorts behavior and endangers the control it is meant to achieve. Over time, these concerns regarding the methodological costs of the use of deception have been subjected to empirical analysis. We review the evidence stemming from these studies.

The following is a set of reading notes on, and questions for, the Neuroeconomics enterprise. My reading of neuroscience evidence seems to be at odds with basic conceptions routinely assumed in the Neuroeconomics literature. I also summarize methodological concerns regarding design, implementation, and statistical evaluation of Neuroeconomics experiments.

This response reinforces the major themes of our target article. The impact of key methodological variables should not be taken for granted. Rather, we suggest grounding experimental practices in empirical evidence. If no evidence is available, decisions about design and implementation ought to be subjected to systematic experimentation. In other words, we argue against empirically blind conventions and against methodological choices based on beliefs, habits, or rituals. Our approach will neither inhibit methodological diversity nor constrain experimental creativity. More likely, it (...) will promote both goals. (shrink)

Over the past decades psychological theories have made significant headway into economics, culminating in the 2002 (partially) and 2017 Nobel prizes awarded for work in the field of Behavioral Economics. Many of the insights imported from psychology into economics share a common trait: the presumption that decision makers use shortcuts that lead to deviations from rational behaviour (the Heuristics-and-Biases program). Many economists seem unaware that this viewpoint has long been contested in cognitive psychology. Proponents of an alternative program (the Ecological-Rationality (...) program) argue that heuristics need not be irrational, particularly when judged relative to characteristics of the environment. We sketch out the historical context of the antagonism between these two research programs and then review more recent work in the Ecological-Rationality tradition. While the heuristics-and-biases program is now well-established in (mainstream neo-classical) economics via Behavioral Economics, we show there is considerable scope for the Ecological-Rationality program to interact with economics. In fact, we argue that there are many existing, yet overlooked, bridges between the two, based on independently derived research in economics that can be construed as being aligned with the tradition of the Ecological-Rationality program. We close the paper with a discussion of the open challenges and difficulties of integrating the Ecological Rationality program with economics. (shrink)

This response reinforces the major themes of our target article. The impact of key methodological variables should not be taken for granted. Rather, we suggest grounding experimental practices in empirical evidence. If no evidence is available, decisions about design and implementation ought to be subjected to systematic experimentation. In other words, we argue against empirically blind conventions and against methodological choices based on beliefs, habits, or rituals. Our approach will neither inhibit methodological diversity nor constrain experimental creativity. More likely, it (...) will promote both goals. (shrink)

My main argument is that the advice offered to experimental psychologists by Hertwig & Ortmann overlooks fundamental differences between the goals of researchers in psychology and economics. Furthermore, it is argued that the reduction of data variability is not always an end to be sought by psychologists. Variability that originates in individual differences constitutes valuable data for psychological research.

The use of high hypothetical payoffs has been justified by the realism and relevance of large monetary consequences and by the impracticality of making high cash payments. We argue that subjects may not be able to imagine how they would behave in high payoff situations.

The target article suggests that many practices of experimental economists are preferable to those used by psychologists studying judgment and decision making. The advantages of the psychological approach become clear when the focus of research shifts from choice output to choice processes. I illustrate this point with an example from research on similarity comparisons.

Experimental psychologists in the learning tradition stress the importance of three of the authors' four key variables of experimental design. We review research investigating the roles played by these variables in studies of choice from our laboratory. Supporting the authors' claims, these studies show that the effects of these variables are not fixed and should not be taken for granted.

Hertwig and Ortmann suggest paying participants contingent upon performance in order to increase the thoroughness they devote to a decision task. We argue that monetary incentives can yield a number of unintended effects including distortions of the subjective representation of the task and impaired performance. Therefore, we conclude that performance-contingent payment should not be generally employed in judgment and decision research.

For a variety of reasons, including the common use of deception in psychology experiments, participants often disbelieve experimenters' assertions about important task parameters. This can lead researchers to conclude incorrectly that participants are behaving non- normatively. The problem can be overcome by deriving and testing normative models that do not assume full belief in key task parameters. A real experimental example is discussed.

Hertwig and Ortmann suggest greater standardization of procedures in experimental psychology to help with problems of replicability and consistency of findings. It is argued that, this view is inconsistent with their other interesting proposals, and heterogeneity of method is appropriate in psychology.

Features of experimental design impose auxiliary hypotheses on experimenters. Hertwig & Ortmann rightly argue that the ways some variables are implemented in psychology cloud results, whereas the different implementations in economics provide for more robust results. However, not all design variables support this general conclusion. The repetition of trials may confuse results depending on what theory is being tested. We explore this in the case of simple bargaining games.

We applaud the authors' basic message. We note that the negative research emphasis is not special solely to social psychology and judgment and decision-making. We argue that the proposed integration of null hypothesis significance testing (NHST) and Bayesian analysis is promising but will ultimately succeed only if more attention is paid to proper experimental design and implementation.