Experimental design is one aspect of a scientific method. A well-designed, properly conducted experiment aims to control variables in order to isolate and manipulate causal effects and thereby maximize internal validity, support causal inferences, and guarantee reliable results. Traditionally employed in the natural sciences, experimental design has become an important part of research in the social and behavioral sciences. Experimental methods are also endorsed as the most reliable guides to policy effectiveness. Through a discussion of some of (...) the central concepts associated with experimental design, including controlled variation and randomization, this chapter will provide a summary of key ethical issues that tend to arise in experimental contexts. In addition, by exploring assumptions about the nature of causation and by analyzing features of causal relationships, systems, and inferences in social contexts, this chapter will summarize the ways in which experimental design can undermine the integrity of not only social and behavioral research but policies implemented on the basis of such research. (shrink)

This, the third edition of Fundamentals of Experimental Design, has five added chapters - those on regression (Chapters 12, 14, and 15), multivariate analysis (Chapter 18), and the matrix algebra appropriate to the level of presentation of this material (Chapter 13). I have noted in the preface other additions in this third edition. The added material should enhance the value of the book as a textbook and a reference. Given these additions, however, alternative approaches in using the current edition (...) as a textbook may merit consideration. It may help to note that Chapters 16 and 17 (analysis of covariance, trend analysis) do not depend on the material in Chapters 12 through 15, although the student should know something about simple linear regression to be able to understand fully the material in Chapters 16 and 17. In any event, the instructor who wants to teach only the material in the first two editions can do so by dropping the added chapters - 12 through 15, and 18 - from the syllabus. (shrink)

I review three answers to the question: How can phenomenology contribute to the experimental cognitive neurosciences? The first approach, neurophenomenology, employs phenomenological method and training, and uses first-person reports not just as more data for analysis, but to generate descriptive categories that are intersubjectively and scientifically validated, and are then used to interpret results that correlate with objective measurements of behaviour and brain activity. A second approach, indirect phenomenology, is shown to be problematic in a number of ways. Indirect (...) phenomenology is generally put to work after the experiment, in critical or creative interpretations of the scientific evidence. Ultimately, however, proposals for the indirect use of phenomenology lead back to methodological questions about the direct use of phenomenology in experimental design. The third approach, 'front-loaded' phenomenology, suggests that the results of phenomenological investigations can be used in the design of empirical ones. Concepts or clarifications that have been worked out phenomenologically may operate as a partial framework for experimentation. (shrink)

Sections include: experiments and generalised causal inference; statistical conclusion validity and internal validity; construct validity and external validity; quasi-experimental designs that either lack a control group or lack pretest observations on the outcome; quasi-experimental designs that use both control groups and pretests; quasi-experiments: interrupted time-series designs; regresssion discontinuity designs; randomised experiments: rationale, designs, and conditions conducive to doing them; practical problems 1: ethics, participation recruitment and random assignment; practical problems 2: treatment implementation and (...) attrition; generalised causal inference: a grounded theory; generalised causal inference: methods for single studies; generalised causal inference: methods for multiple studies; a critical assessment of our assumptions. (shrink)

We applaud the authors' basic message. We note that the negative research emphasis is not special solely to social psychology and judgment and decision-making. We argue that the proposed integration of null hypothesis significance testing (NHST) and Bayesian analysis is promising but will ultimately succeed only if more attention is paid to proper experimental design and implementation.

In many areas of the social and behavioral sciences, the nature of the experiments and theories that best capture the underlying constructs are themselves areas of active inquiry. Integrative experiment design risks being prematurely exploitative, hindering exploration of experimental paradigms and of diverse theoretical accounts for target phenomena.

To what extent does the design of statistical experiments, in particular sequential trials, affect their interpretation? Should postexperimental decisions depend on the observed data alone, or should they account for the used stopping rule? Bayesians and frequentists are apparently deadlocked in their controversy over these questions. To resolve the deadlock, I suggest a three‐part strategy that combines conceptual, methodological, and decision‐theoretic arguments. This approach maintains the pre‐experimental relevance of experimental design and stopping rules but vindicates their evidential, postexperimental (...) irrelevance. †To contact the author, please write to: Tilburg Center for Logic and Philosophy of Science, Tilburg University, P.O. Box 90153, 5000 LE Tilburg, The Netherlands; e‐mail: [email protected]. (shrink)

Peter Spirtes, Clark Glymour, Richard Scheines, Christopher Meek, S. Fineberg, E. Slate. Prediction and Experimental Design with Graphical Causal Models.

This paper considers ways that experimental design can affect judgments about informally presented context shifting experiments. Reasons are given to think that judgments about informal context shifting experiments are affected by an exclusive reliance on binary truth value judgments and by experimenter bias. Exclusive reliance on binary truth value judgments may produce experimental artifacts by obscuring important differences of degree between the phenomena being investigated. Experimenter bias is an effect generated when, for example, experimenters disclose (even unconsciously) their (...) own beliefs about the outcome of an experiment. Eliminating experimenter bias from context shifting experiments makes it far less obvious what the “intuitive” responses to those experiments are. After it is shown how those different kinds of bias can affect judgments about informal context shifting experiments, those experiments are revised to control for those forms of bias. The upshot of these investigations is that participants in the contextualist debate who employ informal experiments should pay just as much attention to the design of their experiments as those who employ more formal experimental techniques if they want to avoid obscuring the phenomena they aim to uncover. (shrink)

Computational modeling plays an important role in prediction and optimization of real systems and processes. Models usually have some parameters which should be set up to the proper value. Therefore, parameter estimation is known as an important part of the modeling and system identification. It usually refers to the process of using sampled data to estimate the optimum values of parameters. The accuracy of model can be increased by adjusting its parameters to the optimum value which need a richer dataset. (...) One simple solution for having a richer dataset is increasing the amount of data, but that can be costly and time consuming. When using data from animals or people, it is especially important to have a proper plan. There are several available methods for parameter estimation in dynamical systems; however there are some basic differences in chaotic systems due to their sensitivity to initial condition. Accordingly, in this paper, a new cost function which is proper for chaotic systems is applied to the chaotic one-dimensional map. Then the efficiency of a newly introduced intelligent method experimental design in extracting proper data is investigated. The results show the success of the proposed method. (shrink)

During the twentieth century statistical methods have transformed research in the experimental and social sciences. Qualitative evidence has largely been replaced by quantitative results and the tools of statistical inference have helped foster a new ideal of objectivity in scientific knowledge. The paper will investigate this transformation by considering the genesis of analysis of variance and experimental design, statistical methods nowadays taught in every elementary course of statistics for the experimental and social sciences. These methods were developed (...) by the mathematician and geneticist R. A. Fisher during the 1920s, while he was working at Rothamsted Experimental Station, where agricultural research was in turn reshaped by Fisher’s methods. Analysis of variance and experimental design required new practices and instruments in field and laboratory research, and imposed a redistribution of expertise among statisticians, experimental scientists and the farm staff. On the other hand the use of statistical methods in agricultural science called for a systematization of information management and made computing an activity integral to the experimental research done at Rothamsted, permanently integrating the statisticians’ tools and expertise into the station research programme. Fisher’s statistical methods did not remain confined within agricultural research and by the end of the 1950s they had come to stay in psychology, sociology, education, chemistry, medicine, engineering, economics, quality control, just to mention a few of the disciplines which adopted them. (shrink)

Context: The integration of data measured in first- and third-person frameworks is a challenge that becomes more prominent as we attempt to refine the ties between the dimensions we assume to be objective and our experience itself. As a result, cognitive science has been a target for criticism from the epistemological and methodological point of view, which has resulted in the emergence of new approaches. Neurophenomenology has been proposed as a means to address these limitations. The methodological application of this (...) discipline, even in its mildest form, enriches the methodology typically used in cognitive sciences. Problem: Nowadays psychological studies are difficult to replicate. As a way to achieve replication of results published in a previous study in order to develop a methodological adaptation suitable for electroencephalographic measurements in a subsequent experiment, first-person accounts from the participants in our pilot study were included in the experiment construction. This study’s objective is to show the benefit of including a mild-neurophenomenology-inspired approach in the adaptation from an original paradigm, which requires, foremost, the ability to replicate the original results. Method: Interviews with open and semi-structured questions were carried out at the end of an Approach-Avoidance Task (AAT. The first-person reports, together with the behavioral outcomes of each pilot, were taken into account for the development of the next piloting phase until replication of the original results was achieved, and the final experimental design was elaborated. Results: A sequence of four pilots, where the integration of third- and first-person information derived from subjects’ behavior and reported experiences while carrying them out rendered the behavioral replication we sought to achieve, providing support for a first-person enriched cognitive science paradigm. Implications: Including first-person accounts systematically during the development and performance of classic cognitive paradigms ensures that those paradigms are measuring what they claim to measure. This is the next logical step to improve replication rates, to refine the explanation of the results and avoid confounding third-person data interpretation. Constructivist content: Including first-person experiences and acknowledging the active role that participants’ experiences regarding the paradigm had in the modeling of its final version is in concordance with a constructivist standing. (shrink)

Asking and answering questions is the cornerstone of science yet formal training in understanding this key process is often overlooked. "Asking Questions in Biology" unpacks this crucial process of enquiry, from a biological perspective, at its various stages. It begins with an overview of scientific question-asking in general, before moving on to demonstrate how to derive hypotheses from unstructured observations. It then explains in the main sections of the book, how to use statistical tests as tools to analyse data and (...) answer those questions before, finally, showing the best practice in presenting scientific reports. As such, it is an indispensable companion to all students of biology, but particularly those enrolled in courses concerning experimental design; data analysis; hypothesis testing; research methods; or any practical project work. The late Chris Barnard was Professor of Animal Behaviour at Nottingham University. Francis Gilbert is Associate Professor of Ecology at Nottingham University. Peter McGregor isa Reader in Applied Zoologyat Cornwall College. ". (shrink)

The Hooks et al. review of microbiota-gut-brain literature provides a constructive criticism of the general approaches encompassing MGB research. This commentary extends their review by: highlighting capabilities of advanced systems-biology “-omics” techniques for microbiome research and recommending that combining these high-resolution techniques with intervention-based experimental design may be the path forward for future MGB research.

Almaatouq et al. propose an “integrative approach” to increase the generalisability and commensurability of experiments. Yet their metascientific approach has one glaring omission (and misinterpretation of) – the role of sample diversity in generalisability. In this commentary, we challenge false notions of subsumed duality between contexts, population, and diversity, and propose modifications to their design space to accommodate sample diversity.

Standard practices in experimental economics arise for different reasons. The “no deception” rule comes from a cost-benefit tradeoff; other practices have to do with the uses to which economists put experiments. Because experiments are part of scientific conversations that mostly go on within disciplines, differences in standard practices between disciplines are likely to persist.

A “stopping rule” in a sequential experiment is a rule or procedure for deciding when that experiment should end. Accordingly, the “stopping rule principle” states that, in a sequential experiment, the evidential relationship between the final data and an hypothesis under consideration does not depend on the experiment’s stopping rule: the same data should yield the same evidence, regardless of which stopping rule was used. In this essay, I reconstruct and rebut five independent arguments for the SRP. Reminding oneself that (...) the stopping rule is a part of an experiment’s design and is no more mysterious than many other design aspects helps elucidate why some of these arguments for the SRP are unsound. (shrink)

Almaatouq et al.'s prescription for more integrative experimental designs is welcome but does not address an equally important problem: Lack of adequate theories. We highlight two features theories ought to satisfy: “Well-specified” and “grounded.” We discuss the importance of these features, some positive exemplars, and the complementarity between the target article's prescriptions and improved theorizing.

Recently, there has been an increased interest in developing statistical methodologies for analyzing single case experimental design data to supplement visual analysis. Some of these are simulation-driven such as Bayesian methods because Bayesian methods can compensate for small sample sizes, which is a main challenge of SCEDs. Two simulation-driven approaches: Bayesian unknown change-point model and simulation modeling analysis were compared in the present study for three real datasets that exhibit “clear” immediacy, “unclear” immediacy, and delayed effects. Although SMA estimates (...) can be used to answer some aspects of functional relationship between the independent and the outcome variables, they cannot address immediacy or provide an effect size estimate that considers autocorrelation as required by the What Works Clearinghouse Standards. BUCP overcomes these drawbacks of SMA. In final analysis, it is recommended that both visual and statistical analyses be conducted for a thorough analysis of SCEDs. (shrink)

Stroke continues to be a leading cause of disability. Basic neurorehabilitation research is necessary to inform the neuropathophysiology of impaired motor control, and to develop targeted interventions with potential to remediate disability post-stroke. Despite knowledge gained from basic research studies, the effectiveness of research-based interventions for reducing motor impairment has been no greater than standard of practice interventions. In this perspective, we offer suggestions for overcoming translational barriers integral to experimental design, to augment traditional protocols, and re-route the rehabilitation (...) trajectory toward recovery and away from compensation. First, we suggest that researchers consider modifying task practice schedules to focus on key aspects of movement quality, while minimizing the appearance of compensatory behaviors. Second, we suggest that researchers supplement primary outcome measures with secondary measures that capture emerging maladaptive compensations at other segments or joints. Third, we offer suggestions about how to maximize participant engagement, self-direction, and motivation, by embedding the task into a meaningful context, a strategy more likely to enable goal-action coupling, associated with improved neuro-motor control and learning. Finally, we remind the reader that motor impairment post-stroke is a multidimensional problem that involves central and peripheral sensorimotor systems, likely influenced by chronicity of stroke. Thus, stroke chronicity should be given special consideration for both participant recruitment and subsequent data analyses. We hope that future research endeavors will consider these suggestions in the design of the next generation of intervention studies in neurorehabilitation, to improve translation of research advances to improved participation and quality of life for stroke survivors. (shrink)

Accumulating research has identified the interactive effects of catechol-O-methyltransferase gene Val158Met polymorphism and environmental factors on aggression. However, available evidence was mainly based upon correlational design, which yields mixed findings concerning who are more affected by environmental conditions and has been challenged for the low power of analyses on gene–environment interaction. Drawing on a mixed design, we scrutinized how COMT Val158Met polymorphism impacts on aggression, assessed by hostility, aggressive motivation, and aggressive behavior, under different social conditions in a sample of (...) 70 Chinese male undergraduate students. We found that both Val/Val homozygote and Met alleles carriers showed differences in the feelings of hostility and aggressive motivation under conditions of exclusion versus inclusion, but these differences were more pronounced for Met allele carriers. These findings implied that COMT Val158Met polymorphism did not respond to environmental stimuli in an all-or-none way and shed light on the importance of examining the gene–environment interaction using a mixed design. (shrink)

This argument map presents Paul Loewi’s crucial experiment in which he showed that neural transmissions of signals are chemical in nature, not electrical, in form of an argumentation. The map can be used in science education to show how the formulation of hypotheses should be related to a corresponding determination of experimental designs.

Much of the early history of developmental and physiological genetics in Germany remains to be written. Together with Carl Correns and Richard Goldschmidt, Alfred Kühn occupies a special place in this history. Trained as a zoologist in Freiburg im Breisgau, he set out to integrate physiology, development and genetics in a particular experimental system based on the flour moth Ephestia kühniella Zeller. This paper is meant to reconstruct the crucial steps in the experimental pathway that led Kühn and (...) his collaborators at the University of Göttingen, and later at the Kaiser Wilhelm Institutes of Biology and Biochemistry in Berlin, to formulate, in their specific way, what later became known as the "one gene-one enzyme hypothesis." Special attention will be given to the interaction of the different parts of Kühn's Ephestia-based project, which were rooted in different research traditions. The paper retraces how, roughly between 1925 and 1945, these elements came to form a mixed experimental setup composed of genetic, embryological, physiological and, finally, biochemical constituents. Accordingly, emphasis is laid on the development of the terminology in which the results were cast, and how it reflected the hybrid state of an experimental system successively acquiring new epistemic layers. (shrink)

As etiologic concepts of diseases gradually changed from humoral to microbial in the 19th century, syphilis presented particularly great challenges to doctors and scientists. Was “syphilis” merely a synonym for “venereal disease,” and did all manifestations attributed to it have the same cause? The discovery in 1879 of the gonococcus by Albert L. Neisser , and of the cause of chancroid, or soft chancre, in 1890 by Augusto Ducrey , established that venereal disease and syphilis were not synonymous, but syphilis (...) continued to mystify investigators. Symptoms could disappear for years and then—in most but not all persons—reappear with entirely different “tertiary” manifestations. .. (shrink)

Can you think of a meme that made you laugh or a political post that made you angry? These examples illustrate how social media use (SMU) impacts how people feel. Similarly, how people feel when they initiate SMU may impact the emotional effects of SMU. Someone feeling happy may feel more positively during SMU, whereas someone feeling sad may feel more negatively. Using an experimental design, we examined whether following SMU, those in a happy mood would experience increases in (...) positive affect (PA) and those in a sad mood would experience increases in negative affect (NA). A large sample of college students (N = 703) were randomly assigned to a happy, sad, or neutral mood induction before SMU. PA and NA were assessed at baseline, post-mood induction, and after SMU. Contrary to hypotheses, after SMU, people in happy moods experienced decreases in PA, and those in sad moods experienced decreases in NA, reflecting SMU having a dampening effect on emotions. PA and NA were significantly lower after SMU compared to baseline and did not vary by condition. How young adults feel when they log onto SMU matters in understanding how SMU impacts PA and NA, but on average, emotional experiences are dampened. (shrink)

PurposeThe purpose of this study was to introduce an orthogonal experimental design to improve the efficiency of building and optimizing models for freezing of gait prediction.MethodsA random forest model was developed to predict FOG by using acceleration signals and angular velocity signals to recognize possible precursor signs of FOG. An OED was introduced to optimize the feature extraction parameters.ResultsThe main effects and interaction among the feature extraction hyperparameters were analyzed. The false-positive rate, hit rate, and mean prediction time were (...) 27%, 68%, and 2.99 s, respectively.ConclusionThe OED was an effective method for analyzing the main effects and interactions among the feature extraction parameters. It was also beneficial for optimizing the feature extraction parameters of the FOG prediction model. (shrink)

Methodological practices differ between economics and psychology because economists use game theory as the basis for the design and interpretation of experiments, while psychologists do not. This methodological choice explains the “four key variables” stressed by Hert-wig and Ortmann. Game theory is currently the most rigorous basis for modeling strategic choice.

Research on expert‐novice differences has mainly focused on how experts solve familiar problems. We know far less about the skills and knowledge used by experts when they are confronted with novel problems within their area of expertise. This article discusses a study in which verbal protocols were taken from subjects of various expertise designing an experiment in an area with which they were unfamiliar. The results showed that even when domain knowledge is lacking, experts solve a novel problem within their (...) area of expertise by dividing the problem into a number of subproblems that are solved in a specified order. The lack of domain knowledge is compensated for by using abstract knowledge structures and domain‐specific heuristic strategies. However, the quality of their solutions is considerably lower than the quality attained by experts who were familiar with the type of problem to be solved. The results suggest that when experts are confronted with novel problems as compared with familiar problems, their form of reasoning remains intact, but the content of their reasoning suffers due to lack of domain knowledge. (shrink)

In this first of two companion papers to a logico-mathematic, structural methodology , a meta-level analysis of the non metric structure is presented in relation to critiques based on standard experimental, statistical, and computational methods of contemporary psychology and cognitive science. The concept of a non metric methodology is examined as it relates to the epistemological and scientific goals of experimental, statistical, and computational methods. While sharing in these goals, differences and similarities between the two methodological approaches are (...) outlined. It is argued that typical experimental methods are not sufficient to extract and validate semantic information in verbal narratives. It is further suggested that a logico-mathematic, structural methodology can yield invariant law-like cognitive processes by careful methodological control of the specific case — instead of those found by current methods that produce “laws” based on statistical frequency. Lastly, the issue of experimental manipulation in relation to the logico-mathematic, structural methodology is examined. (shrink)