In recent debates about the replication crisis, two positions have been dominant: one that focuses on methodological reforms and one that focuses on theory building. This paper takes up the suggestion that there might be a deeper difference in play, concerning the ways the very subject matter of psychology is construed by opposing camps, i.e., in terms of stable effects versus in terms of complexity. I argue that each gets something right, but neither is sufficient. My analysis suggests (...) that the context sensitivity of the psychological subject matter needs to be front and center of methodological and theoretical efforts. (shrink)

Popper’s (1983, 2002) philosophy of science has enjoyed something of a renaissance in the wake of the replication crisis, offering a philosophical basis for the ensuing science reform movement. However, adherence to Popper’s approach may also be at least partly responsible for the sense of “crisis” that has developed following multiple unexpected replication failures. In this article, I contrast Popper’s approach with Lakatos’ (1978) approach and a related approach called naïve methodological falsificationism (NMF; Lakatos, 1978). The (...) Popperian approach is powerful because it is based on logical refutation, but its theories are noncausal and, therefore, lacking in scientific value. In contrast, the Lakatosian approach considers causal theories, but it concedes that these theories are not logically refutable. Finally, the NMF approach subjects Lakatosian causal theories to Popperian logical refutations. However, its approach of temporarily accepting a ceteris paribus clause during theory testing may be viewed as scientifically inappropriate, epistemically inconsistent, and “completely redundant” (Lakatos, 1978, p. 40). I conclude that the replication “crisis” makes the most sense in the context of the Popperian and NMF approaches because it is only in these two approaches that replication failures represent logical refutations of theories. In contrast, replication failures are less problematic in the Lakatosian approach because they do not logically refute theories. Indeed, in the Lakatosian approach, replication failures can be legitimately ignored or used to motivate theory development. (shrink)

Nowadays, almost everyone seems to agree that science is facing an epistemological crisis – namely the replicability crisis – and that we need to take action. But as to precisely what to do or how...

The replication crisis is perceived by many as one of the most significant threats to the reliability of research. Though reporting of the crisis has emphasized social science, all signs indicate that it extends to many other fields. This paper investigates the possibility that the crisis and related challenges to conducting research also extend to philosophy. According to one possibility, philosophy inherits a crisis similar to the one in science because philosophers rely on unreplicated or (...) unreplicable findings from science when conducting philosophical research. According to another possibility, the crisis likely extends to philosophy because philosophers engage in similar research practices and face similar structural issues when conducting research that have been implicated by the crisis in science. Proposals for improving philosophical research are offered in light of these possibilities. (shrink)

This paper introduces the concept of ‘hyper-ambition’ in academia as a contributing factor to what has been termed a ‘replication crisis’ across some sciences. The replication crisis is an umbrella term that covers a range of ‘questionable research practices’, from sloppy reporting to fraud. There are already many proposals to address questionable research practices, some of which focus on the values, norms, and motivations of researchers and institutes, and suggest measures to promote research integrity. Yet it (...) is not easy to promote integrity in hyper-competitive academic environments that value high levels of ambition. I argue that in such contexts, it is as likely that a kind of hyper-ambition is fostered that (inadvertently or otherwise) prioritises individual success above all, including to the detriment of scientific quality. In addition, efforts to promote values like integrity falter because they rely on sufficient uniformity in motivations or tendencies. Codes and guidance promoting integrity are, however, likely to influence those for whom such values are not optional, while others simply find ways around them. To demonstrate this I offer a thought experiment in which we consider the imaginary working situations of two ordinary academics. I conclude that tackling questionable research practices in the light of the replication crisis requires robust ‘top down’ measures that expect and accommodate a broader range of academic values, motivations, and tendencies, while challenging those that help to promote hyper-ambition. (shrink)

A growing body of cross-cultural survey research shows high percentages of clinicians report using placebos in clinical settings. One motivation for clinicians using placebos is to help patients by capitalising on the placebo effect’s reported health benefits. This is not surprising, given that placebo studies are burgeoning, with increasing calls by researchers to ethically harness placebo effects among patients. These calls propose placebos/placebo effects offer clinically significant benefits to patients. In this paper, we argue many findings in this highly cited (...) and ‘hot’ field have not been independently replicated. Evaluating the ethicality of placebo use in clinical practice involves first understanding whether placebos are efficacious clinically. Therefore, it is crucial to consider placebo research in the context of the replication crisis and what can be learnt to advance evidence-based knowledge of placebos/placebo effects and their clinical relevance (or lack thereof). In doing so, our goal in this paper is to motivate both increased awareness of replication issues and to help pave the way for advances in scientific research in the field of placebo studies to better inform ethical evidence-based practice. We argue that, only by developing a rigorous evidence base can we better understand how, if at all, placebos/placebo effects can be harnessed ethically in clinical settings. (shrink)

The replication (replicability, reproducibility) crisis in social psychology and clinical medicine arises from the fact that many apparently well-confirmed experimental results are subsequently overturned by studies that aim to replicate the original study. The culprit is widely held to be poor science: questionable research practices, failure to publish negative results, bad incentives, and even fraud. In this article I argue that the high rate of failed replications is consistent with high-quality science. We would expect this outcome if the (...) field of science in question produces a high proportion of false hypotheses prior to testing. If most of the hypotheses under test are false, then there will be many false hypotheses that are apparently supported by the outcomes of well conducted experiments and null hypothesis significance tests with a type-I error rate (α) of 5%. Failure to recognize this is to commit the fallacy of ignoring the base rate. I argue that this is a plausible diagnosis of the replication crisis and examine what lessons we thereby learn for the future conduct of science. (shrink)

Is science in the midst of a crisis of replicability and false discoveries? In a recent article, Alexander Bird offers an explanation for the apparent lack of replicability in the biomedical sciences. Bird argues that the surprise at the failure to replicate biomedical research is a result of the fallacy of neglecting the base rate. The base-rate fallacy arises in situations in which one ignores the base rate—or prior probability—of an event when assessing the probability of this event in (...) the light of some observed evidence. By extension, the replication crisis would result from ignoring the low prior probability of biomedical hypotheses. In this paper, my response to Bird’s claim is twofold. First, I show that the argument according to which the replication crisis is due to the low prior of biomedical hypotheses is incomplete. Second, I claim that a simple base-rate fallacy model does not account for some important methodological insights that have emerged in discussions of the replication crisis. (shrink)

The replication crisis has prompted many to call for statistical reform within the psychological sciences. Here we examine issues within Frequentist statistics that may have led to the replication crisis, and we examine the alternative—Bayesian statistics—that many have suggested as a replacement. The Frequentist approach and the Bayesian approach offer radically different perspectives on evidence and inference with the Frequentist approach prioritising error control and the Bayesian approach offering a formal method for quantifying the relative strength (...) of evidence for hypotheses. We suggest that rather than mere statistical reform, what is needed is a better understanding of the different modes of statistical inference and a better understanding of how statistical inference relates to scientific inference. (shrink)

Replicability is widely taken to ground the epistemic authority of science. However, in recent years, important published findings in the social, behavioral, and biomedical sciences have failed to replicate, suggesting that these fields are facing a “replicability crisis.” For philosophers, the crisis should not be taken as bad news but as an opportunity to do work on several fronts, including conceptual analysis, history and philosophy of science, research ethics, and social epistemology. This article introduces philosophers to these discussions. (...) First, I discuss precedents and evidence for the crisis. Second, I discuss methodological, statistical, and social-structural factors that have contributed to the crisis. Third, I focus on the philosophical issues raised by the crisis. Finally, I discuss proposed solutions and highlight the gaps that philosophers could focus on. (shrink)

ABSTRACTPossibly, the replication crisis constitutes the most important problem in psychology. It calls into question whether psychology is a science. Existing conceptualizations of replicability depend on effect sizes; the larger the population effect size, the greater the probability of replication. This is problematic and contributes to the replication crisis. A different conceptualization, not dependent on population effect sizes, is desirable. The proposed solution features the closeness of sample means to their corresponding population means, in both (...) the original and replication experiments. If the researcher has specified the sampling precision desired, it is possible to calculate the probability of replication, prior to data collection, and without dependence on the population effect size or expected population effect size. In addition, it is not necessary to know population means or standard deviations, nor sample means or standard deviations, to employ the proposed a priori way of thinking about... (shrink)

Replicability and reproducibility of computational models has been somewhat understudied by “the replication movement.” In this paper, we draw on methodological studies into the replicability of psychological experiments and on the mechanistic account of explanation to analyze the functions of model replications and model reproductions in computational neuroscience. We contend that model replicability, or independent researchers' ability to obtain the same output using original code and data, and model reproducibility, or independent researchers' ability to recreate a model without original (...) code, serve different functions and fail for different reasons. This means that measures designed to improve model replicability may not enhance (and, in some cases, may actually damage) model reproducibility. We claim that although both are undesirable, low model reproducibility poses more of a threat to long-term scientific progress than low model replicability. In our opinion, low model reproducibility stems mostly from authors' omitting to provide crucial information in scientific papers and we stress that sharing all computer code and data is not a solution. Reports of computational studies should remain selective and include all and only relevant bits of code. (shrink)

The crisis of replication has led many to blame statistical significance tests for making it too easy to find impressive looking effects that do not replicate. However, the very fact it becomes difficult to replicate effects when features of the tests are tied down actually serves to vindicate statistical significance tests. While statistical significance tests, used correctly, serve to bound the probabilities of erroneous interpretations of data, this error control is nullified by data-dredging, multiple testing, and other biasing (...) selection effects. Arguments claiming to vitiate statistical significance tests attack straw person variants of tests that commit well-known fallacies and misinterpretations. There is a tension between popular calls for preregistration – arguably, one of the most promising ways to boost replication – and accounts that downplay error probabilities: Bayes Factors, Bayesian posteriors, likelihood ratios. By underscoring the importance of error control for well testedness, the replication crisis points to reformulating tests so as to avoid fallacies and report the extent of discrepancies that are and are not indicated with severity. (shrink)

The ongoing debate about a “replication crisis” has put scientific failure in the spotlight, not only in psychological research and the social sciences but also in the life sciences. However, despite this increased salience of failure in research, the concept itself has so far received little attention in the literature (for an exception, see Ref. 1). The lack of a systematic perspective on scientific failure—a daily experience for researchers—hampers our understanding of this complex phenomenon and the development of (...) efficient policies and measures to address it. Without a better grasp of the multiple dimensions of scientific failure, there is a risk that necessary measures will be neglected or that inadequate policies will be adopted because different kinds of failures require different responses. Developing a basic taxonomy of scientific failure will help to identify connections between different types of failures and benefit the formulation of policy measures for improving replicability. (shrink)

Numerous philosophers of science have argued that the incentive structure of science is a major contributor to the replication crisis. In this article I review FiveThirtyEight’s election forecasting model and show that it confronts similar problems in political polling and successfully mitigates them, not by changing the incentives, but by weighting a pollster’s effect on the model by their past reliability. I argue that a similar weighting procedure holds a promising approach to addressing the replication crisis (...) in science. (shrink)

While it is widely acknowledged that psychology is in the throes of a replication ‘crisis’, relatively little attention has been paid to the role theory plays in our evaluation of replications as ‘failed’ or ‘successful’. This paper applies well-known arguments in philosophy of science about the interplay between theory and experiment to a contemporary case study of infants’ understanding of false belief (Onishi and Baillargeon [2005]), and attempts to replicate it. It argues that the lack of consensus about (...) over-arching theories informing both the concepts under study and the methodologies used to track them means that researchers disagree over which experiments constitute replications of the original. The second part of the paper places this specific debate within a broader discussion of the replication crisis as a crisis of ‘theory’, developing work by Muthukrishna and Henrich ([2018]) and Bird ([2018]). Bird argues that the lack of agreed over-arching theories in psychology means that a high rate of replication failure is to be expected; this paper agrees with his diagnosis but challenges his proposal that more replication will resolve the problem. (shrink)

Current debates about the replication crisis in psychology take it for granted that direct replication is valuable and focus their attention on questionable research practices in regard to statistical analyses. This paper takes a broader look at the notion of replication as such. It is argued that all experimentation/replication involves individuation judgments and that research in experimental psychology frequently turns on probing the adequacy of such judgments. In this vein, I highlight the ubiquity of conceptual (...) and material questions in research, and I argue that replication is not as central to psychological research as it is sometimes taken to be. (shrink)

Since the advent of the twenty-first century, science has experienced a crisis pertaining to the replicability of quantitative research findings, which has become known as the ‘replication crisis’. The replication crisis has particularly afflicted research in the behavioural sciences, and psychology in particular. Given the relevance of psychology to education, it is unsurprising that the replication crisis also presents an issue for quantitative educational research, thus potentially compromising its practical usefulness. This article outlines (...) the replication crisis in psychology, and highlights its significance for quantitative educational research. Greater methodological rigour, aimed at addressing methodological deficiencies in the conduct of scientific research, has been suggested as a response to the replication crisis. Following a review of various calls for greater methodological rigour, the current article argues that, whilst methodological deficiencies may be a contributory factor, there is potentially a more fundamental reason for the replication crisis. The case is made that the non-separability of a measured attribute from the measurement instrument, and the irreducible uncertainty in unmeasured attributes, may be the principal reasons for the replication crisis in psychology and education. Implications of such an explanation for the crisis are articulated in relation to the evidence-based policy agenda in education. (shrink)

The replication crisis has caused researchers to distinguish between exact replications, which duplicate all aspects of a study that could potentially affect the results, and direct replications, which duplicate only those aspects of the study that are thought to be theoretically essential to reproduce the original effect. The replication crisis has also prompted researchers to think more carefully about the possibility of making Type I errors when rejecting null hypotheses. In this context, the present article considers (...) the utility of two types of Type I error probability: the Neyman–Pearson long run Type I error rate and the Fisherian sample-specific Type I error probability. It is argued that the Neyman–Pearson Type I error rate is inapplicable in social science because it refers to a long run of exact replications, and social science deals with irreversible units that make exact replications impossible. Instead, the Fisherian sample-specific Type I error probability is recommended as a more meaningful way to conceptualize false positive results in social science because it can be applied to each sample-specific decision about rejecting the same substantive null hypothesis in a series of direct replications. It is concluded that the replication crisis may be partly due to researchers’ unrealistic expectations about replicability based on their consideration of the Neyman–Pearson Type I error rate across a long run of exact replications. (shrink)

The replicability crisis refers to the apparent failures to replicate both important and typical positive experimental claims in psychological science and biomedicine, failures which have gained increasing attention in the past decade. In order to provide evidence that there is a replicability crisis in the first place, scientists have developed various measures of replication that help quantify or “count” whether one study replicates another. In this nontechnical essay, I critically examine five types of replication measures used (...) in the landmark article “Estimating the reproducibility of psychological science” (Open Science Collaboration, Science, 349, ac4716, 2015) based on the following techniques: subjective assessment, null hypothesis significance testing, comparing effect sizes, comparing the original effect size with the replication confidence interval, and meta-analysis. The first four, I argue, remain unsatisfactory for a variety of conceptual or formal reasons, even taking into account various improvements. By contrast, at least one version of the meta-analytic measure does not suffer from these problems. It differs from the others in rejecting dichotomous conclusions, the assumption that one study replicates another or not simpliciter. I defend it from other recent criticisms, concluding however that it is not a panacea for all the multifarious problems that the crisis has highlighted. (shrink)

Why the replication crisis might be less severe than it seems at first.

A large number of scientists and several news platforms have, over the last few years, been speaking of a replication crisis in various academic disciplines, especially the biomedical and social sciences. This paper answers the novel question of whether we should also pursue replication in the humanities. First, I create more conceptual clarity by defining, in addition to the term “humanities,” various key terms in the debate on replication, such as “reproduction” and “replicability.” In doing so, (...) I pay attention to what is supposed to be the object of replication: certain studies, particular inferences, of specific results. After that, I spell out three reasons for thinking that replication in the humanities is not possible and argue that they are unconvincing. Subsequently, I give a more detailed case for thinking that replication in the humanities is possible. Finally, I explain why such replication in the humanities is not only possible, but also desirable. (shrink)

The replication crisis has spawned discussions on the meaning of replication. In fact, in order to determine whether an experiment fails to replicate, it is necessary to establish what replication is. This is, however, a difficult task, as it is possible to attribute different meanings to it. This paper offers a solution to this problem of ambiguity by engineering a concept of replication that, if compared to other proposals, stands out for being not only broadly (...) applicable but also sufficiently specific. It features a minimal level of operationalism, which would otherwise limit its applicability, while it heavily relies on replication’s specific epistemic functions, which are inter-disciplinary. Another merit is its context sensitivity, which enables it to differentiate instances of replication from non-instances of replication in every scientific discipline according to the discipline’s own standards. (shrink)

Discussions about a replicability crisis in science have been driven by the normative claim that all of science should be replicable and the empirical claim that most of it isn’t. Recently, such crisis talk has been challenged by a new localism, which argues a) that serious problems with replicability are not a general occurrence in science and b) that replicability itself should not be treated as a universal standard. The goal of this article is to introduce this emerging (...) strand of the debate and to discuss some of its implications and limitations. I will in particular highlight the issue of demarcation that localist accounts have to address, i.e. the question of how we can distinguish replicable science from disciplines where replicability does not apply. (shrink)

This special issue on what some regard as a crisis of replicability in cognitive science (i.e. the observation that a worryingly large proportion of experimental results across a number of areas cannot be reliably replicated) is informed by three recent developments. -/- First, philosophers of mind and cognitive science rely increasingly on empirical research, mainly in the psychological sciences, to back up their claims. This trend has been noticeable since the 1960s (see Knobe, 2015). This development has allowed philosophers (...) to draw on a wider range of relevant resources, but it also makes them vulnerable to relying on claims that may not survive further scrutiny. If we have reasons to believe that a large proportion of findings in the psychological sciences cannot be reliably replicated, this would be a problem for philosophers who use such findings in their work. Second, philosophers are increasingly designing and carrying out their own experiments to back up claims, or to test claims earlier made from the armchair, for example, on the perceived permissibility of diverting trolleys or on the nature of free will. This growing field of experimental philosophy has diversified the intellectual field in philosophy, but may also be vulnerable to issues of replicability that philosophers did not face before. -/- Third, the recent evidence of apparently widespread non-replicability in the social sciences (and other fields) has forced philosophers of science to grapple with long standing questions from their field from a new perspective. To what extent does replicability matter for theory construction? How do the notions of replicability and scientific progress interact? How can normative insights from philosophy of science be used in order to improve scientific practice? -/- It is with these three developments in mind—the increasing importance of empirically-informed philosophy, of experimental philosophy, and of philosophy of science around replicability—that this special issue has been conceived. (shrink)

Replicability is widely regarded as one of the defining features of science and its pursuit is one of the main postulates of meta-research, a discipline emerging in response to the replicability crisis. At the same time, replicability is typically treated with caution by philosophers of science. In this paper, we reassess the value of replicability from an epistemic perspective. We defend the orthodox view, according to which replications are always epistemically useful, against the more prudent view that claims that (...) it is useful in very limited circumstances. Additionally, we argue that we can learn more about the original experiment and the limits of the discovered effect from replications at different levels. We hold that replicability is a crucial feature of experimental results and scientists should continue to strive to secure it. (shrink)

The results of forensic science are believed to be reliable, and are widely used in support of verdicts around the world. However, due to the lack of suitable empirical studies, we actually know very little about the reliability of such results. In this paper, I argue that phenomena analogous to the main culprits for the replication crisis in psychology are also present in forensic science. Therefore forensic results are significantly less reliable than is commonly believed. I conclude that (...) in order to obtain reliable estimates for the reliability of forensic results, we need to conduct studies analogous to the large-scale replication projects in psychology. Additionally, I point to some ways for improving the reliability of forensic science, inspired by the reforms proposed in response to the replicability crisis. (shrink)

The life sciences are said to be in the midst of a replication crisis because a majority of published results are irreproducible, and scientists rarely replicate existing data. Here I argue that point 2 of this assessment is flawed because there is a hitherto unidentified form of replication in the experimental life sciences, which I call ‘microreplications’. Using a case study from biochemistry, I illustrate how MRs depend on a key element of experimentation, namely, experimental controls. I (...) end by reflecting on what MRs mean for the broader debate about the replication crisis. (shrink)

Replicability is usually considered to be one of the cornerstones of science; however, the growing recognition of nonreplicable experiments and studies in scientific journals—a phenomenon that has been called ‘replicability crisis’—has spurred a debate on the meaning, function, and significance of replicability in science. Amid this discussion, it has become clear that replicability is not a monolithic concept; what is still controversial is exactly how the distinction between different kinds of replicability should be laid out terminologically and conceptually, and (...) to what extent it bears on the more general debate on the centrality of replicability in science. This paper’s goals are to clarify the different uses of the terms related to replicability and, more importantly, to conceptually specify the kinds of replicability and their respective epistemic functions. (shrink)

Scientists are becoming increasingly aware of a “replicability crisis” in the behavioral, social, and biomedical sciences. Researchers have made progress identifying statistical and methodological...

The replication or reproducibility crisis in psychological science has renewed attention to philosophical aspects of its methodology. I provide herein a new, functional account of the role of replication in a scientific discipline: to undercut the underdetermination of scientific hypotheses from data, typically by hypotheses that connect data with phenomena. These include hypotheses that concern sampling error, experimental control, and operationalization. How a scientific hypothesis could be underdetermined in one of these ways depends on a scientific discipline’s (...) epistemic goals, theoretical development, material constraints, institutional context, and their interconnections. I illustrate how these apply to the case of psychological science. I then contrast this “bottom-up” account with “top-down” accounts, which assume that the role of replication in a particular science, such as psychology, must follow from a uniform role that it plays in science generally. Aside from avoiding unaddressed problems with top-down accounts, my bottom-up account also better explains the variability of importance of replication of various types across different scientific disciplines. (shrink)

This research presents the results of a survey regarding scientific misconduct and questionable research practices elicited from a sample of 1215 management researchers. We find that misconduct is not encountered often by reviewers nor editors. Yet, there is a strong prevalence of misrepresentations. When it comes to potential methodological improvements, those that are skeptical about the empirical body of work being published see merit in replication studies. Yet, a sizeable majority of editors and authors eschew open data policies, which (...) points to hidden costs and limited incentives for data sharing in management research. (shrink)

SummaryThis essay explains why and how nineteenth-century chemists sought to stabilize the melting and boiling points of organic substances as reliable characteristics of identity and purity and how, by the end of the century, they established these values as ‘Constants of Nature’. Melting and boiling points as characteristic values emerge from this study as products of laboratory standardization, developed by chemists in their struggle to classify, understand and control organic nature. A major argument here concerns the role played by the (...) introduction of organic synthesis in driving these changes. Synthetic organic chemistry vastly increased the number of known organic substances, precipitating the chemical identity crisis of my title. Successful natural product synthesis, moreover, depended on chemists’ ability to demonstrate the absolute identity of synthetic product and natural target – something late nineteenth-century chemists eventually achieved by making reliable, replicable melting and boiling point measurements. In the period before the establishment of national standards laboratories, chemists and scientific glassblowers worked together to standardize melting and boiling points as physical constants, such collaborations highlighting the essential importance of chemical glassware and glassblowing skill in the development of nineteenth-century organic chemistry. (shrink)

The Director-General of the WHO has suggested that China’s approach to the COVID-19 crisis could be the standard of care for global epidemics. However, as remarkable as the Chinese strategy might be, it cannot be replicated in other countries and certainly not in Europe. In Europe, there is a distribution of power between the European Union and its member states. In contrast, China’s political power is concentrated in the central government. This enables it to take immediate measures that affect (...) the entire country, such as massive quarantines or closing borders. Moreover, the Chinese legal framework includes restrictions on privacy and other human rights that are unknown in Europe. In addition, China has the technological power to easily impose such restrictions. In most European countries, that would be science fiction. These conditions have enabled China to combat epidemics like no other country can. However, the WHO might have been overoptimistic. The Chinese standard of care for treating COVID-19 also raises problematic issues for human rights, and the real consequences of these actions remain to be seen. (shrink)

Because of the misspecification of models and specificity of operationalizations, many studies produce claims of limited utility. We suggest a path forward that requires taking a few steps back. Researchers can retool large-scale replications to conduct the descriptive research which assesses the generalizability of constructs. Large-scale construct validation is feasible and a necessary next step in addressing the generalizability crisis.

The social sciences are facing numerous crises including those related to replication, theory, and applicability. We highlight that these crises imply epistemic malfunctions and affect science communication negatively. Several potential solutions have already been proposed, ranging from statistical improvements to changes in norms of scientific conduct. In this paper, we propose a structural solution: the elimination of the discussion section from social science research papers. We point out that discussion sections allow for an inappropriate narrativization of research that disguises (...) actual results and enables the misstatement of true limitations. We go on to claim that removing this section and outsourcing it to other publications provides several epistemic advantages such as a division of academic labour, adversarial modes of progress, and a better alignment of the personal aims of scientists with the aims of science. After responding to several objections, we conclude that the potential benefits of moving away from the traditional model of academic papers outweigh the costs and have the potential to play a part in addressing the crises in the social sciences alongside other reforms. As such, we take our paper as proffering a further potential solution that should be applied complimentarily with other reform movements such as Open Science and hope that our paper can start a debate on this or similar proposals. (shrink)

Most discussions of the reproducibility crisis focus on its epistemic aspect: the fact that the scientific community fails to follow some norms of scientific investigation, which leads to high rates of irreproducibility via a high rate of false positive findings. The purpose of this paper is to argue that there is a heretofore underappreciated and understudied dimension to the reproducibility crisis in experimental psychology and neuroscience that may prove to be at least as important as the epistemic dimension. (...) This is the communication dimension. The link between communication and reproducibility is immediate: independent investigators would not be able to recreate an experiment whose design or implementation were inadequately described. I exploit evidence of a replicability and reproducibility crisis in computational science, as well as research into quality of reporting to support the claim that a widespread failure to adhere to reporting standards, especially the norm of descriptive completeness, is an important contributing factor in the current reproducibility crisis in experimental psychology and neuroscience. (shrink)

Currently, many scientific fields such as psychology or biomedicine face a methodological crisis concerning the reproducibility, replicability, and validity of their research. In neuroimaging, similar methodological concerns have taken hold of the field, and researchers are working frantically toward finding solutions for the methodological problems specific to neuroimaging. This article examines some ethical and legal implications of this methodological crisis in neuroimaging. With respect to ethical challenges, the article discusses the impact of flawed methods in neuroimaging research in (...) cognitive and clinical neuroscience, particularly with respect to faulty brain-based models of human cognition, behavior, and personality. Specifically examined is whether such faulty models, when they are applied to neurological or psychiatric diseases, could put patients at risk, and whether this places special obligations on researchers using neuroimaging. In the legal domain, the actual use of neuroimaging as evidence in United States courtrooms is surveyed, followed by an examination of ways that the methodological problems may create challenges for the criminal justice system. Finally, the article reviews and promotes some promising ideas and initiatives from within the neuroimaging community for addressing the methodological problems. (shrink)