We consider a confidence parametrization of binary information sources in terms of appropriate likelihood ratios. This parametrization is used for Bayesian belief updates and for the equivalent comparison of binary experiments. In contrast to the standard parametrization of a binary information source in terms of its specificity and its sensitivity, one of the two confidence parameters is sufficient for a Bayesian belief update conditional on a signal realization. We introduce a confidence-augmented receiver operating characteristic for comparisons of binary experiments for (...) a class of “balanced” decision problems, relative to which the confidence order offers a higher resolution than Blackwell’s informativeness order. Where observation is concerned, Chance favors only the prepared mind.—Louis Pasteur (1822–1895). (shrink)

Bayesianism and likelihoodism are two of the most important frameworks philosophers of science use to analyse scientific methodology. However, both frameworks face a serious objection: much scientific inquiry takes place in highly idealized frameworks where all the hypotheses are known to be false. Yet, both Bayesianism and likelihoodism seem to be based on the assumption that the goal of scientific inquiry is always truth rather than closeness to the truth. Here, I argue in favour of a verisimilitude framework for inductive (...) inference. In the verisimilitude framework, scientific inquiry is conceived of, in part, as a process where inference methods ought to be calibrated to appropriate measures of closeness to the truth. To illustrate the verisimilitude framework, I offer a reconstruction of parsimony evaluations of scientific theories, and I give a reconstruction and extended analysis of the use of parsimony inference in phylogenetics. By recasting phylogenetic inference in the verisimilitude framework, it becomes possible to both raise and address objections to phylogenetic methods that rely on parsimony. 1Introduction2Problems with the Law of Likelihood3Introducing Verisimilitude-Based Inference4Examples of Verisimilitude-Based Inference Procedures4.1Parsimony inference over theories4.2Parsimony inference in phylogenetics5Conclusion. (shrink)

The likelihood principle (LP) is a core issue in disagreements between Bayesian and frequentist statistical theories. Yet statements of the LP are often ambiguous, while arguments for why a Bayesian must accept it rely upon unexamined implicit premises. I distinguish two propositions associated with the LP, which I label LP1 and LP2. I maintain that there is a compelling Bayesian argument for LP1, based upon strict conditionalization, standard Bayesian decision theory, and a proposition I call the practical relevance principle. In (...) contrast, I argue that there is no similarly compelling argument for or against LP2. I suggest that these conclusions lead to a restrictedly pluralistic view of Bayesian confirmation measures. (shrink)

Disputes between advocates of Bayesians and more orthodox approaches to statistical inference presuppose that Bayesians must regard must regard stopping rules, which play an important role in orthodox statistical methods, as evidentially irrelevant.In this essay, I show that this is not the case and that the stopping rule is evidentially relevant given some Bayesian confirmation measures that have been seriously proposed. However, I show that accepting a confirmation measure of this sort comes at the cost of rejecting two useful ancillaryBayesian (...) principles. (shrink)

Bayesian model selection has frequently been the focus of philosophical inquiry (e.g., Forster, Br J Philos Sci 46:399–424, 1995; Bandyopadhyay and Boik, Philos Sci 66:S390–S402, 1999; Dowe et al., Br J Philos Sci 58:709–754, 2007). This paper argues that Bayesian model selection procedures are very diverse in their inferential target and their justification, and substantiates this claim by means of case studies on three selected procedures: MML, BIC and DIC. Hence, there is no tight link between Bayesian model selection and (...) Bayesian philosophy. Consequently, arguments for or against Bayesian reasoning based on properties of Bayesian model selection procedures should be treated with great caution. (shrink)

To what extent does the design of statistical experiments, in particular sequential trials, affect their interpretation? Should postexperimental decisions depend on the observed data alone, or should they account for the used stopping rule? Bayesians and frequentists are apparently deadlocked in their controversy over these questions. To resolve the deadlock, I suggest a three‐part strategy that combines conceptual, methodological, and decision‐theoretic arguments. This approach maintains the pre‐experimental relevance of experimental design and stopping rules but vindicates their evidential, postexperimental irrelevance. †To (...) contact the author, please write to: Tilburg Center for Logic and Philosophy of Science, Tilburg University, P.O. Box 90153, 5000 LE Tilburg, The Netherlands; e‐mail: [email protected]. (shrink)

Why are conditional degrees of belief in an observation E, given a statistical hypothesis H, aligned with the objective probabilities expressed by H? After showing that standard replies are not satisfactory, I develop a suppositional analysis of conditional degree of belief, transferring Ramsey’s classical proposal to statistical inference. The analysis saves the alignment, explains the role of chance-credence coordination, and rebuts the charge of arbitrary assessment of evidence in Bayesian inference. Finally, I explore the implications of this analysis for Bayesian (...) reasoning with idealized models in science. (shrink)

I I set out my view that all inference is essentially deductive and pinpoint what I take to be the major shortcomings of the induction rule.II The import of data depends on the probability model of the experiment, a dependence ignored by the induction rule. Inductivists admit background knowledge must be taken into account but never spell out how this is to be done. As I see it, that is the problem of induction.III The induction rule, far from providing a (...) method of discovery, does not even serve to detect pattern. Knowing that there is uniformity in the universe is no help to discovering laws. A critique of Reichenbach's justification of the straight rule is constructed along these lines.IV The induction rule, by itself, cannot account for the varying rates at which confidence in an hypothesis mounts with data. The mathematical analysis of this salient feature of inductive reasoning requires prior probabilities. We also argue, against orthodox statisticians, that prior probabilities make a substantive contribution to the objectivity of inductive methods, viz. to the design of experiments and the selection of decision rules.V Carnap's general criticisms of various estimation rules, like the straight rule and the ‘impervious rule’, are seen to be misguided when the prior densities to which they correspond are taken into account.VI Analysis of Hempel's definition of confirmation qua formalization of the enumerative (naive) conception of instancehood. We show that from the standpoint of the quantitative measure P(H/E):P(H) for the degree to which E confirms H, Hempel's classificatory concept yields correct results only for sampling at large from a finite population with a two-way classification all of whose compositions are equally probable. We extend the analysis to Goodman's paradox, finding cases in which grue-like hypotheses do receive as much confirmation as their opposite numbers. We argue, moreover, the irrelevancy of entrenchment, and maintain that Goodman's paradox is no more than a straightforward counter-example to the enumerative conception of instancehood embodied in Hempel's definition.VII We rebutt the objection that prior probabilities, qua inputs of Bayesian analysis, can only be obtained by enumerative induction (insofar as they are objective). The divergence in the prior densities of two rational agents is less a function of subjectivity, we maintain, than of vagueness.VIII Our concluding remarks stress that, for Bayesians, there is no problem of induction in the usual sense. (shrink)

This paper is an extended review of the book Error and Inference, edited by Deborah Mayo and Aris Spanos, about their frequentist and philosophical perspective on testing of hypothesis and on the criticisms of alternatives like the Bayesian approach.

To whatever extent the use of a behavioral, not an evidential, interpretation of decisions in the Lindley-Savage argument for Bayesian theory undermines its cogency as a criticism of typical standard practice, it also undermines the Neyman-Pearson theory as a support for typical standard practice. This leaves standard practice with far less theoretical support than Bayesian methods. It does nothing to resolve the anomalies and paradoxes of standard methods. (Similar statements apply to the common protestation that the models are not real (...) anyway.) The appropriate interpretation of tests as evidence, if possible at all, is difficult and counterintuitive. Any attempt to support tests as more than rules of thumb is doomed to failure. (shrink)

The history of statistics is filled with many controversies, in which the prime focus has been the difference in the “interpretation of probability” between Fre- quentist and Bayesian theories. Many philosophical arguments have been elabo- rated to examine the problems of both theories based on this dichotomized view of statistics, including the well-known stopping-rule problem and the catch-all hy- pothesis problem. However, there are also several “hybrid” approaches in theory, practice, and philosophical analysis. This poses many fundamental questions. This paper (...) reviews three cases and argues that the interpretation problem of probabil- ity is insufficient to begin a philosophical analysis of the current issues in the field of statistics. A novel viewpoint is proposed to examine the relationship between the stopping-rule problem and the catch-all hypothesis problem. (shrink)

The injunction to “analyze the way you randomize” is well known to statisticians since Fisher advocated for randomization as the basis of inference. Yet even those convinced by the merits of randomization-based inference seldom follow this injunction to the letter. Bernoulli randomized experiments are often analyzed as completely randomized experiments, and completely randomized experiments are analyzed as if they had been stratified; more generally, it is not uncommon to analyze an experiment as if it had been randomized differently. This article (...) examines the theoretical foundation behind this practice within a randomization-based framework. Specifically, we ask when is it legitimate to analyze an experiment randomized according to one design as if it had been randomized according to some other design. We show that a sufficient condition for this type of analysis to be valid is that the design used for analysis should be derived from the original design by an appropriate form of conditioning. We use our theory to justify certain existing methods, question others, and finally suggest new methodological insights such as conditioning on approximate covariate balance. (shrink)

I document some of the main evidence showing that E. S. Pearson rejected the key features of the behavioral-decision philosophy that became associated with the Neyman-Pearson Theory of statistics (NPT). I argue that NPT principles arose not out of behavioral aims, where the concern is solely with behaving correctly sufficiently often in some long run, but out of the epistemological aim of learning about causes of experimental results (e.g., distinguishing genuine from spurious effects). The view Pearson did hold gives a (...) deeper understanding of NPT tests than their typical formulation as accept-reject routines, against which criticisms of NPT are really directed. The Pearsonian view that emerges suggests how NPT tests may avoid these criticisms while still retaining what is central to these methods: the control of error probabilities. (shrink)

In this paper, I show how one might resist two influential arguments for the Likelihood Principle by appealing to the ontological significance of creative intentions. The first argument for the Likelihood Principle that I consider is the argument from intentions. After clarifying the argument, I show how the key premiss in the argument may be resisted by maintaining that creative intentions sometimes independently matter to what experiments exist. The second argument that I consider is Gandenberger’s :475–503, 2015) rehabilitation of Birnbaum’s (...) :269–306, 1962) proof of the Likelihood Principle from the more intuitively obvious principles of conditionality and sufficiency. As with the argument from intentions, I show how Gandenberger’s argument for his Experimental Conditionality Principle may be resisted by maintaining that creative intentions sometimes independently matter to what experiments exist. (shrink)

I present a new proof of the likelihood principle that avoids two responses to a well-known proof due to Birnbaum ([1962]). I also respond to arguments that Birnbaum’s proof is fallacious, which if correct could be adapted to this new proof. On the other hand, I urge caution in interpreting proofs of the likelihood principle as arguments against the use of frequentist statistical methods. 1 Introduction2 The New Proof3 How the New Proof Addresses Proposals to Restrict Birnbaum’s Premises4 A Response (...) to Arguments that the Proofs Are Fallacious5 Conclusion. (shrink)

The likelihood theory of evidence (LTE) says, roughly, that all the information relevant to the bearing of data on hypotheses (or models) is contained in the likelihoods. There exist counterexamples in which one can tell which of two hypotheses is true from the full data, but not from the likelihoods alone. These examples suggest that some forms of scientific reasoning, such as the consilience of inductions (Whewell, 1858. In Novum organon renovatum (Part II of the 3rd ed.). The philosophy of (...) the inductive sciences. London: Cass, 1967), cannot be represented within Bayesian and Likelihoodist philosophies of science. (shrink)

A “stopping rule” in a sequential experiment is a rule or procedure for deciding when that experiment should end. Accordingly, the “stopping rule principle” states that, in a sequential experiment, the evidential relationship between the final data and an hypothesis under consideration does not depend on the experiment’s stopping rule: the same data should yield the same evidence, regardless of which stopping rule was used. In this essay, I reconstruct and rebut five independent arguments for the SRP. Reminding oneself that (...) the stopping rule is a part of an experiment’s design and is no more mysterious than many other design aspects helps elucidate why some of these arguments for the SRP are unsound. (shrink)

The three main approaches in statistical inference—classical statistics, Bayesian and likelihood—are in current use in phylogeny research. The three approaches are discussed and compared, with particular emphasis on theoretical properties illustrated by simple thought-experiments. The methods are problematic on axiomatic grounds, extra-mathematical grounds relating to the use of a prior or practical grounds. This essay aims to increase understanding of these limits among those with an interest in phylogeny.

Elliott Sober is both an empiricist and an instrumentalist. His empiricism rests on a principle called actualism, whereas his instrumentalism violates this. This violation generates a tension in his work. We argue that Sober is committed to a conflicting methodological imperative because of this tension. Our argument illuminates the contemporary debate between realism and empiricism which is increasingly focused on the application of scientific inference to testing scientific theories. Sober’s position illustrates how the principle of actualism drives a wedge between (...) two conceptions of scientific inference and at the same time brings to the surface a deep conflict between empiricism and instrumentalism. (shrink)

The paper examines the claim that significance testing violates the Principle of Total Evidence. I argue that p-values violate PTE for two-sided tests but satisfy PTE for one-sided tests invoking a sufficient test statistic independent of the preferred theory of evidence. While the focus of the paper is to evaluate a particular claim about the relationship of significance testing and PTE, I clarify the reading of this methodological principle along the way.

The Dempster–Shafer approach to expressing beliefabout a parameter in a statistical model is notconsistent with the likelihood principle. Thisinconsistency has been recognized for some time, andmanifests itself as a non-commutativity, in which theorder of operations (combining belief, combininglikelihood) makes a difference. It is proposed herethat requiring the expression of belief to be committed to the model (and to certain of itssubmodels) makes likelihood inference very nearly aspecial case of the Dempster–Shafer theory.

The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as p-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view. I argue that questionable and other research practices do not usually inflate relevant Type I error rates. I begin (...) with an introduction to Type I error rates that distinguishes them from theoretical errors. I then illustrate my argument with respect to model misspecification, multiple testing, selective inference, forking paths, exploratory analyses, p-hacking, optional stopping, double dipping, and HARKing. In each case, I demonstrate that relevant Type I error rates are not usually inflated above their nominal level, and in the rare cases that they are, the inflation is easily identified and resolved. I conclude that the replication crisis may be explained, at least in part, by researchers’ misinterpretation of statistical errors and their underestimation of theoretical errors. (shrink)

My dissertation explores the ways in which Rudolf Carnap sought to make philosophy scientific by further developing recent interpretive efforts to explain Carnap’s mature philosophical work as a form of engineering. It does this by looking in detail at his philosophical practice in his most sustained mature project, his work on pure and applied inductive logic. I, first, specify the sort of engineering Carnap is engaged in as involving an engineering design problem and then draw out the complications of design (...) problems from current work in history of engineering and technology studies. I then model Carnap’s practice based on those lessons and uncover ways in which Carnap’s technical work in inductive logic takes some of these lessons on board. This shows ways in which Carnap’s philosophical project subtly changes right through his late work on induction, providing an important corrective to interpretations that ignore the work on inductive logic. Specifically, I show that paying attention to the historical details of Carnap’s attempt to apply his work in inductive logic to decision theory and theoretical statistics in the 1950s and 1960s helps us understand how Carnap develops and rearticulates the philosophical point of the practical/theoretical distinction in his late work, offering thus a new interpretation of Carnap’s technical work within the broader context of philosophy of science and analytical philosophy in general. (shrink)

Abstract: The Pull Bayesian Significance Test (FBST) for precise hy- potheses is applied to a Multivariate Normal Structure (MNS) model. In the FBST we compute the evidence against the precise hypothesis. This evi- dence is the probability of the Highest Relative Surprise Set (HRSS) tangent to the sub-manifold (of the parameter space) that defines the null hypothesis. The MNS model we present appears when testing equivalence conditions for genetic expression measurements, using micro-array technology.

The Full Bayesian Significance Test, FBST, is extensively reviewed. Its test statistic, a genuine Bayesian measure of evidence, is discussed in detail. Its behavior in some problems of statistical inference like testing for independence in contingency tables is discussed.

Philosophical theories of evidence have been on offer, but they are mostly evaluated in terms of all-or-none desiderata — if they fail to meet one of the desiderata, they are not a satisfactory theory. In this thesis, I aim to accomplish three missions. Firstly, I construct a new way of evaluating theories of evidence, which I call a tool-based view. Secondly, I analyse the nature of what I will call the various relevance-mediating vehicles that each theory of evidence employs. Thirdly, (...) I articulate the comparative core of evidential reasoning in the historical sciences, one which is overlooked in major theories of evidence. On the first mission, I endorse a meta-thesis of pluralism on theories of evidence, namely a tool-based view. I regard a theory of evidence as a purpose-specific and setting-sensitive tool which has its own strengths, difficulties and limitations. Among the major theories of evidence I have reviewed, I focus on Achinstein’s explanationist theory, Cartwright’s argument theory and Reiss’s inferentialist account, scrutinising and evaluating them against the purposes they set out and the scope of their applications. On the second mission, I note that there is no such thing as intrinsically ‘being evidence’. Rather, I hold that relevance-mediating vehicles configure data, materials or claims in such ways that some of them are labelled evidence. I identify the relevance-mediating vehicles that the theories of evidence employ. On the final mission, I argue that the likelihoodist account is an appropriate tool for explaining the evidential reasoning in poorly specified settings where likelihoods can be only imprecisely compared. Such settings, I believe, are typical in the historical sciences. Using the reconstruction of proto-sounds in historical linguistics as a case study, I formalise the rationale behind it by means of the law of likelihood. (shrink)

Bayesianism and likelihoodism are two of the most important frameworks philosophers of science use to analyse scientific methodology. However, both frameworks face a serious objection: much scientific inquiry takes place in highly idealized frameworks where all the hypotheses are known to be false. Yet, both Bayesianism and likelihoodism seem to be based on the assumption that the goal of scientific inquiry is always truth rather than closeness to the truth. Here, I argue in favor of a verisimilitude framework for inductive (...) inference. In the verisimilitude framework, scientific inquiry is conceived of, in part, as a process where inference methods ought to be calibrated to appropriate measures of closeness to the truth. To illustrate the verisimilitude framework, I offer a reconstruction of parsimony evaluations of scientific theories, and I give a reconstruction and extended analysis of the use of parsimony inference in phylogenetics. By recasting phylogenetic inference in the verisimilitude framework, it becomes possible to both raise and address objections to phylogenetic methods that rely on parsimony. (shrink)

The likelihood principle is typically understood as a constraint on any measure of evidence arising from a statistical experiment. It is not sufficiently often noted, however, that the LP assumes that the probability model giving rise to a particular concrete data set must be statistically adequate—it must “fit” the data sufficiently. In practice, though, scientists must make modeling assumptions whose adequacy can nevertheless then be verified using statistical tests. My present concern is to consider whether the LP applies to these (...) techniques of model verification. If one does view model verification as part of the inferential procedures that the LP intends to constrain, then there are certain crucial tests of model verification that no known method satisfying the LP can perform. But if one does not, the degree to which these assumptions have been verified is bracketed from the evidential evaluation under the LP. Although I conclude from this that the LP cannot be a universal constraint on any measure of evidence, proponents of the LP may hold out for a restricted version thereof, either as a kind of “ideal” or as defining one among many different forms of evidence. (shrink)