A Battle in the Statistics Wars: a simulation-based comparison of Bayesian, Frequentist and Williamsonian methodologies

Synthese 199 (5-6):13689-13748 (2021)
  Copy   BIBTEX

Abstract

The debates between Bayesian, frequentist, and other methodologies of statistics have tended to focus on conceptual justifications, sociological arguments, or mathematical proofs of their long run properties. Both Bayesian statistics and frequentist (“classical”) statistics have strong cases on these grounds. In this article, we instead approach the debates in the “Statistics Wars” from a largely unexplored angle: simulations of different methodologies’ performance in the short to medium run. We conducted a large number of simulations using a straightforward decision problem based around tossing a coin with unknown bias and then placing bets. In this simulation, we programmed four players, inspired by Bayesian statistics, frequentist statistics, Jon Williamson’s version of Objective Bayesianism, and a player who simply extrapolates from observed frequencies to general frequencies. The last player functions as a benchmark: a statistical methodology should at least outperform a crude form of induction. We focused on the performance of these methodologies in guiding the players towards good decisions. Unlike an earlier simulation study of this type, we found no systematic difference in performance between the Bayesian and frequentist players, provided the Bayesian used a flat prior and the frequentist used a low confidence level. Unlike that study, we were able to use Big Data methods to mitigate problems of random error in the simulation results. The Williamsonian player, who is a novel element of our study, also had no systematic differences in their performance, provided that they use a low confidence level. These players performed similarly even in the very short run, when players were making different decisions. Our study indicates that all three methodologies should be taken seriously by philosophers and practitioners of statistics. However, the frequentist and Williamsonian performed poorly when their confidence levels were high, and the Bayesian was surprisingly harmed by biased priors, providing some unexpected lessons for these methodologies when facing this type of decision problem.

Similar books and articles

Bayesian statistics in medicine: A 25 year review.Deborah Ashby - 2006 - Statistics in Medicine 25:3589-3631.
The Philosophical Relevance of Statistics.Deborah G. Mayo - 1980 - PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association 1980:97 - 109.
Statistical Inference and the Replication Crisis.Lincoln J. Colling & Dénes Szűcs - 2018 - Review of Philosophy and Psychology 12 (1):121-147.
Bayeswatch: an overview of Bayesian statistics.Peter C. Austin, Lawrence J. Brunner & S. M. Janet E. Hux Md - 2002 - Journal of Evaluation in Clinical Practice 8 (2):277-286.
Bayes and health care research.Peter Allmark - 2004 - Medicine, Health Care and Philosophy 7 (3):321-332.

Analytics

Added to PP
2021-08-29

Downloads
42 (#332,036)

6 months
7 (#175,814)

Historical graph of downloads
How can I increase my downloads?

Author Profiles

Citations of this work

Formal Epistemology Meets Mechanism Design.Jürgen Landes - 2023 - Journal for General Philosophy of Science / Zeitschrift für Allgemeine Wissenschaftstheorie 54 (2):215-231.

Add more citations

References found in this work

The Logic of Decision.Richard C. Jeffrey - 1965 - New York, NY, USA: University of Chicago Press.
Scientific reasoning: the Bayesian approach.Peter Urbach & Colin Howson - 1993 - Chicago: Open Court. Edited by Peter Urbach.
A treatise on probability.John Maynard Keynes - 1921 - Mineola, N.Y.: Dover Publications.
The Logic of Scientific Discovery.Karl Popper - 1959 - Studia Logica 9:262-265.

View all 63 references / Add more references