A probability forecast scored ex post using a probability scoring rule (e.g. Brier) is analogous to a risky financial security. With only superficial adaptation, the same economic logic by which securities are valued ex ante – in particular, portfolio theory and the capital asset pricing model (CAPM) – applies to the valuation of probability forecasts. Each available forecast of a given event is valued relative to each other and to the “market” (all available forecasts). A forecast is seen to be (...) more valuable the higher its expected score and the lower the covariance of its score with the market aggregate score. Forecasts that score highly in trials when others do poorly are appreciated more than those with equal success in “easy” trials where most forecasts score well. The CAPM defines economically rational (equilibrium) forecast prices at which forecasters can trade shares in each other’s ex post score – or associated monetary payoff – thereby balancing forecast risk against return and ultimately forming optimally hedged portfolios. Hedging this way offers risk averse forecasters an “honest” alternative to the ruse of reporting conservative probability assessments. (shrink)

Simulation evidence obtained within a Bayesian model of price-setting in a betting market, where anonymous gamblers queue to bet against a risk-neutral bookmaker, suggests that a gambler who wants to maximize future profits should trade on the advice of the analyst cum probability forecaster who records the best probability score, rather than the highest trading profits, during the preceding observation period. In general, probability scoring rules, specifically the log score and better known “Brier” (quadratic) score, are found to have higher (...) probability of ranking rival analysts in predetermined “correct” order than either (i) the more usual method of counting categorical forecast errors (misclassifications), or (ii) an economic measure of forecasting success, described here as the “Kelly score” and defined as the trading profits accumulated by making log optimal bets (i.e. Kelly betting) against the market maker based on the probability forecasts of the analyst being assessed. This runs counter to the conventional wisdom that financial forecasts are more aptly evaluated in terms of their financial consequences than by an abstract non-monetary measure of statistical accuracy such as the number of misclassifications or a probability score. (shrink)

Cognitive scientists have revealed systematic errors in human reasoning. There is disagreement about what these errors indicate about human rationality, but one upshot seems clear: human reasoning does not seem to fit traditional views of human rationality. This concern about rationality has made its way through various fields and has recently caught the attention of philosophers. The concern is that if philosophers are prone to systematic errors in reasoning, then the integrity of philosophy would be threatened. In this paper, I (...) present some of the more famous work in cognitive science that has marshaled this concern. Then I present reasons to think that those with training in philosophy will be less prone to certain systematic errors in reasoning. The suggestion is that if philosophers could be shown to be less prone to such errors, then the worries about the integrity of philosophy could be constrained. Then I present evidence that, according to performance on the CRT (Frederick 2005), those who have benefited from training and selection in philosophy are indeed less prone to one kind of systematic error: irrationally arbitrating between intuitive and reflective responses. Nonetheless, philosophers are not entirely immune to this systematic error, and their proclivity for this error is statistically related to their responses to a variety of philosophical questions. So, while the evidence herein puts constraints on the worries about the integrity of philosophy, it by no means eliminates these worries. The conclusion, then, is that the present evidence offers prima facie reasons to ascribe a mitigated privilege to philosophers' ability to rationally arbitrate between intuitive and reflective responses. (shrink)