Stochastic forecasts in complex environments can benefit from combining the estimates of large groups of forecasters (“judges”). But aggregating multiple opinions faces several challenges. First, human judges are notoriously incoherent when their forecasts involve logically complex events. Second, individual judges may have specialized knowledge, so different judges may produce forecasts for different events. Third, the credibility of individual judges might vary, and one would like to pay greater attention to more trustworthy forecasts. These considerations limit the value of simple aggregation methods like linear averaging. In this paper, a new algorithm is proposed for combining probabilistic assessments from a large pool of judges. Two measures of a judge’s likely credibility are introduced and used in the algorithm to determine the judge’s weight in aggregation. The algorithm was tested on a data set of nearly half a million probability estimates of events related to the 2008 U.S. presidential election (∼ 16000 judges).
Keywords No keywords specified (fix it)
Categories (categorize this paper)
Edit this record
Mark as duplicate
Export citation
Find it on Scholar
Request removal from index
Revision history

Download options

PhilArchive copy

Upload a copy of this paper     Check publisher's policy     Papers currently archived: 70,307
External links

Setup an account with your affiliations in order to access resources via your University's proxy server
Configure custom proxy (use this if your affiliation does not provide a proxy)
Through your library

References found in this work BETA

Add more references

Citations of this work BETA

Graded Incoherence for Accuracy-Firsters.Glauber De Bona & Julia Staffel - 2017 - Philosophy of Science 84 (2):189-213.

View all 6 citations / Add more citations

Similar books and articles

Calibration, Coherence, and Scoring Rules.Teddy Seidenfeld - 1985 - Philosophy of Science 52 (2):274-294.
Aggregating Causal Judgments.Richard Bradley, Franz Dietrich & Christian List - 2014 - Philosophy of Science 81 (4):491-515.
Rationality as Weighted Averaging.Keith Lehrer - 1983 - Synthese 57 (3):283 - 295.


Added to PP index

Total views
78 ( #148,512 of 2,507,805 )

Recent downloads (6 months)
1 ( #416,820 of 2,507,805 )

How can I increase my downloads?


My notes