An analysis of about one million resolved questions and contracts found that collective forecasting platforms generally matched observed event rates across 12 domains, while some highly consequential areas showed systematic bias.
Published in the Proceedings of the National Academy of Sciences, the study examined forecasts from four platforms. The prediction periods ranged from hours to years.
The researchers assessed calibration, an aggregate measure of how closely assigned probabilities match the frequency of outcomes across many forecasts. The paper found that events given a particular probability occurred at approximately that rate across the sample.
That finding describes performance across groups of forecasts. It does not mean any individual prediction is certain. The authors said aggregated forecasts can inform estimates of event likelihood, while cautioning against treating the result as certainty about a specific event.
More detailed analyses found systematic tendencies to overestimate or underestimate outcomes in some highly consequential areas.
