Forecasting for Policy: helping public institutions to understand the future
Information about judgemental forecasting and our forecasting project in the Czech Republic
"Forecasting is an interesting approach to aggregating views on possible future developments that is useful in personal and public decision-making."
FORPOL is a joint project between Czech Priorities and Metaculus. It is funded by the Effective Altruism Infrastructure Fund. It aims to increase our understanding of practical ways, how to improve public policy-making with the use of judgemental forecasting.
The main objective of the FORPOL (Forecasting for Policy) project is to provide public institutions with useful predictions on relevant societal topics. We currently operate forecasting tools in the Czech republic, but are interested in establishing international cooperations as well. FORPOL is also an opportunity for (currently only czech speaking) participants to improve their forecasting skills.
We are using judgmental forecasting, a method for obtaining estimates of future scenarios based on probabilistic reasoning and aggregation of a large number of qualified forecasters. Judgmental forecasting is especially appropriate for predictions on novel issues or where there is insufficient data (e.g., societal impacts of a pandemic, new technologies, etc.). It is also used on highly complex topics such as; geopolitics, crisis response and other social challenges.
Judgemental forecasting is based on quantitative data but also takes into account qualitative analysis. It thus presents the possibility of assessing the situation on the basis of currently available information.
A key element of the judgmental forecasting method is collective intelligence - instead of relying on the judgment of just one expert it estimates the outcome as the result of a group of forecasters aggregated predictions. This way of forecasting minimizes information noise and bias and, at the same time, maximizes the volume of information on which the aggregated forecasts are based.
FORPOL currently operates in the Czech Republic, but we are interested in other international cooperations. Contact us!
Our ongoing forecasting tournament (in Czech) takes place Oct 5, 2022 - March 30, 2023 at forpol.metaculus.com
Benefits of forecasting tournaments
Compared to other methods of aggregating expert opinions (interviews and panels, questionnaires, Delphi), forecasting tournaments are focused on three aspects:
- Accuracy: By using numerical or probabilistic predictions instead of vague verbal expressions ("most likely", "very likely", etc.), it is possible to share more accurate information and subsequently evaluate and modify one's own predictions. These probabilities also make it possible to assess the accuracy of predictions and, after a period of time, identify consistently above-average forecasters whose predictions can get more weight in the model overall score based on objective criteria.
- Balance: By using an online platform, it is possible to involve more participants with diverse expertise and experience to boost information sharing.
- Engagement: The online platform allows forecasters to further comment about the questions they are solving. Participants have strong incentives to get involved in these discussions through rewards which elevate their prediction score.
The online platform automatically collects all the inputs, which serve as a digital repository of views and resources to improve prediction skills within the forecaster's community.
Information for public institutions
The public administration must make decisions based on data and previous knowledge. It also needs to prepare for potential future scenarios and thus increase its ability to respond adequately. Foresight methods serve this purpose, and Czech Priorities strongly supports its frequent and systematic use.
FORPOL applies one of the foresight methods - judgmental forecasting. It aims to provide state authorities with qualified estimates for their decision-making. We also seek to share our practical knowledge with the international community.
Judgmental forecasting is a method for obtaining estimates of future results based on probabilistic reasoning and aggregation of a large number of qualified judgments. Using judgmental forecasting to estimate future circumstances and parameters has the following benefits:
- Wide range of issues - Judgmental Forecasting can be applied to address brand-new and multi-sectoral issues.
- Data-intensive phenomena - Judgmental Forecasting can provide estimates of phenomena where mathematical models cannot be reliably used (e.g. due to lack or absence of data).
- Speed to get an estimate - The necessary estimates can be obtained relatively quickly - within weeks.
- Context - Judgmental Forecasting gives the opportunity to forecasters to elaborate further about their reasoning, context and criteria in their predictions. Other experts can build upon that to expand community knowledge.
Our standard collaboration process has three phases - Scoping, Forecasting, and Utilization.
Scoping: At this stage, we will usually take you through two workshops to define the objectives, future use of the predictions, and select topics for forecasting. We will then design the final structure of the questions.
Forecasting: We will announce a new tournament on the platform with these questions. Forecasters will have the opportunity to predict and write down their comments for around three weeks. In the meantime, there will be a guided debate with selected experts. We will reward the authors with the best ideas. In this phase, you will have access to the platform at any time.
Utilization: After a final talk with the experts, we will provide you with a final report that includes the aggregate of predictions from validated forecasters, a summary with the most frequent considerations, and the main justifications for their results. Finally, we will apply the outputs to advise real-world public stakeholders.